A Technology Journey
Launch Microsoft SQL Server Management Studio (SSMS) with an administrative account.
Type the fully qualified domain name in the Server name: textbox.
Change the Authentication method from the drop-down box.
Click the Connect button.
Right-click the Databases folder > click New Database…
Type the name of the new database to create in the Database name: textbox.
Click the OK button.
Expand the Databases folder and verify the database you created is present.
At least one App Volumes Manager should be installed before following this page. Here is the page for Installing VMware App Volumes Manager.
Ensure Domain Name System (DNS) records exist for forward and reverse look up of all servers in the infrastructure.
Certificates for App Volumes Managers are in Privacy Enhanced Mail (PEM) format. The following table is a quick reference to the required files with more in-depth details to follow.
| Use Case | File Name | File System Location |
|---|---|---|
| Web Interface Certificate | <user-defined>.crt | C:\Program Files (x86)\CloudVolumes\Manager\nginx\conf |
| Web Interface Key | <user-defined>.key | C:\Program Files (x86)\CloudVolumes\Manager\nginx\conf |
| Active Directory (LDAPS) | adCA.pem | C:\Program Files (x86)\Cloud Volumes\Manager\config |
| vCenter Certificate | cacert.pem | C:\Program Files (x86)\Cloud Volumes\Manager\config |
The web interface uses a self-signed certificate installed when the App Volumes Manager is installed initially. This should be replaced with a PEM certificate and key. If planning to use a load balancer, ensure the load balanced fully qualified domain name is set for the Common Name (CN) attribute with each App Volumes Manager’s fully qualified domain name added to the Subject Alternative Name (SAN) attribute of the certificate.
Add the certificate and key file to the C:\Program Files (x86)\CloudVolumes\Manager\nginx\conf directory.
Open the nginx.conf file as an administrative user.
Edit lines 57 (ssl_certificate) and 58 (ssl_certificate_key) to reflect the names of the uploaded certificate and key.
Restart the App Volumes Manager service.
The intermediate and root certificates that signed the Active Directory Domain Controller certificate are necessary to use Secure LDAP (LDAPS). The order of the certificate chain I found that works is root + intermediate. The certificate chain is named adCA.pem and located at C:\Program Files (x86)\CloudVolumes\Manager\config.
Open each file individually, starting with the root certificate, select all (ctrl + a) the contents, copy all (ctrl + c) the contents, including the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–, and paste (ctrl + v) into a new file. Continue pasting intermediate certificates after the root certificate, without adding extra spaces, if there is more than one intermediate certificate. See below for example of a certificate chain.
-----BEGIN CERTIFICATE----- root certificate details removed for brevity -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- intermediate certificate details removed for brevity -----END CERTIFICATE-----
Save this file as adCA.pem into the C:\Program Files (x86)\CloudVolumes\Manager\config directory.
The vCenter certificate is needed in order to trust the vCenter Server when setting up a Machine Manager. The order of the certificate chain I found that works is root + intermediate + machine. This file is saved as cacert.pem into the C:\Program Files (x86)\CloudVolumes\Manager\config directory.
Open each file individually, starting with the root certificate, select all (ctrl + a) the contents, copy all (ctrl + c) the contents, including the —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–, and paste (ctrl + v) into a new file. Continue pasting intermediate certificates after the root certificate, without adding extra spaces, if there is more than one intermediate certificate. Continue pasting the machine certificate after the intermediate certificates, without adding extra spaces. See below for example of a certificate chain.
-----BEGIN CERTIFICATE----- root certificate details removed for brevity -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- intermediate certificate details removed for brevity -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- machine certificate details removed for brevity -----END CERTIFICATE-----
Save this file as cacert.pem into the C:\Program Files (x86)\CloudVolumes\Manager\config directory.
Extract the certificate.
openssl pkcs12 -in mycaservercert.pfx -nokeys -out mycaservercert.pem
Extract the key.
openssl pkcs12 -in mycaservercert.pfx -nodes -nocerts -out mycaservercertkey.pem
Convert the key to RSA.
openssl rsa -in mycaservercertkey.pem -check -out mycaservercertkeyrsa.pem
You will likely want to create a service account to use with App Volumes Manager to connect to Active Directory.
Another account is required to connect App Volumes Manager into vCenter Server. This can be the same account as above, another Active Directory account, or a local vCenter user account. The user can be assigned the Administrator role in vCenter or a custom vCenter Server role can be created with a subset of permissions. (https://docs.vmware.com/en/VMware-App-Volumes/2009/app-volumes-admin-guide/GUID-505624F3-F3EB-428C-BEA0-5BD7F6095A1F.html)
You will likely want to create various security groups; at a minimum, a security group for App Volumes Manager Administrators. I would suggest other groups for placing end-users into, though. This will make assigning applications much easier and aide in license tracking.
Open a web browser and type the fully qualified domain name for the App Volumes Manager you are configuring.
A Welcome to App Volumes Manager window displays.
Click the Get Started button.
Update the license to the license obtained when purchased. This will allow you to attach more than three App Volumes to a virtual machine.
[No screenshot]
Fill in the details to register an Active Directory domain on the AD Domains tab.
Click the Register button.
Click the Next button.
Add users or security groups to permit administration activities in the Choose Group: text box.
Click the Search button.
Ensure the correct group populates in the Choose Group: drop-down selection.
Click the Assign button.
Confirm the group you added is assigned the Administrators role. Note: One thing here is that if your administrative users only have Smart Card login, this will not work with App Volumes. A username and password is the only method, right now, to log in. The service account can be used or alternate accounts can be created that use username and password authentication.
Click the Next button.
Fill in the details for the machine managers on the Machine Managers tab.
Click the Save button.
Click the Next button.
Choose the Default Storage Location from Packages and Writable Volumes panes.
Click the Next button.
Choose the Import volumes immediately radio button.
Click the Set Defaults button.
Click the checkboxes according to your needs or the checkbox in the table header to select all.
Click the Upload button.
Click the Upload button in the Confirm Upload Templates window.
Review the settings here. One setting I change is in the Active Directory section. I change the Non-Domain Entities to Allow so that non-domain joined computers are visible and available for provisioning.
If you make any changes, be sure to click the Save button.
[No screenshot]
Web Interface Not Trusted: See Replacing the Self-Signed Certificate with a CA-Signed Certificate
Active Directory Domain Not Trusted: See Active Directory (LDAPS) Certificate
vCenter Server Certificate Not Trusted: See vCenter Certificate
Replace the Self-Signed Certificate with CA-signed Certificate: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.admin.doc/GUID-EFCC36A2-1609-4B47-969E-2A0CF9BC9B68.html
Configure CA Certificates in App Volumes Manager: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.admin.doc/GUID-4EA6EF73-7800-4241-9162-2C407AC4AA7A.html
App Volumes 2.12.1 – vCenter Certificate becomes unverified. (2150281): https://kb.vmware.com/s/article/2150281
Each App Volumes Manager server needs a machine certificate. If there is going to be a load balancer in front, add the load balanced fully qualified domain name to the Common Name (CN) attribute and each server’s fully qualified domain name to the Subject Alternative Name (SAN) attribute of the certificate.
Ensure the Microsoft SQL Server is secured with Transport Layer Security (TLS) certificates. See this page: Securing Microsoft SQL Server with Transport Layer Security (TLS) Certificates.
Ensure there is a designated database for the App Volumes Managers. See this page: Creating an App Volumes Manager Database.
Ensure each App Volumes Manager server’s Computer Account is added to the Microsoft SQL Server and the db_owner role is assigned to the App Volumes database. See this page: Adding Active Directory Computer Accounts to Microsoft SQL Server.
Download VMware App Volumes media from downloads.vmware.com.
Mount the App Volumes media. Right-click the downloaded file and click on Mount.
The App Volumes software media location opens.
Double-click the Installation folder.
As an administrative user, right-click the setup.msi and choose Install.
Note: If you are not logged on as an administrative user, open an elevated command prompt or PowerShell window.
The App Volumes Installer Setup opens.
Click the Next button.
Review the End-User License Agreement.
Check the I accept the terms in the License Agreement checkbox.
Click the Next button.
Choose the Install App Volumes Manager radio button.
Click the Install button.
Respond to the optional User Account Control window.
The App Volumes Manager Setup window opens.
Click the Next button.
Choose the Connect to an existing SQL Server Database radio button.
Click the Next button.
Type the fully qualified domain name of the Microsoft SQL Server or the Availability Group Listener in the Choose local or remote database server to use: textbox.
Ensure the Windows Integrated Authentication radio button is selected.
Click the Browse… button.
Choose the database target database.
Click the OK button.
The Name of database catalog to use or create: textbox is populated with the chosen database.
Warning: Do not check the Overwrite existing database (if any) checkbox unless you are wanting to start from scratch. This will overwrite the current database and will result in permanent data loss of existing data. If you are installing additional App Volumes Managers, leave this checkbox unchecked as well. You have been warned…
Ensure the Enable SQL Server certificate validation checkbox is checked.
Click the Next button.
Verify the HTTPS Port: is set to 443.
Verify the Allow connections over HTTP (insecure) checkbox is not checked.
Click the Next button.
Change the Location: or leave the default.
Click the Next button.
Click the Install button.
Optional: Check the checkbox to Show the Windows Installer log.
Click the Finish button.
If installation errors occur, review the installation logs located at C:\Program Files (x86)\CloudVolumes\Manager\log if the default location was used during the Wizard.
A common reason is that the Private Key for the Microsoft SQL Server is not available or the permissions are not set for the service account. Review the following page for help with securing Microsoft SQL Server. Securing Microsoft SQL Server with Transport Layer Security (TLS) Certificates
VMware App Volumes Documentation
https://docs.vmware.com/en/VMware-App-Volumes/index.html
If using a single server, the fully qualified domain name is only needed for the Common Name (CN) attribute. If using Always On High Availability, the fully qualified domain name of the individual SQL server needs to be in the CN attribute and the Availability Group Listener fully qualified domain name needs to be in the Subject Alternative Name (SAN) attribute. The Windows Server Failover Cluster (WSFC) fully qualified domain name does not need to be in the SAN.
When importing the certificate into the Certificates – Local Computer console (certlm.msc):
The SQL servers are most likely using at least one service account. If you are unsure of the service account name, you can open the Services console (services.msc) and view the Log On As column.
Open the Certificates – Local Computer console (certlm.msc)
Respond to optional User Account Control pop-up
Right-click the Personal folder > expand All Tasks > choose Import…
The Certificate Import Wizard opens.
Click the Next button.
Click the Browse… button.
Change the file type drop-down to Personal Information Exchange (*.pfx,*.p12).
Choose the machine certificate that contains a private key (the icon contains a key).
Click the Open button.
You are returned to the File to Import screen with the File name: path populated.
Click the Next button.
Type in a password, if the certificate has one.
Ensure the Mark this key as exportable. This will allow you to back up or transport your keys at a later time. checkbox is checked.
Verify the Include all extended properties. checkbox is checked.
Click the Next button.
Verify the Certificate store: is set to Personal.
Click the Next button.
Review the details.
Click the Finish button.
You should receive a pop-up with the message, “The import was successful.”
Click the OK button.
Verify the certificate installed in the Personal folder. Double-click the certificate to open.
On the General tab, verify the You have a private key that corresponds to this certificate. message is displayed
Click the OK button.
Open the Intermediate Certificate Authorities folders and verify the intermediate certificate installed.
Open the Trusted Root Certification folder and verify the root certificate Installed.
Right-click the certificate > expand All Tasks > choose Manage Private Keys…
Add or modify the Group or usernames: permissions for the private keys.
Click the Add… button to add the Service Account.
Type the name of the Service Account and click the Check Names button.
Choose the correct account if there are multiple names found.
Click the OK button.
Click the OK button.
Ensure the added account has Full Control.
Click the OK button.
Close the Certificates – Local Computer console.
Open SQL Server Configuration Manager.
Respond to optional User Account Control pop-up.
Expand SQL Server Network Configuration.
Right-click Protocols for <Instance_Name>.
Choose Properties.
Click the Certificates tab.
Use the drop-down to select the certificate.
Click the OK button.
You should receive a warning pop-up.
Click the OK button.
Click SQL Server Services in the SQL Server Configuration Manager.
Right-click the SQL Server (<Instance_Name>) service.
Click Restart on the service.
Note: If you receive an error, make sure the certificate has the private key marked as exportable and that the service account for the SQL Server service has Full Control.
Certificate Management (SQL Server Configuration Manager) https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/manage-certificates?view=sql-server-2016
# https://kb.vmware.com/s/article/2118048 # https://powershellexplained.com/2016-10-21-powershell-installing-msi-files/# # Install VMware Horizon Agent # Start-Process C:\ImageBuild\agents\VMware-Horizon-Agent-x86_64-7.13.0-16975066.exe -Wait -ArgumentList '/s /v"/qn VDM_VC_MANAGED_AGENT=1 ADDLOCAL=BlastUDP,ClientDriveRedirection,Core,FLASHMMR,HelpDesk,NGVC,PerfTracker,RDP,RTAV,SmartCard,ThinPrint,TSMMR,V4V,VmVideo,VMWMediaProviderProxy,VmwVaudio,VmwVidd VDM_FIPS_ENABLED=1 REBOOT=ReallySuppress /l*v C:\ImageBuild\logs\HorizonAgent-Install.log"'# # Install VMware Dynamic Environment Manager (DEM) Agent # Start-Process msiexec.exe -Wait -ArgumentList '/i "C:\ImageBuild\agents\VMware Dynamic Environment Manager Enterprise 10.1 x64.msi" /qn NOADCONFIGFILEPATH="\\synology.aaronrombaut.com\Dynamic Environment Manager\dem-configuration\general" /l*v C:\ImageBuild\logs\DEMAgent-Install.log'# # Install VMware App Volumes Agent # Start-Process msiexec.exe -Wait -ArgumentList '/i "C:\ImageBuild\agents\App Volumes Agent.msi" /norestart /qn MANAGER_ADDR=app-001v.aaronrombaut.com MANAGER_PORT=443 EnforceSSLCertificateValidation=1 /l*v C:\ImageBuild\logs\AppVolumesAgent-Install.log'
This file can be downloaded below. Make sure to view the contents before running. Change the file extension to .ps1.
ref: https://kb.vmware.com/s/article/67789
You work in a hardened environment and you don’t have an administrative username and password because you only have smart cards or tokens.
According to VMware, this is expected behavior. Uh, what?
According to VMware, “Use Windows SSPI to pass through the Windows logged session Smart Card credentials to PowerCLI to authenticate to vCenter.”
What does that even mean? That was a lot of words that didn’t really explain how to “work around” the issue. So, let’s break it down.
Windows SSPI is Security Support Provider Interface. More details can be found here, https://docs.microsoft.com/en-us/windows/win32/rpc/security-support-provider-interface-sspi-
The optimal way to address this is to have a dedicated workstation or server in the data center that you can log into with PowerShell and PowerCLI installed. Since you are hopefully logging in with your administrator token and your administrator account has either been given permission directly on vSphere or through a security group, then you should be able to just log into the server when you try to use Connect-VIServer (https://vdc-repo.vmware.com/vmwb-repository/dcr-public/85a74cac-7b7b-45b0-b850-00ca08d1f238/ae65ebd9-158b-4f31-aa9c-4bbdc724cc38/doc/Connect-VIServer.html). The credential of the currently logged on user, that’s you, will get passed along with the request.
If you work in IT in the DoD in any capacity, then you know your systems can be a pain to work with if you followed a Security Technical Implementation Guide (STIG). This can be even more of a pain when following a commercial vendor’s installation or configuration documentation, since they write in the general sense and can’t possibly know every quirk in our environment.
From the Windows Server 2016 STIG, (Finding ID: V-73691) the LAN Manager authentication level must be set to send NTLMv2 response only and to refuse LM and NTLM.
From the VMware vSphere 6.5 ESXi STIG, (Finding ID: V-94023) the ESXi host must use the vSphere Authentication Proxy to protect passwords when adding ESXi hosts to Active Directory.
So even though the current STIG is only referencing vSphere 6.5, this can still be used for newer versions. During the configuration of a vCenter Server 7.0 Authentication Proxy, I found that the proxy kept not authenticating. After reviewing the logs on the Domain Controller, it was found that Likewise was sending the authentication to the Domain Controller with an NTLM response and not NTLMv2.
Reconfigure the Likewise service to send NTLMv2 and try to configure the Authentication Proxy again. This time, the configuration should be successful. You can further test by adding an ESXi host to vCenter and ensure it properly joins the domain.
Fill in the Domain, Domain User, and Domain Password settings. Click the SAVE button.
Notice the Operation failed! message.
You can also view the log located at /var/log/vmware/vmcam/vmcam-sca.log. Below is an excerpt, notice the Enter password for Service.vSphereAuth line. That indicates that it was not sending the correct password because we know that we typed it into the window.
I ended up tailing the log with the following so I could see in real-time what was occurring.
tail -f /var/log/vmware/vmcam/vmcam-sca.log
Run the following bold commands, the rest is for context.
/opt/likewise/bin/lwregshell
\> cd HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> list_values
"SendNTLMv2" REG_DWORD 0x00000000 (0)
"Support128bit" REG_DWORD 0x00000001 (1)
"Support56bit" REG_DWORD 0x00000001 (1)
"SupportKeyExchange" REG_DWORD 0x00000001 (1)
"SupportNTLM2SessionSecurity" REG_DWORD 0x00000001 (1)
"SupportUnicode" REG_DWORD 0x00000001 (1)
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> set_value SendNTLMv2 1
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> list_values
+ "SendNTLMv2" REG_DWORD 0x00000001 (1)
"Support128bit" REG_DWORD 0x00000001 (1)
"Support56bit" REG_DWORD 0x00000001 (1)
"SupportKeyExchange" REG_DWORD 0x00000001 (1)
"SupportNTLM2SessionSecurity" REG_DWORD 0x00000001 (1)
"SupportUnicode" REG_DWORD 0x00000001 (1)
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> quit
/opt/likewise/bin/lwsm restart lwio
Stopping service reverse dependency: lsass
Stopping service reverse dependency: rdr
Stopping service: lwio
Starting service: lwio
Starting service reverse dependency: rdr
Starting service reverse dependency: lsass
Go back to vSphere Client and configure the Authentication Proxy again. You may have to disable and enable the service again if you didn’t restart the service while connected through SSH. This time, it should work without an error. If you are tailing the log, then you should see Active Directory domain added successfully at the bottom.
From here you can Import the vSphere Authentication Proxy Certificate to ESXi Host.