A Technology Journey
I use a YubiKey 5Ci (by Yubico) in my lab. This allows me to log in with a smart card interface. If you are looking for information on how to configure smart card access in your lab, please reference the following post: MyLab: Smart Card Authentication
Continue reading “Configuring Smart Card | Common Access Card (CAC) | Personal Identity Verification (PIV) in VMware vSphere and VMware Horizon”The other day I was working on a vCenter and had to evacuate the vCLS virtual machines from a failed cluster I was working on. If you are unfamiliar with this, here is the kb that explains how to do it. The short version is that you navigate to the cluster, capture the cluster id from the URL bar, and then add an advanced setting. After about a minute or less, the vCLS virtual machines should power off and be deleted.
Continue reading “Removing Advanced vCenter Server Settings”I am using VMware vSphere 7.0.3 for this lab. Before starting the deployment, ensure there are forward and reverse DNS records created for the ESXi host that vCenter will be installed on, the vCenter Server appliance itself, and optionally any NTP servers. Ideally, there should be localized NTP.
Continue reading “MyLab: VMware vSphere”
This occurred for me when upgrading to or installing a new vCenter 7 and replacing the self-signed certificate. I tested in a lab and was able to successfully install both vCenter 7 and Orchestrator 8.3. I was able to successfully configure both appliances and log in, as well. I did use vSphere Authentication as the Orchestrator’s Identity Provider. As soon as I replaced the self-signed certificate on vCenter, I immediately received the following when logging into Orchestrator:
Uh-oh! So after two weeks or so and lot’s of doing this and that and trying this and that, I think I finally found the resolution. This is actually in the VMware documents, but the document is not quite complete with the information needed to successfully run the commands. Here is the document, https://docs.vmware.com/en/vRealize-Orchestrator/8.3/com.vmware.vrealize.orchestrator-install-config.doc/GUID-66B37DF2-052E-44A0-929E-E4F53E1BCCE3.html. I have detailed the process in full later in this blog post.
For completeness sake, I am going to show the entire process. Please feel free to scroll to the interesting sections below to resolve. I am not going to show how to deploy the appliances, just that they will be in vSphere and available as a starting point.
Installed, configured, and checking the services for a “known good”.
When I navigate to my vCenter appliance, I can see that it is using an untrusted certificate.
I perform the necessary steps to continue on. Your browser may be different and your organization’s policies may be different. If your organization is using HTTP Strict Transport Security (HSTS), you will likely be unable to continue without some very tricky manipulation or replacing the self-signed certificate to a known and trusted certificate. This is likely how or why you are in this predicament in the first place and had to search for this blog post.
The log in window is presented to me.
I verified I was able to successfully log in.
Navigate to the Orchestrator 8.3 appliance, I am presented with the following.
Since this appliance is fresh, I need to click on the Start the Control Center link and establish an authentication provider. I have to log in with the root account.
Click on Configure Authentication Provider
On this page, I chose vSphere for the Authentication mode setting and the Host address is my vCenter 7 appliance. I am presented with an Accept Certificate box. This will accept the current self-signed certificate, since that is all that is available. NOTE: You could wait to do this step until after you alter the TLS certificate on vCenter, but this article assumes you did not or that you already had an Orchestrator appliance deployed like I did.
Complete the Identity Service window with an administrative or service account that allows users to be queried. Click Register.
Type in a group to use as an Admin group, I used admins, then click the Search button.
A window will display that allows you to pick a security group based off your search criteria. Click Save Changes.
The Orchestrator appliance will be configuring in the background. This is not a fast process! Click on the home icon and choose Validate Configuration. You will see a message stating that a server restart is required…This will automatically happen after a two minute wait. Please be patient here…
You can continue clicking the Refresh button until you have all green check marks. This signifies the appliance rebooted and all services are back up.
Go back to the vco tab in the browser and choose the START THE ORCHESTRATOR CLIENT link. You should be presented the VMware vSphere log on screen. This signifies that your authentication provider is set up correctly to use vSphere. Try logging in.
I can verify that I can successfully log in without trouble.
Log in to vCenter server if you are not already. Lot’s of assumptions in the next few sections…I am going to assume you are logged in with an administrative user that also can perform cryptographic operations (https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-17568345-E59E-43A8-A811-92F8BE9C7719.html), then navigate to Menu > Administration > Certificate Management.
I am going to assume you know how to request a Certificate Signing Request (CSR), have already had the certificate signed, and have the necessary certificates in possession. If not, here is a VMware resource to get you started: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-E0609A99-A8D1-4336-BD3B-DE707E261A63.html. Under Machine SSL Certificate, click on Actions > Import and Replace Certificate.
In my case, I chose to use Replace with external CA certificate(requires private key) option and clicked the Next button.
Add in the machine certificate, the root and intermediate certificates chained together (for the chain, I always start with the root and add the intermediate certificates below), and the private key file. Click the Replace button.
If the certificates are successfully replaced, you should get logged out relatively quickly. Give this a few minutes as the vCenter server services are rebooting in the background.
Click the Login button after a few minutes. You may get a no healthy upstream message. Be patient and refresh your browser periodically, there is a lot going on with these appliances.
Eventually, you should get the vCenter log in screen. You can verify that your vCenter is secured by looking for the lock symbol. Go ahead and log back in to verify you can.
Click the Launch vSphere Client (HTML 5) button. Enter your credentials.
Click the Login button.
You may receive a Error occurred while fetching vmca root cert: com.vmware.vcenter.certificate_authority.get_root message. This just indicates that the vCenter server services are not fully restarted, yet. There may be a running task in Recent Tasks. Once this is complete, the message will go away after you refresh the User Interface (UI) or the browser window. Again, be patient as this may seem like an eternity in computer time or that it is broken, but it should come back up.
Again, be patient as this may seem like an eternity in computer time or that it is broken, but it should come back up.
We can confirm that we have logged in and that the message went away.
Ok, let’s try to log into the Orchestrator appliance.
So far, it looks promising. Click on the Start the Orchestrator Client link. Warning: you may actually get logged in. This is most likely due to a cookie on your browser. If you close your browser and try to log in again, you will most likely not be able to log in. That is what we are going to fix.
Enter your credentials and click the Login button.
Et voilà! There we are for us English speakers, the broken UI that is extremely frustrating to fix.
Here is the article from VMware on how to solve this (https://docs.vmware.com/en/vRealize-Orchestrator/8.3/com.vmware.vrealize.orchestrator-install-config.doc/GUID-66B37DF2-052E-44A0-929E-E4F53E1BCCE3.html). Unfortunately, not all the details are there to run the commands and if you are not experienced with the underlying technology of the Orchestrator appliance, like I wasn’t and really still am not, then this will just likely frustrate you even further. Let’s break this down…I added an indicator where I added steps to the original documented procedure.
1. Log in to the vRealize Orchestrator command line as root. (Added) I used an SSH session, but you can do this on the console with VMRC. I just wanted to be able to copy and paste commands.
2. (Added) Obtain the name of the <vRO pod> you will need for the next step.
kubectl -n prelude get pods
3. Run the kubectl -n prelude exec command. (Added) I used the last line from the clue in the example command of vco-server-app. I really did not know and the document does not explain.
Command from document.
kubectl -n prelude exec -it <vRO pod> -c vco-server-app -- bash
Command used with the <vRO pod> substituted.
kubectl -n prelude exec -it vco-app-77c8fb6659-fsr5v -c vco-server-app -- bash
4. Run the rpm command.
rpm -hiv --nodeps /vco-cfg-cli.rpm
5. Navigate to the /usr/lib/vco-cli/bin/ directory.
6. Run the following ./vro-configure-inner.sh trust commands.
From the document.
./vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri <vSphere-Auth-Provider-URI> --accept
With substituted <vSphere-Auth-Provider-URI>
./vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri vcsa70.aaronrombaut.com --accept
A lot of information will scroll past. I am only including a screenshot of the end of the command.
From the document.
./vro-configure-inner.sh trust --alias vco.sso.ssl.certificate --uri <vSphere-Auth-Provider-URI> --accept
With substituted <vSphere-Auth-Provider-URI>
./vro-configure-inner.sh trust --alias vco.sso.ssl.certificate --uri vcsa70.aaronrombaut.com --accept
A lot of information will scroll past. I am only including a screenshot of the end of the command, again.
7. Log out of the vRealize Orchestrator Appliance by using the exit command and log in again. (Added) If you only type exit here once, you will only exit the rpm command. You actually have to end the SSH session or console. You can type exit a second time to close the SSH session.
8. Run the following deploy.sh commands.
/opt/scripts/deploy.sh --onlyClean
A lot of information will scroll past. I am only including a screenshot of the end of the command. This command will take a few minutes to complete.
/opt/scripts/deploy.sh
A lot of information will scroll past. This command will take even longer to complete than the last command. Notice: if you prematurely end this command, your appliance will likely not be recoverable. Trust me when I tell you this. Learn from my pain…
You may even see messages that state Exit code and + return 0 like the screenshot below.
This is not complete, yet. Keep waiting until you see the following screen. (If you are nervous or impatient, get up and take a walk, this seriously takes a really long time, the appliance is going through a restart as part of this process).
Navigate to the appliance. Click on the Start the Orchestrator Client link to log on.
Type in your credentials and click the Login button.
Assuming everything went well, you should now be able to log back into the VMware vRealize Orchestrator appliance without error.
Please let me know if this helped you or if something I typed did not line up with what you experienced.
If you work in IT in the DoD in any capacity, then you know your systems can be a pain to work with if you followed a Security Technical Implementation Guide (STIG). This can be even more of a pain when following a commercial vendor’s installation or configuration documentation, since they write in the general sense and can’t possibly know every quirk in our environment.
From the Windows Server 2016 STIG, (Finding ID: V-73691) the LAN Manager authentication level must be set to send NTLMv2 response only and to refuse LM and NTLM.
From the VMware vSphere 6.5 ESXi STIG, (Finding ID: V-94023) the ESXi host must use the vSphere Authentication Proxy to protect passwords when adding ESXi hosts to Active Directory.
So even though the current STIG is only referencing vSphere 6.5, this can still be used for newer versions. During the configuration of a vCenter Server 7.0 Authentication Proxy, I found that the proxy kept not authenticating. After reviewing the logs on the Domain Controller, it was found that Likewise was sending the authentication to the Domain Controller with an NTLM response and not NTLMv2.
Reconfigure the Likewise service to send NTLMv2 and try to configure the Authentication Proxy again. This time, the configuration should be successful. You can further test by adding an ESXi host to vCenter and ensure it properly joins the domain.
Fill in the Domain, Domain User, and Domain Password settings. Click the SAVE button.
Notice the Operation failed! message.
You can also view the log located at /var/log/vmware/vmcam/vmcam-sca.log. Below is an excerpt, notice the Enter password for Service.vSphereAuth line. That indicates that it was not sending the correct password because we know that we typed it into the window.
I ended up tailing the log with the following so I could see in real-time what was occurring.
tail -f /var/log/vmware/vmcam/vmcam-sca.log
Run the following bold commands, the rest is for context.
/opt/likewise/bin/lwregshell
\> cd HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> list_values
"SendNTLMv2" REG_DWORD 0x00000000 (0)
"Support128bit" REG_DWORD 0x00000001 (1)
"Support56bit" REG_DWORD 0x00000001 (1)
"SupportKeyExchange" REG_DWORD 0x00000001 (1)
"SupportNTLM2SessionSecurity" REG_DWORD 0x00000001 (1)
"SupportUnicode" REG_DWORD 0x00000001 (1)
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> set_value SendNTLMv2 1
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> list_values
+ "SendNTLMv2" REG_DWORD 0x00000001 (1)
"Support128bit" REG_DWORD 0x00000001 (1)
"Support56bit" REG_DWORD 0x00000001 (1)
"SupportKeyExchange" REG_DWORD 0x00000001 (1)
"SupportNTLM2SessionSecurity" REG_DWORD 0x00000001 (1)
"SupportUnicode" REG_DWORD 0x00000001 (1)
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> quit
/opt/likewise/bin/lwsm restart lwio
Stopping service reverse dependency: lsass
Stopping service reverse dependency: rdr
Stopping service: lwio
Starting service: lwio
Starting service reverse dependency: rdr
Starting service reverse dependency: lsass
Go back to vSphere Client and configure the Authentication Proxy again. You may have to disable and enable the service again if you didn’t restart the service while connected through SSH. This time, it should work without an error. If you are tailing the log, then you should see Active Directory domain added successfully at the bottom.
From here you can Import the vSphere Authentication Proxy Certificate to ESXi Host.
ref: https://code.vmware.com/docs/11794/cmdlet-reference/doc/Get-VIPrivilege.html
ref: https://code.vmware.com/docs/11794/cmdlet-reference/doc/New-VIRole.html
ref: https://code.vmware.com/docs/11794/cmdlet-reference/doc/Set-VIRole.html
$VIRoleName = "View Manager Role"
$VIRolePrivileges = @(`
# Folder
'Create Folder', 'Delete Folder',`
# Datastore
'Allocate space',`
# Virtual Machine - Configuration
'Add or remove device', 'Advanced configuration', 'Modify device settings',`
# Virtual Machine - Interaction
'Power off', 'Power on', 'Reset', 'Suspend', 'Perform wipe or shrink operations',`
# Virtual Machine - Inventory
'Create new', 'Create from existing', 'Remove',`
# Virtual Machine - Provisioning
'Customize guest', 'Deploy template', 'Read customization specifications', 'Clone template', 'Clone Virtual Machine',`
# Resource
'Assign virtual machine to resource pool',`
# Global
'Act as vCenter Server',`
# Host
'Advanced settings',`
# Profile-driven Storage
'Profile-driven storage view', 'Profile-driven storage update'
)
try {
# Get list of current Roles
$VIRoles = Get-VIRole
# Check if Role exists
foreach($VIRole in $VIRoles) {
if ($VIRole.Name -like $VIRoleName) {
# Role exists
exit
}
}
# Assume the Role does not exist
# Create the new Role
New-VIRole -Name $VIRoleName
# Add the Privileges to the Role
foreach($VIRolePrivilege in $VIRolePrivileges) {
Set-VIRole -Role $VIRoleName -AddPrivilege $VIRolePrivilege
}
} catch {
}
Copy and paste the contents above to a new PowerShell file. This script will check if the given Role exists and exit or it will create the Role and add the Privileges. This script will not check the current assigned Privileges if the Role exists.
I wanted to update vCenter Server Appliance update 1 (vCenter Appliance 6.7 U1 (6.7.0.20000)) to the most current patch level which at the time of this post is Update 3k.
I navigated to VMware’s patch repository, https://my.vmware.com/group/vmware/patch#search, and selected VC and 6.7.0. I already have the appliance, so I downloaded the Appliance-Patch file as highlighted below.
Once downloaded, I added the .iso file to the VCSA virtual machine’s CD/DVD drive using the VMware Remote Console window. (It’s my personal preference to not add ISO files to datastores as I feel it is a waste of space and unnecessary)
I logged into the VMware Appliance Interface (VAMI), https://fqdn:5480, and chose Update > CHECK UPDATES > Check CD ROM. I was presented with a screen stating, “No applicable update found.” I was a little concerned and thought maybe I added the wrong ISO file, so I double checked.
After a little digging and documentation reading, I found that this upgrade path is not supported. I first have to patch to the base appliance level, in this case vCenter Appliance 6.7.0 update 3, before I can apply the security patch. The following is an excerpt from the document, Patching the vCenter Server Appliance and Platform Services Controller Appliance.
VMware makes patches available on a monthly basis. These patches can only be applied in between major releases of vCenter Server Appliance. For example, patches released for the initial release of vCenter Server Appliance 6.7, are not applicable to vCenter Server Appliance 6.7 Update 1, as any patches previously made available will be included with the Update 1 release.
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vcenter.upgrade.doc/GUID-043EF6BD-78F7-412F-837F-CBDF844F850C.html
So now that I have RTFM, I can try again in the correct order. I downloaded the vCenter Appliance Update 3 (VMware-vCenter-Server-Appliance-6.7.0.40000-14367737-patch-FP.iso) ISO from the patches repository and loaded it into the vCenter Server Appliance’s CD ROM. I went back to the VAMI and checked for updates again, this time with a positive result!
To update:
Ref: Patching the vCenter Server Appliance by Using the Appliance Management Interface (https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vcenter.upgrade.doc/GUID-E2E359B1-5834-4BFF-AEFE-6CEBFC8CC3D5.html) and subsequently, Install vCenter Server Appliance Patches (https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vcenter.upgrade.doc/GUID-E5E78149-4AC8-4DD7-BBA8-19CC17711D40.html).
Wait for the server to install the update and show Installation succeeded. Click the Close button.
Depending on how long the process took, you may need to log into the VAMI again. Verify the Version is updated from the previous.
This time you can load up the most recent patch and follow the update procedure again.
At the end, you should have an updated appliance with the most recent patch.
