A Technology Journey
I am opting to install VMware App Volumes 2303.1 (current at the time of writing) on Window Server 2022.
I am choosing to install VMware App Volumes to separate applications from the desktop to keep the golden image as lean as possible.
Continue reading “MyLab: VMware App Volumes”This is my attempt to keep track of various ports, protocols, and services for successful deployments of solutions.
| Port Number | TCP/UDP | IANA Service Name | IANA Description | Common Use |
|---|---|---|---|---|
| 135 | TCP | epmap | DCE endpoint resolution | RPC Endpoint Mapper |
| 389 | TCP/UDP | ldap | Lightweight Directory Access Protocol | LDAP |
| 636 | TCP | ldaps | ldap protocol over TLS/SSL (was sldap) | LDAP SSL |
| 3268 | TCP | msft-gc | Microsoft Global Catalog | LDAP GC |
| 3269 | TCP | msft-gc-ssl | Microsoft Global Catalog with LDAP/SSL | LDAP GC SSL |
| 53 | TCP/UDP | domain | Domain Name Server | DNS |
| 88 | TCP/UDP | kerberos | Kerberos | Kerberos |
| 445 | TCP | microsoft-ds | Microsoft-DS | SMB |
| 464 | TCP/UDP | kpasswd | kpasswd | Kerberos Password V5 |
| Port Number | TCP/UDP | IANA Service Name | IANA Description | Common Use |
|---|---|---|---|---|
| 67 | UDP | bootps | Bootstrap Protocol Server | DHCP (Server) |
| 68 | UDP | bootpc | Bootstrap Protocol Client | DHCP (Client) |
TCP/UDP: 53 domain Domain Name Server DNS
| Port Number | TCP/UDP | IANA Service Name | IANA Description | Common Use |
|---|---|---|---|---|
| 53 | TCP/UDP | domain | Domain Name Server | DNS |
| Port Number | TCP/UDP | IANA Service Name | IANA Description | Common Use |
|---|---|---|---|---|
| 123 | TCP/UDP | ntp | Network Time Protocol | NTP |
| Port Number | TCP/UDP | IANA Service Name | IANA Description | Common Use |
|---|---|---|---|---|
| 139 | TCP | netbios-ssn | NETBIOS Session Service | SMB |
| 445 | TCP | microsoft-ds | Microsoft-DS | SMB |
| Port Number | TCP/UDP | IANA Service Name | IANA Decription | Common Use |
|---|---|---|---|---|
| 1433 | TCP | ms-sql-s | Microsoft-SQL-Server | Microsoft SQL |
| Port Number | TCP/UDP | IANA Service Name | IANA Description | Common Use |
|---|---|---|---|---|
| 443 | TCP | https | http protocol over TLS/SSL | HTTP/S |
| 22 | TCP | ssh | The Secure Shell (SSH) Protocol | SSH |
| 5480 | TCP | – | – | VMware Appliance Management Interface (VAMI) |
| 9543 | TCP | – | – | |
| 902 | TCP | – | – | |
| 514 | TCP | shell | – | Syslog |
| Port Number | TCP/UDP | IANA Service Name | IANA Description | Common Use |
|---|---|---|---|---|
| 8443 | TCP | pcsync-https | PCsync HTTPS | Blast Extreme traffic |
| 443 | TCP | https | http protocol over TLS/SSL | Authentication |
| 22443 | TCP/UDP | – | – | Blast Extreme traffic |
| 3389 | TCP | ms-wbt-server | MS WBT Server | Remote Desktop Protocol (RDP) |
The following post is very long and will contain updates as the technology changes and I figure out better ways to accomplish these tasks. VMware Cloud Foundation, or VCF, requires at least four nodes for the Management Domain. Unfortunately, I do not have hundreds of thousands of dollars for physical hardware to test and learn VCF.
Since this is a nested lab, there are a few things that will need to be set up to make this work. These items, in particular networking, have to be configured in a way to allow the nested virtual machines to communicate. This is not applicable in production where the physical hosts are cabled into Top of Rack switches.
Continue reading “Building a Nested ESXi Lab for VMware Cloud Foundation (VCF) (updated 2023-Dec)”I have been working with clients that are using VxRail for their infrastructure. While administering these VxRail deployments, sometimes the mystic or root accounts get locked, the password’s expire or are just plain lost. Either way, it is a very frustrating situation to find yourself in.
localhost login: root
Password:
Login incorrect
localhost login: _Turn to Google and search for reset VxRail Manager password and come upon the following Dell kb, Dell VxRail: VxRail Manager root password is lost.(https://www.dell.com/support/kbdoc/en-us/000064579/vxrail-how-to-reset-the-root-password-for-vxrail-manager)
Even after following this article, you realize almost immediately that this hasn’t been updated to reflect newer versions. The very first picture depicts a SUSE Linux Enterprise 12 screenshot.
More recent versions of VxRail are running on SUSE Linux Enterprise 15. The following procedure will hopefully assist you until Dell can update their documentation.
Start by taking a snapshot of your VxRail Manager!
Open a web or remote console and then restart the virtual machine. When you see the following splash screen, press the ‘e’ key on your keyboard to interrupt the boot sequence.
This will bring you to the GNU GRUB boot menu. Look for the line starting with linux (14 lines down in my case). Press Ctrl-e to go to the end of the line and add init=/bin/bash.
Press Ctrl-x or F10 to boot.
If you are following the numerous sources out there, they will point you to use the pam_tally2 utility. As you can see below, this won’t work…yet.
Create the log directory and change to it with the following:
mkdir -p /var/log
cd /var/logAdd the tallylog file.
touch tallylog
chmod 600 tallylogNow, you should be able to use /sbin/pam_tally2. If you are not familiar with the syntax, the below images should help. The full help is at the end of this post for more information. You can see that the two users have 0 failures currently. If you do know the password and just want to unlock the account so you can log in again, use the following syntax.
/sbin/pam_tally2 -u <user name> -r
As long as the directory was created above, the passwd utility should be able to be used, now. Ignore the message that the password was used already. I tried completely new passwords and still receive this message.
Unfortunately, I have not found a suitable way to reboot the virtual machine, yet. VMware Tools (more accurately, open-vm-tools) is not started since we are not booted in a full multi-user state.
Make sure you have completed your password or unlocking maintenance. When you are ready, go to the power control for the virtual machine and select the Power off option. Wait a moment before powering the virtual machine back on. At this point, the passwords you set or the account should be unlocked.
After you have verified that your accounts work, be sure to remove the snapshot you took in the beginning!
Hopefully this will help you out.
/sbin/pam_tally2: [-f rooted-filename] [--file rooted-filename]
[-u username] [--user username]
[-r] [--reset[=n]] [--quiet]Quick Method: If running on a modern Windows 10, Windows 11, or Windows Server, check that PNPUTIL supports the /enum-devices and /remove-device command. I found that Windows Server 2019 did not support the /enum-devices and therefore did not have a /remove-device command.
If it does, however, the following one liner should work.
foreach ($dev in (Get-PnpDevice -Status UNKNOWN)) { &"pnputil" /remove-device $dev.InstanceId }
If you are on a system that lacks the listed switches in PNPUTIL or you just want the easy method, then you can try using DevManView by NirSoft. This utility allows you to select more than one device at a time as well which is convenitent if there are hundreds of “ghost” devices showing up. (https://www.nirsoft.net/utils/device_manager_view.html)
Have you ever set up VMware App Volumes and found that your virtualized applications didn’t appear to be attached to your virtual desktops? You start out with your basic troubleshooting, beginning with the lower layers of the OSI (Open Systems Interconnection) model. Are the servers hosting the App Volumes Managers online? Is the virtual machine that’s connected visible to App Volumes Manager? Is the service (svservice) running on the virtual machine? Was App Volumes Agent the very last agent installed as per VMware KB 2118048? Can I send and receive basic ICMP (Internet Control Message Protocol) requests from the virtual machine to the manager? Maybe there is a firewall blocking the dataflow? Huh, everything seems to be configured correctly, what could it possibly be?
Finding “Ghost” Devices with PowerShell
Get-PnpDevice -Status UNKNOWN
Look for Disconnected Virtual Disks
Suddenly, while frantically trying to figure out what is going on, you notice that there are virtual disks attached to the virtual machine. Well, now that’s interesting…let us check the Disk Management utility (diskmgmt.msc). Sure enough, there are virtual disks attached, but they are in an offline state. Huh, well that’s odd. We can’t just online them because these are Instant Clones, and that metadata will not persist.
Open Device Manager (devmgmt.msc) to look at the hardware, enabling the Show hidden devices option from the View menu. Let’s poke around a little bit by expanding each device type. Sure enough, there are disk drives, storage controllers, and other devices that appear to be disconnected. They appear as slightly transparent icons and are sometimes referred to as ghosted devices.
The Golden Image should be opened, and all the disconnected devices should be flushed out. Removing all disconnected devices ensures that your Golden Image will be clean and only contain hardware that exists. When the desktop pool is published and an end-user attaches a new device, Windows Plug and Play (PnP) will adapt to the hardware changes with minimal intervention. Refresh the desktop pool and this time when we log on, we have success! The virtualized applications are attached, visible, and function as expected. Open Device Manager and notice there are no disconnected devices. Open the Disk Management utility (diskmgmt.msc) and notice the disks are now online.
Automate The Solution
So now we have a root cause, the effect, and a working solution. However, the solution is very manual, can be painful, and may be error prone. Anytime a human gets in front of a computer, things are bound to go awry. Let’s automate a solution, after all, the computer should be working for us, not the other way around.
IF NOT EXIST C:\TempWork (MKDIR C:\TempWork)
First, let’s make a temporary directory to do our work.
PNPUTIL /enum-devices /disconnected > C:\TempWork\disconnected-devices.txt
Second, let’s get a list of the devices that are disconnected. This will print the disconnected devices into a text file. Unfortunately, there is too much information in this, so we will pare it down to what we do need.
FINDSTR /C:Instance "C:\TempWork\disconnected-devices.txt" > C:\TempWork\devices-to-delete.txt
Third, let us get just the lines that have the Instance ID that we need and store that in another file. Great! So now we have a file of zero, one, or more instance identifiers. Unfortunately, there is still more work to do. Don’t fret, though, the challenging work is almost done and certainly worth the reward, here. If you open this file, you will see each line begins with Instance ID:, a long space, and then the actual information we need to remove the device. The Windows Command Line utility, FOR, will treat each non-space word as a token. So, the first token is the word Instance, the second is ID:, and the third is what we are looking for.
FOR /F "tokens=3" %%G IN (C:\TempWork\devices-to-delete.txt) DO (PNPUTIL /remove-device %%G)
All we need to do from here is iterate through the file, acting upon each Instance ID and removing the device.
@echo off IF NOT EXIST C:\TempWork (MKDIR C:\TempWork) PNPUTIL /enum-devices /disconnected > C:\TempWork\disconnected-devices.txt FINDSTR /C:Instance "C:\TempWork\disconnected-devices.txt" > C:\TempWork\devices-to-delete.txt FOR /F "tokens=3" %%G IN (C:\TempWork\devices-to-delete.txt) DO (PNPUTIL /remove-device %%G) PNPUTIL /enum-devices /disconnected RD /S /Q C:\TempWork
We can put all these statements into a batch file and then all we need is to run it from an elevated command prompt. The work is done instantly, and we can be assured that we did not miss a device or accidentally remove a necessary connected device. If you desire to keep the files for a record of what was detected and removed, just comment out line 11 with REM.
When to run this Batch File
As a best practice, perform this task after removing unnecessary devices from the virtual machine, such as hard disks and CD/DVD devices, running the VMware OS Optimization tool, but before sealing the Golden Image. This way, you can be assured that the virtual machine’s hardware reflects what is actually installed and end-users will get the best experience possible.
This occurred for me when upgrading to or installing a new vCenter 7 and replacing the self-signed certificate. I tested in a lab and was able to successfully install both vCenter 7 and Orchestrator 8.3. I was able to successfully configure both appliances and log in, as well. I did use vSphere Authentication as the Orchestrator’s Identity Provider. As soon as I replaced the self-signed certificate on vCenter, I immediately received the following when logging into Orchestrator:
Uh-oh! So after two weeks or so and lot’s of doing this and that and trying this and that, I think I finally found the resolution. This is actually in the VMware documents, but the document is not quite complete with the information needed to successfully run the commands. Here is the document, https://docs.vmware.com/en/vRealize-Orchestrator/8.3/com.vmware.vrealize.orchestrator-install-config.doc/GUID-66B37DF2-052E-44A0-929E-E4F53E1BCCE3.html. I have detailed the process in full later in this blog post.
For completeness sake, I am going to show the entire process. Please feel free to scroll to the interesting sections below to resolve. I am not going to show how to deploy the appliances, just that they will be in vSphere and available as a starting point.
Installed, configured, and checking the services for a “known good”.
When I navigate to my vCenter appliance, I can see that it is using an untrusted certificate.
I perform the necessary steps to continue on. Your browser may be different and your organization’s policies may be different. If your organization is using HTTP Strict Transport Security (HSTS), you will likely be unable to continue without some very tricky manipulation or replacing the self-signed certificate to a known and trusted certificate. This is likely how or why you are in this predicament in the first place and had to search for this blog post.
The log in window is presented to me.
I verified I was able to successfully log in.
Navigate to the Orchestrator 8.3 appliance, I am presented with the following.
Since this appliance is fresh, I need to click on the Start the Control Center link and establish an authentication provider. I have to log in with the root account.
Click on Configure Authentication Provider
On this page, I chose vSphere for the Authentication mode setting and the Host address is my vCenter 7 appliance. I am presented with an Accept Certificate box. This will accept the current self-signed certificate, since that is all that is available. NOTE: You could wait to do this step until after you alter the TLS certificate on vCenter, but this article assumes you did not or that you already had an Orchestrator appliance deployed like I did.
Complete the Identity Service window with an administrative or service account that allows users to be queried. Click Register.
Type in a group to use as an Admin group, I used admins, then click the Search button.
A window will display that allows you to pick a security group based off your search criteria. Click Save Changes.
The Orchestrator appliance will be configuring in the background. This is not a fast process! Click on the home icon and choose Validate Configuration. You will see a message stating that a server restart is required…This will automatically happen after a two minute wait. Please be patient here…
You can continue clicking the Refresh button until you have all green check marks. This signifies the appliance rebooted and all services are back up.
Go back to the vco tab in the browser and choose the START THE ORCHESTRATOR CLIENT link. You should be presented the VMware vSphere log on screen. This signifies that your authentication provider is set up correctly to use vSphere. Try logging in.
I can verify that I can successfully log in without trouble.
Log in to vCenter server if you are not already. Lot’s of assumptions in the next few sections…I am going to assume you are logged in with an administrative user that also can perform cryptographic operations (https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-17568345-E59E-43A8-A811-92F8BE9C7719.html), then navigate to Menu > Administration > Certificate Management.
I am going to assume you know how to request a Certificate Signing Request (CSR), have already had the certificate signed, and have the necessary certificates in possession. If not, here is a VMware resource to get you started: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-E0609A99-A8D1-4336-BD3B-DE707E261A63.html. Under Machine SSL Certificate, click on Actions > Import and Replace Certificate.
In my case, I chose to use Replace with external CA certificate(requires private key) option and clicked the Next button.
Add in the machine certificate, the root and intermediate certificates chained together (for the chain, I always start with the root and add the intermediate certificates below), and the private key file. Click the Replace button.
If the certificates are successfully replaced, you should get logged out relatively quickly. Give this a few minutes as the vCenter server services are rebooting in the background.
Click the Login button after a few minutes. You may get a no healthy upstream message. Be patient and refresh your browser periodically, there is a lot going on with these appliances.
Eventually, you should get the vCenter log in screen. You can verify that your vCenter is secured by looking for the lock symbol. Go ahead and log back in to verify you can.
Click the Launch vSphere Client (HTML 5) button. Enter your credentials.
Click the Login button.
You may receive a Error occurred while fetching vmca root cert: com.vmware.vcenter.certificate_authority.get_root message. This just indicates that the vCenter server services are not fully restarted, yet. There may be a running task in Recent Tasks. Once this is complete, the message will go away after you refresh the User Interface (UI) or the browser window. Again, be patient as this may seem like an eternity in computer time or that it is broken, but it should come back up.
Again, be patient as this may seem like an eternity in computer time or that it is broken, but it should come back up.
We can confirm that we have logged in and that the message went away.
Ok, let’s try to log into the Orchestrator appliance.
So far, it looks promising. Click on the Start the Orchestrator Client link. Warning: you may actually get logged in. This is most likely due to a cookie on your browser. If you close your browser and try to log in again, you will most likely not be able to log in. That is what we are going to fix.
Enter your credentials and click the Login button.
Et voilà! There we are for us English speakers, the broken UI that is extremely frustrating to fix.
Here is the article from VMware on how to solve this (https://docs.vmware.com/en/vRealize-Orchestrator/8.3/com.vmware.vrealize.orchestrator-install-config.doc/GUID-66B37DF2-052E-44A0-929E-E4F53E1BCCE3.html). Unfortunately, not all the details are there to run the commands and if you are not experienced with the underlying technology of the Orchestrator appliance, like I wasn’t and really still am not, then this will just likely frustrate you even further. Let’s break this down…I added an indicator where I added steps to the original documented procedure.
1. Log in to the vRealize Orchestrator command line as root. (Added) I used an SSH session, but you can do this on the console with VMRC. I just wanted to be able to copy and paste commands.
2. (Added) Obtain the name of the <vRO pod> you will need for the next step.
kubectl -n prelude get pods
3. Run the kubectl -n prelude exec command. (Added) I used the last line from the clue in the example command of vco-server-app. I really did not know and the document does not explain.
Command from document.
kubectl -n prelude exec -it <vRO pod> -c vco-server-app -- bash
Command used with the <vRO pod> substituted.
kubectl -n prelude exec -it vco-app-77c8fb6659-fsr5v -c vco-server-app -- bash
4. Run the rpm command.
rpm -hiv --nodeps /vco-cfg-cli.rpm
5. Navigate to the /usr/lib/vco-cli/bin/ directory.
6. Run the following ./vro-configure-inner.sh trust commands.
From the document.
./vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri <vSphere-Auth-Provider-URI> --accept
With substituted <vSphere-Auth-Provider-URI>
./vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri vcsa70.aaronrombaut.com --accept
A lot of information will scroll past. I am only including a screenshot of the end of the command.
From the document.
./vro-configure-inner.sh trust --alias vco.sso.ssl.certificate --uri <vSphere-Auth-Provider-URI> --accept
With substituted <vSphere-Auth-Provider-URI>
./vro-configure-inner.sh trust --alias vco.sso.ssl.certificate --uri vcsa70.aaronrombaut.com --accept
A lot of information will scroll past. I am only including a screenshot of the end of the command, again.
7. Log out of the vRealize Orchestrator Appliance by using the exit command and log in again. (Added) If you only type exit here once, you will only exit the rpm command. You actually have to end the SSH session or console. You can type exit a second time to close the SSH session.
8. Run the following deploy.sh commands.
/opt/scripts/deploy.sh --onlyClean
A lot of information will scroll past. I am only including a screenshot of the end of the command. This command will take a few minutes to complete.
/opt/scripts/deploy.sh
A lot of information will scroll past. This command will take even longer to complete than the last command. Notice: if you prematurely end this command, your appliance will likely not be recoverable. Trust me when I tell you this. Learn from my pain…
You may even see messages that state Exit code and + return 0 like the screenshot below.
This is not complete, yet. Keep waiting until you see the following screen. (If you are nervous or impatient, get up and take a walk, this seriously takes a really long time, the appliance is going through a restart as part of this process).
Navigate to the appliance. Click on the Start the Orchestrator Client link to log on.
Type in your credentials and click the Login button.
Assuming everything went well, you should now be able to log back into the VMware vRealize Orchestrator appliance without error.
Please let me know if this helped you or if something I typed did not line up with what you experienced.
If you work in IT in the DoD in any capacity, then you know your systems can be a pain to work with if you followed a Security Technical Implementation Guide (STIG). This can be even more of a pain when following a commercial vendor’s installation or configuration documentation, since they write in the general sense and can’t possibly know every quirk in our environment.
From the Windows Server 2016 STIG, (Finding ID: V-73691) the LAN Manager authentication level must be set to send NTLMv2 response only and to refuse LM and NTLM.
From the VMware vSphere 6.5 ESXi STIG, (Finding ID: V-94023) the ESXi host must use the vSphere Authentication Proxy to protect passwords when adding ESXi hosts to Active Directory.
So even though the current STIG is only referencing vSphere 6.5, this can still be used for newer versions. During the configuration of a vCenter Server 7.0 Authentication Proxy, I found that the proxy kept not authenticating. After reviewing the logs on the Domain Controller, it was found that Likewise was sending the authentication to the Domain Controller with an NTLM response and not NTLMv2.
Reconfigure the Likewise service to send NTLMv2 and try to configure the Authentication Proxy again. This time, the configuration should be successful. You can further test by adding an ESXi host to vCenter and ensure it properly joins the domain.
Fill in the Domain, Domain User, and Domain Password settings. Click the SAVE button.
Notice the Operation failed! message.
You can also view the log located at /var/log/vmware/vmcam/vmcam-sca.log. Below is an excerpt, notice the Enter password for Service.vSphereAuth line. That indicates that it was not sending the correct password because we know that we typed it into the window.
I ended up tailing the log with the following so I could see in real-time what was occurring.
tail -f /var/log/vmware/vmcam/vmcam-sca.log
Run the following bold commands, the rest is for context.
/opt/likewise/bin/lwregshell
\> cd HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> list_values
"SendNTLMv2" REG_DWORD 0x00000000 (0)
"Support128bit" REG_DWORD 0x00000001 (1)
"Support56bit" REG_DWORD 0x00000001 (1)
"SupportKeyExchange" REG_DWORD 0x00000001 (1)
"SupportNTLM2SessionSecurity" REG_DWORD 0x00000001 (1)
"SupportUnicode" REG_DWORD 0x00000001 (1)
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> set_value SendNTLMv2 1
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> list_values
+ "SendNTLMv2" REG_DWORD 0x00000001 (1)
"Support128bit" REG_DWORD 0x00000001 (1)
"Support56bit" REG_DWORD 0x00000001 (1)
"SupportKeyExchange" REG_DWORD 0x00000001 (1)
"SupportNTLM2SessionSecurity" REG_DWORD 0x00000001 (1)
"SupportUnicode" REG_DWORD 0x00000001 (1)
HKEY_THIS_MACHINE\Services\lsass\Parameters\NTLM> quit
/opt/likewise/bin/lwsm restart lwio
Stopping service reverse dependency: lsass
Stopping service reverse dependency: rdr
Stopping service: lwio
Starting service: lwio
Starting service reverse dependency: rdr
Starting service reverse dependency: lsass
Go back to vSphere Client and configure the Authentication Proxy again. You may have to disable and enable the service again if you didn’t restart the service while connected through SSH. This time, it should work without an error. If you are tailing the log, then you should see Active Directory domain added successfully at the bottom.
From here you can Import the vSphere Authentication Proxy Certificate to ESXi Host.