I am starting with a fresh install of Red Hat Enterprise Linux 8 (RHEL8) as a VMware Fusion virtual machine (vm) on my MacBook Pro. I also have an ESXi 7.0 virtual machine to connect to.
I have set up a few resources on the virtual machine that I find helpful. The very first is updating the virtual machine until there are no more updates. I also installed Visual Studio Code.
The first thing I did was create an Ansible user name to run Ansible on the Red Hat vm. This is going to be my control computer.
The allowed characters to create a user account in RHEL8 are:
‘a’ … ‘z’
‘A’ … ‘Z’
‘0’ … ‘9’
‘.’ (Period)
‘-‘ (hyphen)
‘_’ (underscore)
Ref: https://access.redhat.com/solutions/30164 (requires an account to access)
The next thing to do is create an account on an ESXi host that Ansible will use to connect to. The root account will not be able to log on since the SSH daemon is configured with PermitRootLogin set to no. You can verify this by viewing the /etc/ssh/sshd_config file.
The allowed characters for a user account for ESXi 7 are not documented, officially, that I could find. When I logged on and tried to a create a user named svc.ansible, I was provided a warning message that only alpha-numeric characters are allowed. Apparently a user name consisting of a hyphen works, though. So I can use svc-ansible, but not svc.ansible.
If you are like me and like things neat and consistent, you won’t want an account on RHEL8 formatted differently than on ESXi. svc.ansible and svc-ansible I will have to decide on a format that allows me consistency across systems.
I recently started experimenting with Ansible. I am still learning the mechanics of how it works and if it can actually be benefiting. The motivation behind this is in learning how to more effectively perform certain tasks using Ansible. In particular, for me, I want to quickly, accurately, and efficiently apply DISA STIG to VMware software and appliances.
DISA is an acronym for the Defense Information Systems Agency. It is a US Department of Defense combat support agency. STIG is a Security Technical Implementation Guide. STIGs are comprised of one or more documents that contain checks and fixes for security vulnerabilities and general hardening guidelines for various computer technologies.
After a little experimenting, though, I quickly realized a few things. Getting Ansible to connect to VMware ESXi is not as easy as I thought. ESXi is not Linux, so it doesn’t just work out of the box. ESXi is ESXi and Linux is Linux.
I will be creating a few separate pages to document this process. Hopefully others who are exploring this option can use these resources as a starting point. A couple resources I have used to start the journey include the following:
You think it would be easier to add a consent banner. You probably already looked under the Administration Console and Appliance Settings. Unfortunately, this setting is not easy to find and there is no global consent banner.
Start by navigating to the Administration Console of the Workspace ONE Access appliance.
Click on the Identity & Access Management tab, then click Setup.
Click on the worker link of the Connector.
Change the selection to Auth Adapters
Click the CertificateAuthAdapter link. You are redirected to the Authentication Adapter page. Note: it may take a few moments to load. If you see just a name text box, refresh the browser page.
Check the Enable Certificate Adapter check box to enable the certificate adapter.
Check the Enable Consent Form before Authentication checkbox to enable the consent form before authentication.
Add the Consent Form Content text to the text box.
Click Save.
Navigate to Identity & Access Management > Manage > Policies.
Click on the Policy Name in use.
Click Edit.
Click on Configuration.
In my case, I am only concerned with the Web Browser Device Type. Click All Ranges link on the line for Web Browser. Rearrange the policy rules as necessary. Make sure the then the user may authenticate using is set to Certificate even if smart cards are not being used.
Click SAVE, NEXT, and SAVE again.
Log off or open a new browser window and verify you receive the Consent Bannerbeforeauthentication takes place.
When logging on to Workspace ONE Access and viewing the catalog, the links that are presented can be just bookmarks, or they can be bookmarks that also sign you in to the service you are navigating to. One such case is trying to log onto other vRealize products. Unfortunately, I am not aware of an easy button to accomplish this. I was also unable to find any documentation on www.vmware.com that publishes an SSO URL.
I did, however, find a couple websites to assist. The following links will take you there.
How to add vRealize Operations Manager 8.x to vRealize Identity Manager Web Apps catalog (https://blanketvm.com/2021/04/14/vrops-vridm-webapps-catalog/)
Integrating VMware Log Insight with Workspace ONE Access for SSO (http://www.techcurmudgeons.com/post/integrating-vmware-vrealize-log-insight-with-workspace-one-access-for-sso)
First Things First
Make sure the Workspace ONE Access appliance has a directory configured. There should ideally be groups synced that contain administrative users.
Ensure you have configured VMware Identity Manager (AKA Workspace ONE Access) for both services.
Next, configure the services to use the administrative groups from Workspace ONE Access and configure appropriate roles.
vRealize Operations Administration > Access > Access Control > User Groups
vRealize Log Insight Administration > Management > Access Control > Directory Groups
Adding vRealize Operations to the Catalog
Log on to the Workspace ONE Access appliance as an admin or switch to the Administration Console.
Choose Catalog > Web Apps
Click NEW
Provide a Name such as vRealize Operations
Click NEXT
Change Authentication Type to Web Application Link and provide a Target URL.
The target URL can be obtained by navigating to the log on page for the vRealize Operations appliance ( you may need to open a new browser or private window for a new session). Don’t log on, though.
View the source of the page and search for getVidmRedirectUrl
Append the value of url to the log on URL so it is similar to the following:
Copy the value of the vidmRedirectURL, this is the Target URL
Click NEXT and verify the New SaaS Application details.
Click SAVE & ASSIGN. Type in Users / User Groups as appropriate.
Click SAVE
Use another browser, private window, or switch to User Portal and refresh the browser. Test logging into Workspace ONE Access by clicking CATALOG and clicking on the Catalog Item you created above. Verify that you successfully are logged on to the vRealize Operations appliance.
Locate the vRealize Log Insight Client ID
From the vRealize Log Insight appliance, once the vRealize Log Insight Authentication Configuration makes a successful connection to the Workspace ONE Access appliance, a client ID will be created.
On the Workspace ONE Access appliance, click the arrow on Catalog and choose Settings.
Under Global Catalog Settings, click Remote App Access.
In the table, under the Client ID table heading, click on the first entry.
Verify the Redirect URI matches the IP address or fully qualified domain name of the vRealize Log Insight Appliance. If this is the case, copy the Client ID. If not, click Back To Clients List and continue clicking on Client ID entries until you find the correct client configuration.
Building the vRealize Log Insight Target URL
The target URL consists of the Workspace ONE Access fully qualified domain name, the client ID, and the vRealize Log Insight fully qualified domain name. In my case, here are the details:
Workspace ONE Access FQDN: vidm.aaronrombaut.com Client ID: 6d95cdc5-60f0-42b6-9f85-815b15b64aa7 vRealize Log Insight FQDN (must match the Redirect URL Host entry from the appliance authentication configuration): 192.168.92.33
Test this out by copying and pasting the configured target url in a new browser window. If successful, you should be redirected to Workspace ONE Access if you are not currently logged in or just taken directly to the vRealize Log Insight appliance.
Adding vRealize Log Insight to the Catalog
Log on to the Workspace ONE Access appliance as an admin or switch to the Administration Console. Choose Catalog > Web Apps
Click NEW
Provide a name such as vRealize Log Insight.
Click NEXT. Change Authentication Type to Web Application Link and provide the Target URL that was built in the Building the vRealize Log Insight Target URL task above.
Click NEXT. Review the New SaaS Application summary.
Click SAVE & ASSIGN. Type in Users / User Groups as appropriate.
Click SAVE.
Conclusion
It would be nice if there was better documentation on this or even just a catalog with common VMware software so that these catalog items could be built easier.
I deployed three new appliances and when I tried to run Chef InSpec against an appliance, I realized I had forgotten to initially set the root password from the console. The result was that the root account locked. I found a VMware knowledge base article to assist, but a few settings did not work for me. This page should be a one-stop resolution to unlock the root account.
Boot into Single User Mode
(Recommended) Take a snapshot of all the appliances in the cluster.
Log in to the VMware vRealize Operations administration interface (admin UI) using the following address
https://hostname.domain-name.tld/admin
Select the node with the locked root account and click Take Node Offline/Online.
If the cluster is not running in High Availability (HA) mode, the cluster will need to be taken offline.
Provide a reason, if desired.
Open a web or remote console of the node.
With the console open, restart the virtual machine.
When the following Photon OS splash screen is visible, press the letter e to open the GNU GRUB menu.
When the GNU GRUB menu appears, arrow down to the line that begins with linux /$photon_linux and press Ctrl + e to go the end of the line.
Add a space and type the following:
rw init=/bin/bash
Press F10
Method 1 – Reset the root account — no new password
If you know the current password or do not want to set a new password due to password rotation limits, try using the following method:
pam_tally2 --user root --reset
Unless the directory has been created at some point, you may receive the following error and the account will not be unlocked.
pam_tally2: Couldn't create /var/log/tallylog: No such file or directory
pam_tally2: Authentication error
List the directory contents to see that the /var/log is actually a link to /storage/log/var/log. This however is a broken link as the /var/log directory does not exist in /storage/log.
Add the missing directory to /storage/log with the following command:
su -
mkdir --parents /storage/log/var/log
Now, when listing the directories, we can see the missing directories exist. All that is needed is to create the tallylog file. Run the following command to create the tallylog file.
touch /storage/log/var/log/tallylog
Run the pam_tally2 command again.
pam_tally2 --user root --reset
As you can see from the screenshot, the Failures column displays a 0, indicating the account is unlocked.
Reboot the appliance using the following commands:
umount /
reboot -f
Bring the Cluster Online
Refresh the browser or log in to the vRealize Operations admin UI and bring the cluster online.
Conclusion
You should be able to log into the console, establish SSH, or continue use of your automation tool. Just be sure that if you do change the password, you update the password in your automation tool.
The policy files required are vdm_blast.admx and vdm_blast.adml. They are downloaded as part of the VMware-Horizon-Extras-Bundle.
In a typical environment, the admx policy files go into \\<domain name>\SYSVOL\<domain name>\Policies\PolicyDefinitions. The adml files go into the language folder, in my case, en-US. If the Group Policy Editor was already open, close and reopen for the changes to take effect.
Quick Method: If running on a modern Windows 10, Windows 11, or Windows Server, check that PNPUTIL supports the /enum-devices and /remove-device command. I found that Windows Server 2019 did not support the /enum-devices and therefore did not have a /remove-device command.
If it does, however, the following one liner should work.
If you are on a system that lacks the listed switches in PNPUTIL or you just want the easy method, then you can try using DevManView by NirSoft. This utility allows you to select more than one device at a time as well which is convenitent if there are hundreds of “ghost” devices showing up. (https://www.nirsoft.net/utils/device_manager_view.html)
Have you ever set up VMware App Volumes and found that your virtualized applications didn’t appear to be attached to your virtual desktops? You start out with your basic troubleshooting, beginning with the lower layers of the OSI (Open Systems Interconnection) model. Are the servers hosting the App Volumes Managers online? Is the virtual machine that’s connected visible to App Volumes Manager? Is the service (svservice) running on the virtual machine? Was App Volumes Agent the very last agent installed as per VMware KB 2118048? Can I send and receive basic ICMP (Internet Control Message Protocol) requests from the virtual machine to the manager? Maybe there is a firewall blocking the dataflow? Huh, everything seems to be configured correctly, what could it possibly be?
Finding “Ghost” Devices with PowerShell
Get-PnpDevice -Status UNKNOWN
Look for Disconnected Virtual Disks
Suddenly, while frantically trying to figure out what is going on, you notice that there are virtual disks attached to the virtual machine. Well, now that’s interesting…let us check the Disk Management utility (diskmgmt.msc). Sure enough, there are virtual disks attached, but they are in an offline state. Huh, well that’s odd. We can’t just online them because these are Instant Clones, and that metadata will not persist.
Open Device Manager (devmgmt.msc) to look at the hardware, enabling the Show hidden devices option from the View menu. Let’s poke around a little bit by expanding each device type. Sure enough, there are disk drives, storage controllers, and other devices that appear to be disconnected. They appear as slightly transparent icons and are sometimes referred to as ghosted devices.
The Golden Image should be opened, and all the disconnected devices should be flushed out. Removing all disconnected devices ensures that your Golden Image will be clean and only contain hardware that exists. When the desktop pool is published and an end-user attaches a new device, Windows Plug and Play (PnP) will adapt to the hardware changes with minimal intervention. Refresh the desktop pool and this time when we log on, we have success! The virtualized applications are attached, visible, and function as expected. Open Device Manager and notice there are no disconnected devices. Open the Disk Management utility (diskmgmt.msc) and notice the disks are now online.
Automate The Solution
So now we have a root cause, the effect, and a working solution. However, the solution is very manual, can be painful, and may be error prone. Anytime a human gets in front of a computer, things are bound to go awry. Let’s automate a solution, after all, the computer should be working for us, not the other way around.
IF NOT EXIST C:\TempWork (MKDIR C:\TempWork)
First, let’s make a temporary directory to do our work.
Second, let’s get a list of the devices that are disconnected. This will print the disconnected devices into a text file. Unfortunately, there is too much information in this, so we will pare it down to what we do need.
Third, let us get just the lines that have the Instance ID that we need and store that in another file. Great! So now we have a file of zero, one, or more instance identifiers. Unfortunately, there is still more work to do. Don’t fret, though, the challenging work is almost done and certainly worth the reward, here. If you open this file, you will see each line begins with Instance ID:, a long space, and then the actual information we need to remove the device. The Windows Command Line utility, FOR, will treat each non-space word as a token. So, the first token is the word Instance, the second is ID:, and the third is what we are looking for.
FOR /F "tokens=3" %%G IN (C:\TempWork\devices-to-delete.txt) DO (PNPUTIL /remove-device %%G)
All we need to do from here is iterate through the file, acting upon each Instance ID and removing the device.
@echo off
IF NOT EXIST C:\TempWork (MKDIR C:\TempWork)
PNPUTIL /enum-devices /disconnected > C:\TempWork\disconnected-devices.txt
FINDSTR /C:Instance "C:\TempWork\disconnected-devices.txt" > C:\TempWork\devices-to-delete.txt
FOR /F "tokens=3" %%G IN (C:\TempWork\devices-to-delete.txt) DO (PNPUTIL /remove-device %%G)
PNPUTIL /enum-devices /disconnected
RD /S /Q C:\TempWork
We can put all these statements into a batch file and then all we need is to run it from an elevated command prompt. The work is done instantly, and we can be assured that we did not miss a device or accidentally remove a necessary connected device. If you desire to keep the files for a record of what was detected and removed, just comment out line 11 with REM.
When to run this Batch File
As a best practice, perform this task after removing unnecessary devices from the virtual machine, such as hard disks and CD/DVD devices, running the VMware OS Optimization tool, but before sealing the Golden Image. This way, you can be assured that the virtual machine’s hardware reflects what is actually installed and end-users will get the best experience possible.
