Prepare Ubuntu 18.04 for Desktop Anywhere

This one was a little tricky to get set up. I am happy to say that after you follow this procedure, you will be able to access AFRC Desktop Anywhere from Ubuntu 18.04!

As always, I like to provide transparency. I am a traditional reservist in the Air Force Reserve and I also work for VMware. Anything provided here is from me and only me. Nothing here represents official notification from the USAF or VMware. Follow at your own risk. Your system and set up are most likely not identical to mine. I am using a VMware virtual machine for this guide but that should not matter for the purposes of this guide.

With that boring stuff out of the way…

Make sure your system is up-to-date.

Download Software and Packages

  • Open Terminal and type in the following:
sudo apt install opensc opensc-pkcs11 pcsc-tools

Install the DoD Certificates into Firefox Certificate Manager

Open Firefox and select the three lines at the top-right of the window, then click on Preferences. In the Find in Preferences search box, type in Certificates. Click on the View Certificates… button.

In Certificate Manager, make sure you are on the Authorities tab and click the Import… button.

Navigate to where you downloaded and extracted the DoD certificates from earlier. Select the Certificates_PKCS7_v5.6_DoD.der.p7b file. Check the two boxes to trust the certificate. You will need to do this for each certificate you need to use. I recommend starting with DoD Root CA 2, DoD Root CA 3, DoD Root CA 4, and DoD Root CA 5. The certificates will be added under the U.S. Government heading.

Add DoD Certificates to the Ubuntu Certificate Trust

While you are here, be sure to choose the DoD Root CA 3 certificate, and then press the Export… button. Make sure the X.509 Certificate (PEM) is selected at the bottom of the window. Also do this for the DOD SW CA-53 certificate necessary for Desktop Anywhere. When you are done exporting the certificates, you can close the Certificate Manager window.

Open Terminal and copy the two files to /usr/share/ca-certificates. Use the following image if you are unsure how to accomplish this.

Now we want to reconfigure the certificate store by typing in sudo dpkg-reconfigure ca-certificates. This will open the ca-certificates configuration window. Choose Ask, then press tab to select <OK>.

Press the Space bar on the top two DoD certificate and press tab to select <OK>.

When you select <OK>, the Terminal will display the operation of adding the chosen certificates to the certificate store. Make sure the number of certificates you selected are added. In this case, there should be two.

Verify the Smart Card Reader is Accessible to the System

On the Terminal, type pcsc_scan. For the purposes of demonstration in this guide, I am going to post an image of before I connected my Smart Card reader so you can see the difference.

And then, here is an image after I connected my Smart Card reader.

Installing VMware Horizon Client for Linux

On the Terminal, navigate to your downloaded file. Add the execution bit to the bundle by typing sudo chmod u+x <filename>. See the image below for a visual.

Type sudo ./VMware-Horizon and press tab. The rest of the filename should have auto-completed. Press Enter to continue to installation. Read and accept the EULA.

Customize the installation as needed. I am going to accept the defaults.

Verify all the product install files are ready, and click the Install button.

Check the box to Register and start…, and click the Scan button.

I had a few failed results, but this should not affect my use at this time.

Go ahead and close the installer.

Configure the Smart Card Module to be Accessible for Horizon Client

Ref: https://docs.vmware.com/en/VMware-Horizon-Client-for-Linux/5.4/horizon-client-linux-installation/GUID-6CB5F6EE-E7DC-4BF7-8E2A-4542E4A78182.html

On the Terminal, make a directory in /usr/lib/vmware/view/pkcs11.

Now we are going to create a symbolic link to the OpenSC module. The VMware article helps with the syntax, but points to the incorrect location of the module. As long as you have been following along up to this point, you should not have any errors.

sudo ln -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so /usr/lib/vmware/view/pkcs11/libopenscpkcs11.so

Follow the image below for the syntax.

Verify Horizon Client for Linux can use your Smart Card

Hopefully you ran the steps above to verify that Ubuntu could see and access your Smart Card reader. This step will check the log and verify that it is working for the Horizon Client.

Open up the VMware Horizon Client and wait for it to show up. Then, on the Terminal, type less /tmp/vmware-<username>/<log-file-name>. For instance I typed:

less /tmp/vmware-arombaut/vmware-horizon-client-15386.log

Look through the log until you find Initializing smartcard modules. You should see it Loaded 1 modules from /usr/lib/vmware/view/pkcs11 and see one or more certificates presented to you. The certificates that have populated UPNs are what is going to be presented to you in the Horizon UI. This information is also seen from the line, certsList has 1 certificates.

You can close the log and the Terminal.

This actually completes the setup of VMware Horizon Client for Linux. From here you can add a Server and select the appropriate CAC certificate to authenticate.

Due to customer-sensitive information, I am not able to provide further information beyond this point here. I can help you individually if you need more assistance from here though.

[email protected]

VMware ESXi SSL Certificate Signing Request (CSR)

Need to replace the certificates with a custom certificate from a commercial or corporate Certificate Authority (CA)? You are going to need a Certificate Signing Request (CSR).

I use the certificate-manager tool (see below) included with vCenter to generate all of my requests. I found recently that you can skip this altogether if you can include the certificate’s key from the CA. In the traditional method, though, you will generate a CSR, submit to CA, and receive back a custom signed certificate.

/usr/lib/vmware-vmca/bin/certificate-manager
  • Connect to your Platform Services Controller (PSC) via SSH. This could be the same as your vCenter server.
  • Create a temporary directory to store your files in when you run the certificate-manager tool. Create a directory for each system you intend to generate CSR files for. I use the following format for my directory:
/tmp/hostname1
/tmp/hostname2
...
/tmp/hostnamen
  • Run the certificate-manager tool:
/usr/lib/vmware-vmca/bin/certificate-manager
  • Select option 1 to replace the Machine SSL.
  • Provide the administrator username if the SSO domain is not default or press Enter.
  • Provide the administrator password.
  • Select option 1 to Generate Certificate Signing Request(s)…
  • Provide an Output directory path such as
/tmp/hostname

The next information you will be requested to enter is for the certificate.

  • Country – Use two digit code (https://www.iso.org/obp/ui/#search)
  • Name – Use the fully qualified domain name of the server here
  • Organization – Name of your organization
  • OrgUnit – Name of your department
  • State – The state where the system resides (no abbreviations)
  • Locality – The city where the system resides
  • IPAddress – IP address of the system
  • Email – Email address for person or department responsible for administration of the system
  • Hostname – It’s best to always include a fully qualified domain name as well as a short name.
  • Proper value of VMCA ‘Name’ – I use the fully qualified domain name here

Press option 2 to Exit certificate-manager. Your csr and key files will be stored at the location you specified.

Apple Computer

If you use an Apple computer, you can use scp in the Terminal to copy the files.

$mkdir ~/Documents/Certificates/hostname/
$cd ~/Documents/Certificates/hostname/
$scp [email protected]:/tmp/hostname/*.* .

Windows Computer

If you use Windows, I recommend WinSCP. You will first need to set the shell on the Platform Services Controller to Bash.

chsh -s /bin/bash root

Follow the instructions of your CA on how to submit the CSR to be signed.

Dell EMC Avamar and VMware 6.5 Snapshot Quiescing Error

During a rebuild of a Dell EMC Avamar backup solution, we ran into an issue where VMware was logging, “An error occurred while quiescing the virtual machine. See the virtual machine’s event log for details.” for a few virtual machine backups. While testing, we noticed that it was not occurring on all backups.

On a functional backup with no errors, a machine had a service called VMware Snapshot Provider and this service was set to manual. On the machine I was troubleshooting, this service didn’t even show up in the list. I opted to uninstall tools, restart the server, and then reinstall VMware tools. Upon inspection of the services listing this time, the service was present and set to manual. I tested a backup and there were no failures and the machine backed up fine.

I tested another virtual machine that I knew previously worked and this time I went to services and set the VMware Snapshot Provider service to disabled and initiated a backup on the Avamar UI (AUI). As expected, VMware logged an event as noted above.

In conclusion, if you notice this error occurring, regardless if there were no errors previously, check for this service. The service can be reinstalled by invoking the VMware Tools installer again and changing the installed features. If all else fails, just remove, reboot, and re-install.

ESXi 6.5 STIG Welcome Message

So the ESXi 6.5 Security Technical Implementation Guide (STIG) finally released in May 2019. Unfortunately, some of the items I would have liked to see fixed or updated were not. One of the biggest pet peeves is the Annotations.WelcomeMessage advanced setting check. This setting updates the /etc/vmware/welcome file in the filesystem.

If the STIG fix text is implemented as stated, then you get an unsightly Direct Console User Interface (DCUI). The first screenshot below is what the text looks like if you copy and paste using vi on the terminal. The second screenshot shows the resulting DCUI.

There is a VMware Fling called DoD Security Technical Implementation Guide(STIG) ESXi VIB found at https://labs.vmware.com/flings/dod-security-technical-implementation-guide-stig-esxi-vib, that when implemented, results in a really nice looking DCUI window. The only thing about implementing this VIB is that it changes the permissions on the files in the filesystem, which I found terribly annoying. I wrote a post, http://box5304.temp.domains/~aaronrp2/vmware-esxi-6-5-stig-default-file-permissions/, that explains what the default permissions are supposed to be and how to change them back.

The first screenshot below shows what /etc/vmware/welcome looks like after installing the VIB. Clearly, the differences are already apparent. The second screenshot shows the resulting DCUI. It is a much more professional looking console to look at.

It would be nice to see the STIG ‘Fix Text’ get updated to reflect this better looking code at some point. The only thing I can think of that prevents this is the STIG Viewer collapses white space and that’s why we get what we get. I am including the better looking text in /etc/vmware/welcome below so that if you want to update your copy, you can. Unfortunately, I have been unable to figure out how to populate the Annotations.WelcomeMessage advanced setting in a Host Profile if you are wanting to implement this STIG at scale. Everytime I copy and paste, something gets messed up with line feeds and white space. The only way I can get the result I am looking for is to write to the /etc/vmware/welcome file directly or install the VIB.

{bgcolor:black} {/color}{align:left}{bgcolor:black}{color:yellow}{hostname} , {ip}{/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:black}{color:yellow}{esxproduct} {esxversion}{/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:black}{color:yellow}{memory} RAM{/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:black}{color:white}        {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only. By      {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  using this IS (which includes any device attached to this IS), you consent to the following conditions:                 {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited     {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law      {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          enforcement (LE), and counterintelligence (CI) investigations.                                                  {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       At any time, the USG may inspect and seize data stored on this IS.                                              {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       Communications using, or data stored on, this IS are not private, are subject to routine monitoring,            {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          interception, and search, and may be disclosed or used for any USG-authorized purpose.                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       This IS includes security measures (e.g., authentication and access controls) to protect USG interests--not     {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          for your personal benefit or privacy.                                                                           {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching    {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          or monitoring of the content of privileged communications, or work product, related to personal representation  {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work       {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          product are private and confidential. See User Agreement for details.                                           {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{align:left}{bgcolor:dark-grey}{color:white}   Accept Conditions and Customize System / View Logs{/align}{align:right} Accept Conditions and Shut Down/Restart  {bgcolor:black} {/color}{/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}

Making a REST API Request in vRealize Orchestrator

This was a tough nut to crack for me. I definitely don’t ever want to struggle through learning this topic again. I hope if you are reading this, I can save you a ton of lost time. I use Postman to scope out the concepts before trying to translate them in vRealize Orchestrator (vRO). When you see text surrounded by curly braces like, {{text}}, you can replace with your environment variables. If you are using Postman, then set this as an environment variable. It will save you a lot of typing in the long run.

First, here is the URL for the Swagger UI:
https://{{vRealize Server Fully Qualified Domain Name (FQDN)}}/vco/api/doc/index.html

Recover vCenter Appliance after Power/Storage Failure

Well, this would have saved me quite a bit of time in my past. My storage device got unplugged from the network while my lab was running and vCenter refused to come back. This is expected as the appliance has an embedded database.

I found this article from VMware that worked! It took less than 10 minutes.

https://kb.vmware.com/s/article/2149838

Use Shift + PgUp to scroll through the boot process and find out what filesystem, if any, is causing trouble.

Then type df -h to ensure none of the filesystems are full.

Compare /etc/fstab with the output from dh -f.

Once you have identified the missing or corrupt filesystem, run

e2fsck -y /dev/filesystem-that-needs-to-be-checked

Reboot the appliance and away you go!

reboot

ESXi Ruleset & Firewall Correlation

Working with VMware is usually a breeze. Unfortunately, this time I ran into an issue while setting up a new Host Profile, specifically the Ruleset Configuration for the Firewall, during the configuration for a DISA STIG. Almost all of the rules match a Firewall named rule but there are Firewall rules that do not have corresponding Ruleset names. I looked through VMware documentation, Reddit, and of course Google. I was unable to find any information where the two were together. So…I decided I would take on the task myself and of course share this with everyone.

VMware does provide an Incoming and Outgoing Firewall Ports for ESXi Hosts article (https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-171B99EA-15B3-4CC5-8B9A-577D8336FAA0.html) but the service is only listed, not the corresponding Ruleset Configuration name. This is what the following table will help fill-in.

ESXi Ruleset & Firewall Correlation Table
  • Default – The service is enabled upon initial install
  • Survive Reboot – The service will be enabled after rebooting the host
  • Ruleset Name – Host Profile > Security and Services > Firewall Configuration > Firewall Configuration > Ruleset Configuration
  • Ruleset Order – This is the order the rules are presented in the Host Profile
  • Firewall Name – The name from Configure > System > Security Profile > Firewall menu
  • In-Ord – This is the order of the incoming rules in the UI
  • In – These are the ingress TCP/UDP ports
  • Out-Ord – This is the order of the outgoing rules in the UI
  • Out – These are the egress TCP/UDP ports