Dell EMC Avamar and VMware 6.5 Snapshot Quiescing Error

During a rebuild of a Dell EMC Avamar backup solution, we ran into an issue where VMware was logging, “An error occurred while quiescing the virtual machine. See the virtual machine’s event log for details.” for a few virtual machine backups. While testing, we noticed that it was not occurring on all backups.

On a functional backup with no errors, a machine had a service called VMware Snapshot Provider and this service was set to manual. On the machine I was troubleshooting, this service didn’t even show up in the list. I opted to uninstall tools, restart the server, and then reinstall VMware tools. Upon inspection of the services listing this time, the service was present and set to manual. I tested a backup and there were no failures and the machine backed up fine.

I tested another virtual machine that I knew previously worked and this time I went to services and set the VMware Snapshot Provider service to disabled and initiated a backup on the Avamar UI (AUI). As expected, VMware logged an event as noted above.

In conclusion, if you notice this error occurring, regardless if there were no errors previously, check for this service. The service can be reinstalled by invoking the VMware Tools installer again and changing the installed features. If all else fails, just remove, reboot, and re-install.

ESXi 6.5 STIG Welcome Message

So the ESXi 6.5 Security Technical Implementation Guide (STIG) finally released in May 2019. Unfortunately, some of the items I would have liked to see fixed or updated were not. One of the biggest pet peeves is the Annotations.WelcomeMessage advanced setting check. This setting updates the /etc/vmware/welcome file in the filesystem.

If the STIG fix text is implemented as stated, then you get an unsightly Direct Console User Interface (DCUI). The first screenshot below is what the text looks like if you copy and paste using vi on the terminal. The second screenshot shows the resulting DCUI.

There is a VMware Fling called DoD Security Technical Implementation Guide(STIG) ESXi VIB found at https://labs.vmware.com/flings/dod-security-technical-implementation-guide-stig-esxi-vib, that when implemented, results in a really nice looking DCUI window. The only thing about implementing this VIB is that it changes the permissions on the files in the filesystem, which I found terribly annoying. I wrote a post, https://www.aaronrombaut.com/vmware-esxi-6-5-stig-default-file-permissions/, that explains what the default permissions are supposed to be and how to change them back.

The first screenshot below shows what /etc/vmware/welcome looks like after installing the VIB. Clearly, the differences are already apparent. The second screenshot shows the resulting DCUI. It is a much more professional looking console to look at.

It would be nice to see the STIG ‘Fix Text’ get updated to reflect this better looking code at some point. The only thing I can think of that prevents this is the STIG Viewer collapses white space and that’s why we get what we get. I am including the better looking text in /etc/vmware/welcome below so that if you want to update your copy, you can. Unfortunately, I have been unable to figure out how to populate the Annotations.WelcomeMessage advanced setting in a Host Profile if you are wanting to implement this STIG at scale. Everytime I copy and paste, something gets messed up with line feeds and white space. The only way I can get the result I am looking for is to write to the /etc/vmware/welcome file directly or install the VIB.

{bgcolor:black} {/color}{align:left}{bgcolor:black}{color:yellow}{hostname} , {ip}{/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:black}{color:yellow}{esxproduct} {esxversion}{/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:black}{color:yellow}{memory} RAM{/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:black}{color:white}        {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only. By      {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  using this IS (which includes any device attached to this IS), you consent to the following conditions:                 {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited     {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law      {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          enforcement (LE), and counterintelligence (CI) investigations.                                                  {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       At any time, the USG may inspect and seize data stored on this IS.                                              {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       Communications using, or data stored on, this IS are not private, are subject to routine monitoring,            {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          interception, and search, and may be disclosed or used for any USG-authorized purpose.                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       This IS includes security measures (e.g., authentication and access controls) to protect USG interests--not     {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          for your personal benefit or privacy.                                                                           {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}  -       Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching    {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          or monitoring of the content of privileged communications, or work product, related to personal representation  {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work       {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}          product are private and confidential. See User Agreement for details.                                           {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{align:left}{bgcolor:yellow}{color:black}                                                                                                                          {/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}
{bgcolor:black} {/color}{align:left}{bgcolor:dark-grey}{color:white}   Accept Conditions and Customize System / View Logs{/align}{align:right} Accept Conditions and Shut Down/Restart  {bgcolor:black} {/color}{/color}{/bgcolor}{/align}
{bgcolor:black} {/color}{bgcolor:dark-grey}{color:black}                                                                                                                          {/color}{/bgcolor}