VxRail: Upgrade Notes

Overview

There are some things that should be done prior to upgrading the VxRail that will help with a smooth upgrade. I will not be performing a VxRail upgrade, here, but rather list resources that I have been provided or used in the past. Some of the resources below require you to log in to the Dell Support page.

Disclaimer: I do not work for Dell; these are my notes and should not be used without Dell Support involved in a production scenario.

Continue reading “VxRail: Upgrade Notes”

Reset VxRail Root and Mystic Accounts

I have been working with clients that are using VxRail for their infrastructure. While administering these VxRail deployments, sometimes the mystic or root accounts get locked, the password’s expire or are just plain lost. Either way, it is a very frustrating situation to find yourself in.

localhost login: root
Password:
Login incorrect

localhost login: _

Turn to Google and search for reset VxRail Manager password and come upon the following Dell kb, Dell VxRail: VxRail Manager root password is lost.(https://www.dell.com/support/kbdoc/en-us/000064579/vxrail-how-to-reset-the-root-password-for-vxrail-manager)

Even after following this article, you realize almost immediately that this hasn’t been updated to reflect newer versions. The very first picture depicts a SUSE Linux Enterprise 12 screenshot.

More recent versions of VxRail are running on SUSE Linux Enterprise 15. The following procedure will hopefully assist you until Dell can update their documentation.

Procedure

Start by taking a snapshot of your VxRail Manager!

Open a web or remote console and then restart the virtual machine. When you see the following splash screen, press the ‘e’ key on your keyboard to interrupt the boot sequence.

This will bring you to the GNU GRUB boot menu. Look for the line starting with linux (14 lines down in my case). Press Ctrl-e to go to the end of the line and add init=/bin/bash.

Press Ctrl-x or F10 to boot.

If you are following the numerous sources out there, they will point you to use the pam_tally2 utility. As you can see below, this won’t work…yet.

Create the log directory and change to it with the following:

mkdir -p /var/log
cd /var/log

Add the tallylog file.

touch tallylog
chmod 600 tallylog

Now, you should be able to use /sbin/pam_tally2. If you are not familiar with the syntax, the below images should help. The full help is at the end of this post for more information. You can see that the two users have 0 failures currently. If you do know the password and just want to unlock the account so you can log in again, use the following syntax.

/sbin/pam_tally2 -u <user name> -r

Reset the passwords

As long as the directory was created above, the passwd utility should be able to be used, now. Ignore the message that the password was used already. I tried completely new passwords and still receive this message.

Reboot the virtual machine

Unfortunately, I have not found a suitable way to reboot the virtual machine, yet. VMware Tools (more accurately, open-vm-tools) is not started since we are not booted in a full multi-user state.

Make sure you have completed your password or unlocking maintenance. When you are ready, go to the power control for the virtual machine and select the Power off option. Wait a moment before powering the virtual machine back on. At this point, the passwords you set or the account should be unlocked.

After you have verified that your accounts work, be sure to remove the snapshot you took in the beginning!

Hopefully this will help you out.

pam_tally2 Help

/sbin/pam_tally2: [-f rooted-filename] [--file rooted-filename]
   [-u username] [--user username]
   [-r] [--reset[=n]] [--quiet]

Dell EMC Avamar and VMware 6.5 Snapshot Quiescing Error

During a rebuild of a Dell EMC Avamar backup solution, we ran into an issue where VMware was logging, “An error occurred while quiescing the virtual machine. See the virtual machine’s event log for details.” for a few virtual machine backups. While testing, we noticed that it was not occurring on all backups.

On a functional backup with no errors, a machine had a service called VMware Snapshot Provider and this service was set to manual. On the machine I was troubleshooting, this service didn’t even show up in the list. I opted to uninstall tools, restart the server, and then reinstall VMware tools. Upon inspection of the services listing this time, the service was present and set to manual. I tested a backup and there were no failures and the machine backed up fine.

I tested another virtual machine that I knew previously worked and this time I went to services and set the VMware Snapshot Provider service to disabled and initiated a backup on the Avamar UI (AUI). As expected, VMware logged an event as noted above.

In conclusion, if you notice this error occurring, regardless if there were no errors previously, check for this service. The service can be reinstalled by invoking the VMware Tools installer again and changing the installed features. If all else fails, just remove, reboot, and re-install.