Table of Contents
Overview
I work in a lot of air-gapped networks and unfortunately, that comes with an extra level of effort for nearly every task. In the world of VMware Cloud Foundation (VCF), SDDC Manager is used to perform a lot of the maintenance tasks through Lifecycle Manager. When it comes times to perform upgrades, however, not having direct access to the Internet requires use of the Bundle Transfer Utility. More information about this utility can be found in the VMware Documentation.
I will warn you now that this post is long and full of snarky comments, but you should hopefully have a better experience in obtaining the bundles using the utility!
Also, the install bundles only need to be downloaded once. So if you are working with multiple stacks of the same version of VCF, the same manifest and install bundles can be used.
The Prerequisites
Unfortunately, the documentation leaves a lot of questions unanswered, beginning with the Prerequisites section. Here are the prerequisites as of version 5.1 with my questions in bold.
* A Windows or Linux computer with internet connectivity for downloading the bundles.
What distributions of Linux? A quick search reveals there are almost 1,000 distributions on the market.
* The computer must have Java 8 or later.
32-bit? 64-bit? Both? Doesn’t matter? Guess what else? How about making sure to include the fact that the keytool utility must be accessible on the System Path?
* A Windows or Linux computer with access to the SDDC Manager appliance for uploading the bundles.
* To upload the manifest file from a Windows computer, you must have OpenSSL installed and configured.
Have you ever tried to actually install OpenSSL on a Windows computer? …And configured…how…what are the configurations that are needed?
* Configure TCP keepalive in your SSH client to prevent socket connection timeouts when using the Bundle Transfer Utility for long-running operations.
An example would be quite helpful.
* This next one is not officially listed as a prerequisite, but this whole process fails without a Customer Connect account that’s able to download the necessary software. This account will also be used in the –depotUser parameter.
* This next one is also not officially listed as a prerequisite, but if using Windows, you will need a utility that can unzip a gz file. My favorite utility for this is 7-zip. I found out during this process that Windows actually includes a built-in utility to handle .tar.gz files! More details below.
* This next one is also not officially listed as a prerequisite, but it should be. If running on a Mac, you are already at a disadvantage as the compatible operating systems listed were Linux or Windows as described above. So on my Mac, I figured I would just install a Linux virtual machine, of course guessing at what distribution to use. I prefer Linux Mint, Ubuntu, or Fedora server. However, the Mac I am using is newer and has an M1 processor (hint hint, where this is going). So now my options for compatible Linux distributions dropped to a much smaller subset. Ubuntu provides a version for 64-bit ARMv8 processors, so I gravitated towards that. It turns out that the Bundle Transfer Utility is only compatible with x86 family of processors.
Download the Bundle Transfer Utility
Download the most recent version of the Bundle Transfer Utility on a computer with internet access.
- Log in to VMware Customer Connect and browse to the Download VMware Cloud Foundation page.
- In the Select Version field, select the version for which you want to download an install bundle. I am matching the version of VCF I am installing.
- Select Drivers & Tools.
- Expand VMware Cloud Foundation Supplemental Tools and select Go To Downloads.
- Select Download Now for the Bundle Transfer Utility.
Bundle Transfer Utility on Windows
For the Windows portion of this post, I am running a non-domain joined Windows 10 Enterprise virtual machine. I am going to start off by taking a snapshot, I tend to forget this very important step for some reason.
So after the virtual machine on Mac failed attempt (ARM-based Mac), I decided to fire up an x86-based Windows virtual machine running in my lab and have a go. Let’s start with the first prerequisite that applies, must have Java 8 or later.
Java is fairly straight forward to download and install. And, if you are wondering, only the 32-bit version is needed, even though the 64-bit version will likely be installed automatically through the installer.
https://www.java.com/en/download
And now, OpenSSL…on Windows…I’ll just leave that there for a minute to sink in.
And, we’re back…
Use this video on YouTube as a reference and download OpenSSL from Shining Light Productions. After the download completes, SmartScreen may block the file. Right-click the file, select Properties and on the General tab, tick the box to Unblock the file. Click Apply and OK, then try to run the installer again.
When installing, I just went with the defaults. When the installation is complete, the OpenSSL command will not quite be ready. It will still need to be added to the System Path. The quickest way to open the Advanced System Properties window it to open a run command and type in, SystemPropertiesAdvanced. A window should open and be on the Advanced tab. Select Environment Variables.
Select the Path variable and select Edit. Add the following New Path environment variable (assuming the default install path was used during installation). While we are here, let’s also add the path for access to the Java KeyTool.
C:\Program Files\OpenSSL-Win64\bin
C:\Program Files\Java\jre-1.8\bin (or)
C:\Program Files (x86)\Java\jre-1.8\bin
Open a new PowerShell window and type openssl version to make sure the openssl runtime is available.
Make a directory on the C:\.
mkdir C:\lcm-tools-prod
Unzip and untar the gz file using an Administrative Windows Command (cmd.exe), not Windows PowerShell and not as a regular user. Also, make sure you are in the directory before running the following command or substitute the absolute path.
tar -xvzf lcm-tools-prod.tar.gz -C C:\lcm-tools-prod
Ok, one last piece before we try this. We need an SSH client and we need to make sure the TCP KeepAlive is set (so two things?).
For this example, I am going to go with WinSCP. javaThe same thing may apply to the downloaded file with SmartScreen. Just Unblock the file and try the installation, accepting the defaults.
Ensure there is network connectivity to the SDDC Manager (another prerequisite). Ping uses Internet Control Message Protocol (ICMP), so this is a different protocol altogether compared to SSH. SSH uses port 22. So just because you get replies via ping, this does not guarantee that the SSH port will also be available!
For the WinSCP session, add the Host name, User name, and Password. The root user is blocked from SSH login, so it must be the vcf user.
Select Advanced > Connection > Keepalives and select Sending of null SSH packets. I left the Seconds between keepalives at the default 30.
Make the connection and Accept the host key if you trust the connection.
Click Continue on the Authentication Banner.
Copy the bundle transfer utility to the SDDC Manager appliance.
SSH in to the SDDC Manager appliance using the vcf user account. This can be performed with the Command Line on Windows, PuTTY, SecureCRT, or even just get on the console of the virtual machine if no other utility is available.
Using a Windows Command use, ssh -l vcf <fully.qualified.domain.name>. Type yes to trust the server’s fingerprint. And finally, provide the password for the vcf user.
Switch to root by typing su and providing the password.
Create the lcm-tools directory.
mkdir /opt/vmware/vcf/lcm/lcm-tools
Copy the Bundle Transfer Utility file
cp lcm-tools-prod.tar.gz /opt/vmware/vcf/lcm/lcm-tools
Change directory and extract the contents of lcm-tools-prod.tar.gz.
cd /opt/vmware/vcf/lcm/lcm-tools tar -xvf lcm-tools-prod.tar.gz
Set the permissions for the lcm-tools directory.
chown vcf_lcm:vcf -R /opt/vmware/vcf/lcm/lcm-tools chmod 750 -R /opt/vmware/vcf/lcm/lcm-tools
Back on the Windows computer (aka the one with Internet access), change to the lcm-tools-prod directory and download the manifest file. I will include one summary image at the end of this section.
cd C:\lcm-tools-prod\bin .\lcm-bundle-transfer-util.bat --download --manifestDownload --depotUser Username
Press Y to acknowledge the message.
Choose whether to enable Customer Experience Improvement Program (CEIP).
Provide the password for the account used as the –depotUser. (This is not the appliance password, but the same password that would be used for Customer Connect)
(–depotUser removed in image for privacy)
I am going to keep the following section and make a start and end section. I was not ever able to get this to work. When I was digging into the lcm-bundle-tranfer-util, it looked like it was expecting to use Cygwin. What I did instead was manually upload the lcmManifest.json file using the SDDC Manager API. I will explain that process after the following section.
+++++++++++++++ The Section Below is For Reference Only +++++++++++++++
Note 1: make sure the Java keytool is in the System Path (C:\Program Files (x86)\Java\jre-1.8\bin) before proceeding!
Note 2: Make sure the following is completed from an administrative Command Line (cmd.exe), not a PowerShell Window!
Next, the documentation says to, “Upload the manifest file to the SDDC Manager appliance.”
When checking the help file for the lcm-bundle-transfer-util, the options are as follows (bold highlights are my own):
--update, --update
This is used to update lcm 2.0 manifest in SDDC Manager. Please do specify sourceManifestDirectory to pick the manifest from the location provided. An additional option of --compatibilityMatrix will only upload compatibility data. Please do specify --inputDirectory to upload compatibility data from the location provided. Also do specify sddcMgrFqdn and sddcMgrUser to perform the operation.
--mfUpload, --sourceManifestDirectory <String>
Can be used with upload option. Used to upload the lcm manifest file. Provide absolute path of the lcmManifest file with this option. (default: )
--sddcMgrFqdn, --sddcMgrFqdn <String>
Can be used with upload option. Used to upload the lcm manifest file. Provide absolute path of the lcmManifest file with this option.
(default: )
--sddcMgrUser, --sddcUsr <String>
Sddc Manager login user name. Should be specified in quotes(if any special characters are included).Use your vSphere SSO credentials for the --sddcMgrUser parameter.
.\lcm-bundle-transfer-util.bat --update --sourceManifestDirectory Manifest-Downloaded-Directory --sddcMgrFqdn FQDN --sddcMgrUser Username
Example of completed command (use double quote for paths with spaces!):
.\lcm-bundle-transfer-util.bat --update --sourceManifestDirectory "C:\Users\Aaron Rombaut\PROD2\evo\vmw\lcm\manifest\v1\lcmManifest.json" --sddcMgrFqdn sddc-90.aaronrombaut.com --sddcMgrUser "[email protected]"
Follow the prompts and provide the SSO password.
Here’s what I noticed along the way…
Enter SDDC Manager User password:
Manifest File Path: C:\Users\Aaron Rombaut\PROD2\evo\vmw\lcm\manifest\v1\lcmManifest.json\lcm\manifest
Setting base path - sddc-90.aaronrombaut.com+++++++++++++++ The Section Above is For Reference Only +++++++++++++++
From the Windows computer that has access to the SDDC Manager and where the lcmManifest.json was downloaded, log in to the SDDC Manager.
On the left side of the screen, at the bottom of the list, select Developer Center.
Select the API Explorer tab.
You can either search the page or use the filter to look for Manifests.
Expand Manifests, POST, and scroll down to the Try it out section. Open the lcmManifest.json file, copy the entire contents (ctrl + a; ctrl + c), and paste the entire contents into the manifest (required) text box for the API.
After the contents are pasted, select Execute. Select Continue on the Are your sure? pop-up. Scroll down to view the Response section. You should see a 202 Accepted Response. I do not have an image from the original upload at this time, but I will supply an image from a second run. As you will notice, the errorCode is LCM_MANIFEST_ALREADY_EXISTS. The Description for the API call states, “Save/Load manifest. Make sure manifest is a valid one. If manifest already exists, it gets overridden.”
Now that the Manifest is uploaded to the SDDC Manager, we can continue to downloading the install bundles.
lcm-bundle-transfer-util.bat -download --outputDirectory absolute-path-output-dir -depotUser depotUser -p vcfVersion --imageType
INSTALLIf you are unsure of the version, you can get it from the SDDC Manager. Inventory > Workload Domains > [choose the Management Domain]. The Version will be displayed.
Below is an example of a complete download request.
lcm-bundle-transfer-util.bat -download --outputDirectory C:\lcm-tools-prod\offline -depotUser [Customer Connect Account] -p 5.1.0.0 --imageType
INSTALLAfter you enter your VMware Customer Connect password, the utility asks Do you want to download vRealize bundles?. Enter Y. The utility displays a list of the available install bundles for the specified version of VMware Cloud Foundation.
Specify the bundles to download. All or individual files, bundle-99542.
Since I am only interested in the Aria products, I am going to specify the bundles individually. Another table will output to confirm the requested bundles to download, then it will start to download the install bundles.
Once the bundles are downloaded, I used WinSCP to transfer the directory, offline, to the SDDC Manager appliance at the /nfs/vmware/vcf/nfs-mount/ location.
Once the bundles have been uploaded to the appliance, run the lcm bundle transfer utility again to be able to use them. This has to be run as the root account, not the vcf account!
/opt/vmware/vcf/lcm/lcm-tools/bin/lcm-bundle-transfer-util -upload -bundleDirectory absolute-path-bundle-dir/opt/vmware/vcf/lcm/lcm-tools/bin/lcm-bundle-transfer-util -upload -bundleDirectory /nfs/vmware/vcf/nfs-mount/offlineThe utility uploads the bundles and displays upload status for each bundle. When the uploads complete, the bundles are available in the SDDC Manager UI. Navigate to Lifecycle Management > Bundle Management > Download History to see the downloaded bundles.