Archived Post
This article has been archived and is no longer maintained.
A consolidated and up-to-date version of this content is available here → https://www.aaronrombaut.com/offline-depot-management-for-vmware-cloud-foundation-vcf-legacy-and-current-tooling/
Overview
If the SDDC Manager appliance is not connected to an online or offline depot, you can use the Bundle Transfer Utility to download the bundles to a different computer and then upload them to the SDDC Manager appliance.
Moreover, in cases where an organization comprises numerous VMware Cloud Foundation (VCF) instances or operates within air-gapped networks, the implementation of an offline depot may be advisable. The utilization of an offline depot enhances the experience for air-gapped customers by minimizing the number of steps required to distribute artifacts across multiple VCF instances repeatedly.
RTFM
VCF authenticated downloads configuration update instructions
Offline Download of VMware Cloud Foundation 5.2.x Upgrade Bundles
Download Bundles to an Offline Depot
Connect SDDC Manager to a Software Depot for Downloading Bundles
VMware Cloud Foundation Offline Depot Introduction
Warnings and Assumptions
First, the warning. This whole procedure is not for the faint of heart. If you are comfortable on the command line, you will likely succeed. Hopefully, in the future, this process will be streamlined and we can laugh about this absurd process someday.
Now, on to the assumptions.
- know what VMware Cloud Foundation (VCF) is and what it does
- understand basic virtualization concepts
- working SDDC Manager
- generated your download token with Broadcom support (link)
- understand how to edit files on the command line
I also hope you have a good heartbeat and a greater sense of humor!
Build an Offline Depot
To get started with an offline depot, we need a virtual machine to host the upgrade bundles.
Deploy and Configure a Virtual Machine
I am going to use a minimal RHEL image for the virtual machine to build the Offline Depot.
Complete the marked items.
Enter the correct credentials or use an appropriate Activation Key to connect to the Red Hat servers.
My plan is to install the operating system to the smaller hard drive (/dev/sda in the example) and then use the second hard drive (/dev/sdb) for package storage.
I only need a Basic Web Server and Guest Agents since I am running on a VMware infrastructure.
Add a root password and confirmation.
While not required at this point, the Network Configuration is configured with static IP address information and a hostname.
The Ethernet toggle may need to toggled off and back on to activate the connection.
Once all the tasks are completed, the Begin Installation button activates.
Wait until the installation completes to reboot.
Permit root Login for SSH
By default, SSH does not allow root login. However, to make it easier to copy and paste commands during setup, I’ll temporarily enable root access via SSH. I’ll disable it again when I’m done.
Start by opening a console window and logging in as root. Then, let’s add a temporary file in the sshd_config.d directory to make the changes.
vi /etc/ssh/sshd_config.d/70-allow-root-ssh.confAdd the following line to the file.
PermitRootLogin yesSave the configuration file.
<Esc>
:wqRestart the sshd.service service and check the status.
systemctl restart sshd.service
systemctl status sshd.service
Update the Virtual Machine
Now that we can log in with SSH, we can update the virtual machine. Log in as root to the virtual machine with SSH.
Update the virtual machine.
dnf update
Reboot the virtual machine.
rebootOptional – Use Additional Storage Device
Since the size of the offline bundles can vary, I would like to use an additional storage device to store the bundles.
Start out listing the available storage devices.
lsblkFrom the listing, I can see that I have an additional disk (sdb) available to use.
I need to add a partition to make the disk usable so I will use gdisk.
gdisk /dev/sdb
You can use the letter ? for help.
Use the letter n to add a new partition. I am also going to use the default partition number of 1 and the other defaults.
Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-1048575966, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-1048575966, default = 1048575966) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8300
Changed type of partition to 'Linux filesystem'When returned back to Command (? for help): use the letter w to write the changes.
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.Now, when viewing the available block devices, there should be a new partition available.
lsblk
Now that I know my disk is partitioned, I need to format it with a file system.
mkfs.ext4 /dev/sdb1
I can check that the filesystem was created using the -f switch.
lsblk -f
Notice the UUID in the output? That’s the device UUID, not the partition UUID. We will need the partition UUID to update the fstab file so that our disk automounts. The device UUID can be helpful if there are many disks and partitions listed.
blkid
Open the fstab file and add the following line to the end of the file. Shift + G to get to the end of the file, little o to enter Insert Mode on the next line.
vi /etc/fstabPARTUUID=<UUID> /var/www/html ext4 defaults 0 0Save and close the file.
<esc>
:wqMake sure there are no files currently in the /var/www/html directory. If there are, move them to somewhere else. Mount all the devices using the fstab file to test.
mount -aVerify the file systems and mount points.
lsblk -f
Restore the file context label since we added the new volume.
restorecon -Rv /var/www/html
Now, when the install bundles are downloaded to /var/www/html, the files will be on the new disk. This will help to prevent the OS disk from filling up and causing trouble in the future.
Apache HTTP Server
Check if Apache is installed.
systemctl status httpd.service
Install Apache
Install the Apache Web server if it wasn’t installed from the initial installation.
dnf install httpdCertificates
Create a temporary directory to prepare the certificate.
mkdir /root/http-certificatesChange to the new directory.
cd /root/http-certificatesGenerate a private key.
openssl genpkey -out /root/http-certificates/server.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048Make a configuration file containing Subject Alternative Name (SAN) information. Modern browsers will not trust the certificate otherwise.
vi offline-depot.cnf[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no
[req_distinguished_name]
C = <country>
ST = <state>
L = <locality>
O = <organization>
OU = <organizational unit>
CN = <fully-qualified domain name>
[req_ext]
subjectAltName = @alt_names
[alt_names]
IP.1 = <ip address>
DNS.1 = <fully-qualified domain name>
DNS.2 = <short name>Generate the Certificate Signing Request (CSR) using the private key and the configuration file.
openssl req -new -key /root/http-certificates/server.key -out /root/http-certificates/request.csr -config /root/http-certificates/offline-depot.cnfHave the Certificate Signing Request (CSR) signed by your Certificate Authority (CA). This will depend on your organization’s policy. Once the certificate request is signed, upload the machine (machine.crt), intermediate (if applicable), and root certificates, in PEM format and encoded in Base64, to the /root/http-certificates directory.
Combine the files, top-to-bottom, starting with the <machine certificate>, <intermediate certificate>, then <root certificate> into a file called server.crt. In my lab, I do not have an intermediate signing authority, so I will skip that.
cat machine.crt >> server.crt
cat intermediate.crt >> server.crt
cat root.crt >> server.crt
Open the file, remove any spaces between the certificates if necessary, and then save it. Move the server key and certificate to the /etc/httpd/conf directory.
mv /root/http-certificates/server.* /etc/pki/tls/certs/Fix the permissions on the key.
chmod 0400 /etc/pki/tls/certs/server.key && chown root:root /etc/pki/tls/certs/server.keyRestore the context of the files added.
restorecon -Rv /etc/pki/tls/certs
Configure Apache HTTP Server
Edit the /etc/httpd/conf/httpd.conf file, setting the ServerName and ServerAdmin directives accordingly.
vi /etc/httpd/conf/httpd.confShow line numbers or use the line-number,character reference mapping natively in vi.
:set numberServerAdmin is near line 91 and ServerName is near line 100.
Save and close the file.
<esc>
:wqOpen up the ssl.conf file.
vi /etc/httpd/conf.d/ssl.conf On about line 85, comment out the original SSLCertificateFile and add the following on the next line.
SSLCertificateFile /etc/pki/tls/certs/server.crtOn about line 93, comment out the original SSLCertificateKeyFile and add the following on the next line.
SSLCertificateKeyFile /etc/pki/tls/certs/server.keyOptionally, if you did not include the full chain in the SSLCertificateFile, add the following on about line 102.
SSLCertificateChainFile /etc/pki/tls/certs/root.crtAdd <Directory> Directives
Continuing to edit the /etc/httpd/conf.d/ssl.conf file.
vi /etc/httpd/conf.d/ssl.confGo to the end of the file.
<shift> + gInsert the following lines after the </VirtualHost> (on about line 203) directive (use :set number in vi to view line numbers).
<Directory /var/www/html/<token id>/PROD/COMP/SDDC_MANAGER_VCF>
# Basic Auth (VCF 4.x and 5.0)
AuthType Basic
AuthName "Basic Authentication"
AuthUserFile /etc/httpd/conf/.htpasswd
require valid-user
</Directory>
<Directory "/var/www/html/<token id>/PROD/COMP/SDDC_MANAGER_VCF/Compatibility/VxrailCompatibilityData.json">
# VxRail VVS Cookie Validation (VCF 5.0)
<If "%{HTTP:Cookie} == 'ngssosession=ngsso-token' ">
Require all granted
</If>
</Directory>
<Directory /var/www/html/<token id>/PROD/vsan/hcl>
<If "%{HTTP:X-vmw-esp-clientid} == 'offline-depot-hcl-vcf' ">
Require all granted
</If>
</Directory>
# These Alias statements are needed only for VCF 5.1.0.0.
Alias /products/v1/bundles/lastupdatedtime /var/www/html/<token id>/PROD/vsan/hcl/lastupdatedtime.json
Alias /products/v1/bundles/all /var/www/html/<token id>/PROD/vsan/hcl/all.jsonNotice you will need to replace the <token id> in the snippet above with your unique token generated from the Broadcom Support portal for this. There are five locations highlighted in bold to change. If you are using vi or vim, you can do the replacement with one line.
:%s#<token id>#<your actual token id>#gSave and close the file.
<esc>
:wqConfigure Basic Authentication
Generate a user and password for use with the offline depot. If more than one user will be created, remove the -c after the initial file creation.
htpasswd -c /etc/httpd/conf/.htpasswd <username>
Protect the file by modifying the file’s permissions.
chown apache /etc/httpd/conf/.htpasswd
chmod 0400 /etc/httpd/conf/.htpasswd
Change the SELinux File Context
Run the following commands to restore the file context on the files that were created or modified, if necessary.
restorecon -Rv /etc/httpd/confrestorecon -Rv /etc/pki/tls/certsrestorecon -Rv /var/www/htmlStart and Enable Apache HTTP Server
After making modifications to configuration files, it’s a good idea to test them. If Apache is already running, stop the service.
systemctl stop httpd.serviceTest the configuration.
apachectl configtestor
httpd -t
Assuming that the everything is good, start the server.
systemctl start httpd.serviceCheck the status of the server, ensuring the the service is active (running).
systemctl status httpd.serviceNow, enable the service so that it runs at startup.
systemctl enable httpd.service
Configure firewalld
To allow web traffic to the server, add an ALLOW rule to firewalld and save it.
firewall-cmd --add-service=https --permanentCheck that the rule is configured as expected.
firewall-cmd --list-all
Note: If you don’t see the https service in the list of services, try to reload the firewall configuration.
firewall-cmd --reloadOpen a web browser and navigate to the offline depot URL or IP address.
https://offline-depot.domain.comor
https://<ip address>Clean Up Tasks
Remove the directory used to generate the certificates.
rm -rf /root/http-certificatesComment out all the lines in the welcome.conf file. Note: Do not delete the file! It will be replaced during an upgrade.
vi /etc/httpd/conf.d/welcome.conf:g/^[^#]/s/^/# /
Restart the httpd service after commenting out the file.
systemctl restart httpdIf PermitRootLogin was used for SSH, it is best to remove the file.
rm -f /etc/ssh/sshd_config.d/70-allow-root-ssh.confOffline Depot Bundles Download
For this section, I am going to use the same virtual machine that was just configured above. In an air-gapped network, an additional virtual machine may be required. The offline bundles must be downloaded from an internet connected computer and then transferred over to the air-gapped offline depot virtual machine. In my case, I am cutting out the additional virtual machine and just using the one. The steps will be the same up until the file transfer.
Download the Bundle Transfer Utility
Download the bundle transfer utility (lcm-tools-prod.tar.gz) from support.broadcom.com and transfer it to the offline depot virtual machine to /opt/vmware/vcf/lcm/lcm-tools.
mkdir -p /opt/vmware/vcf/lcm/lcm-tools
cd /opt/vmware/vcf/lcm/lcm-tools
Extract the file.
tar -xvf lcm-tools-prod.tar.gzChange into the bin directory.
cd bin
Make the lcm-bundle-transfer-util file executable.
chmod u+x lcm-bundle-transfer-util
Test that the tool will work.
./lcm-bundle-transfer-util --helpEdit the application-prod.properties file
Prerequisite step required here. Ensure you have already generated your unique token and have it available (VCF Authenticated Downloads). The next few steps are going to require editing the application-prod.properties file.
Open the application-prod.properties file to begin editing.
vi /opt/vmware/vcf/lcm/lcm-tools/conf/application-prod.propertiesLet’s get some line numbers to make this easier.
:set numberLines 25, 26, 27, 39, and 43 need to be edited. I will include the original text as well as updated sample text. If desired, make a backup of the entire file before editing.
Line 25
lcm.depot.adapter.host=depot.vmware.comlcm.depot.adapter.host=dl.broadcom.comor
:%s#depot.vmware.com#dl.broadcom.com#gLine 26
lcm.depot.adapter.remote.rootDir=/PROD2Replace <downloadToken> with the token you received from the Broadcom Support portal.
lcm.depot.adapter.remote.rootDir=/<downloadToken>/PRODLine 27
lcm.depot.adapter.remote.repoDir=/evo/vmwlcm.depot.adapter.remote.repoDir=/COMP/SDDC_MANAGER_VCFLine 39
lcm.depot.adapter.remote.lcmManifestDir=/evo/vmw/lcm/manifestlcm.depot.adapter.remote.lcmManifestDir=/COMP/SDDC_MANAGER_VCF/lcm/manifestLine 43
lcm.depot.adapter.remote.bundletransferconfig.repoDir:/evo/vmw/obtulcm.depot.adapter.remote.bundletransferconfig.repoDir:/COMP/SDDC_MANAGER_VCF/obtuSave the file.
<esc>
:wqSet up the Offline Depot
Create a plain text password file and make it readable to only the user running the bundle transfer utility. (yes, this is real)
vi ~/obtu-user-passwordchmod 400 ~/obtu-user-passwordPrepare to download the bundles required to upgrade VMware Cloud Foundation.
cd /opt/vmware/vcf/lcm/lcm-tools/binHere is the syntax.
./lcm-bundle-transfer-util --setUpOfflineDepot -sv vcf-source-version --offlineDepotRootDir offline-depot-root-dir --offlineDepotUrl url:port --depotUser user-name --depotUserPasswordFile path-to-password-fileHere is an example.
./lcm-bundle-transfer-util --setUpOfflineDepot -sv 5.2.0.0 --offlineDepotRootDir /var/www/html --offlineDepotUrl https://offline-depot.aaronrombaut.com:443 --depotUser [email protected] --depotUserPasswordFile ~/obtu-user-passwordHere is a documentation example. One thing to note is that the --sourceVersion or -sv parameter is supposed to define the version that will be downloaded. However, while I am trying to download patches for 5.2.1.0, the tool will report back that there are no new bundles. If I specify the version as 5.2.0.0, I will get back the updates I am looking for.
./lcm-bundle-transfer-util --setUpOfflineDepot -sv 5.0.0.0 --offlineDepotRootDir /var/www/html --offlineDepotUrl https://10.123.456.78:8282 --depotUser [email protected] --depotUserPasswordFile ../vmw-depot Here is a sample list of bundles that will be downloaded when you specify the source version as 5.2.0.0. Take note of the bundle names—although the product version is listed as 5.2.1.0, the list also includes patches for SDDC Manager 5.2.1.1 and 5.2.1.2.
This can be a bit confusing. I initially tried using 5.2.1.2 as the value for the -sv (source version) parameter, but that consistently returned no results. Using 5.2.0.0 instead resolved the issue and returned the expected bundles.
-------------------------------------------------------------------------------------------------------------------------------------------------
Bundle | Product Version | Bundle Size | Bundle Component | Bundle Type
-------------------------------------------------------------------------------------------------------------------------------------------------
bundle-133762 | 5.2.1.0 | 606.4 MB | ESX_HOST-8.0.3-24280767 | PATCH
bundle-133763 | 5.2.1.0 | 8895.2 MB | NSX_T_MANAGER-4.2.1.0.0-24304122 | PATCH
bundle-202281 | 5.2.1.0 | 2364.8 MB | SDDC_MANAGER_VCF-5.2.1.1-24397777 | PATCH
bundle-214558 | 5.2.1.0 | 2375.9 MB | SDDC_MANAGER_VCF-5.2.1.2-24690695 | PATCH
bundle-133760 | 5.2.1.0 | 2364.8 MB | SDDC_MANAGER_VCF-5.2.1.0-24307856 | PATCH
bundle-133761 | 5.2.1.0 | 0.0 MB | SDDC_MANAGER_VCF-5.2.1.0-24307856 | PATCH (Drift)
bundle-133765 | 5.2.1.0 | 18817.0 MB | VCENTER-8.0.3.00300-24305161 | PATCH
bundle-130870 | 5.2.1.0 | 4238.5 MB | NSX_ALB-22.1.7-24190832 | INSTALL
bundle-133764 | 5.2.1.0 | 11636.7 MB | NSX_T_MANAGER-4.2.1.0.0-24304122 | INSTALL
bundle-133766 | 5.2.1.0 | 11832.3 MB | VCENTER-8.0.3.00300-24305161 | INSTALL
-------------------------------------------------------------------------------------------------------------------------------------------------
Once the downloads complete, copy the files to the document root or if they were downloaded directly, view the directory.
ls -la /var/www/html/Notice your directory will use your token id where I have highlighted the <download token>.
root@offline-depot [ /opt/vmware/vcf/lcm/lcm-tools/bin ]# tree /var/www/html
/var/www/html
└── <download token>
└── PROD
├── COMP
│ └── SDDC_MANAGER_VCF
│ ├── bundles
│ │ ├── bundle-130870.tar
│ │ ├── bundle-133760.tar
│ │ ├── bundle-133761.tar
│ │ ├── bundle-133762.tar
│ │ ├── bundle-133763.tar
│ │ ├── bundle-133764.tar
│ │ ├── bundle-133765.tar
│ │ ├── bundle-133766.tar
│ │ ├── bundle-202281.tar
│ │ └── bundle-214558.tar
│ ├── Compatibility
│ │ └── VmwareCompatibilityData.json
│ ├── deltaFileDownloaded
│ ├── deltaFileDownloaded.md5
│ ├── index.v3
│ ├── lcm
│ │ └── manifest
│ │ └── v1
│ │ └── lcmManifest.json
│ ├── manifests
│ │ ├── bundle-130870.manifest
│ │ ├── bundle-130870.manifest.sig
│ │ ├── bundle-133760.manifest
│ │ ├── bundle-133760.manifest.sig
│ │ ├── bundle-133761.manifest
│ │ ├── bundle-133761.manifest.sig
│ │ ├── bundle-133762.manifest
│ │ ├── bundle-133762.manifest.sig
│ │ ├── bundle-133763.manifest
│ │ ├── bundle-133763.manifest.sig
│ │ ├── bundle-133764.manifest
│ │ ├── bundle-133764.manifest.sig
│ │ ├── bundle-133765.manifest
│ │ ├── bundle-133765.manifest.sig
│ │ ├── bundle-133766.manifest
│ │ ├── bundle-133766.manifest.sig
│ │ ├── bundle-202281.manifest
│ │ ├── bundle-202281.manifest.sig
│ │ ├── bundle-214558.manifest
│ │ └── bundle-214558.manifest.sig
│ └── tmp
│ ├── index.v3
│ └── lcmManifest.json
└── vsan
└── hcl
├── all.json
└── lastupdatedtime.json
13 directories, 39 filesSet the correct file permissions to the directories and files or if a cron job was set, this will likely already have taken place.
chown apache:apache -R /var/www/htmlfind /var/www/html/ -type d -exec chmod 0755 {} \;find /var/www/html/ -type f -exec chmod 0644 {} \;The httpd service may need to be restarted for the changes to take effect.
systemctl restart httpd.serviceTry to navigate to https://<ip address>/<Token ID>/PROD/COMP/SDDC_MANAGER_VCF and verify the site prompts for a log in. Use the log in that was created and added to the .htaccess file to verify the credentials.
SDDC Manager Edits and Connecting to Offline Depot
Edit the application-prod.properties file
After Broadcom changed to VCF authenticated downloads, even using an Offline Depot still requires the SDDC Manager application-prod.properties file to be edited to look at the correct directories.
vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties:set numberLine 66
lcm.depot.adapter.host=depot.vmware.comlcm.depot.adapter.host=dl.broadcom.comLine 68
lcm.depot.adapter.remote.rootDir=/PROD2lcm.depot.adapter.remote.rootDir=/<downloadToken>/PRODLine 69
lcm.depot.adapter.remote.repoDir=/evo/vmwlcm.depot.adapter.remote.repoDir=/COMP/SDDC_MANAGER_VCFLine 79
lcm.depot.adapter.remote.lcmManifestDir=/evo/vmw/lcm/manifestlcm.depot.adapter.remote.lcmManifestDir=/COMP/SDDC_MANAGER_VCF/lcm/manifestLine 60
lcm.depot.adapter.remote.lcmProductVersionCatalogDir=/evo/vmw/lcm/productVersionCataloglcm.depot.adapter.remote.lcmProductVersionCatalogDir=/COMP/SDDC_MANAGER_VCF/lcm/productVersionCatalogSave the file, making sure you add the ! to the command to overwrite the read-only file.
<esc>
:wq!Restart the LCM service.
systemctl restart lcmConnect SDDC Manager to Offline Depot
Note: Make sure the TLS certifcates are configured for both the Offline Depot and the SDDC Manager. Otherwise, it is likely a Secure protocol communication error will be logged. You can tail the lcm-debug.log on SDDC Manager for more details if there are issues connecting to the depot. In the UI, I received the message, “Failed to connect to VMware depot with the provided user credentials. Cause: {0}.” Turns out the credentials are correct, the lack of trust between the two servers is actually the cause.
tail -f /var/log/vmware/vcf/lcm/lcm-debug.log
In the navigation pane, click Administration > Depot Settings.
Connect SDDC Manager to the Offline Depot by clicking the Set Up button.
Enter the appropriate details for the offline depot.
You can view the lcm-debug.log to view what’s going on in the background since the Tasks won’t populate. Be patient here as the Bundle Management populates.
Conclusion
By following this guide, you’ve successfully set up an Offline Depot for VMware Cloud Foundation (VCF), enabling patch and bundle management in air-gapped or restricted environments. This configuration allows SDDC Manager to pull upgrade bundles from a secure, local source without requiring internet access, ensuring compliance, control, and operational continuity across isolated VCF deployments.
While the setup process is intricate and requires command-line proficiency, the end result significantly streamlines bundle distribution, especially in environments with multiple VCF instances. With the Offline Depot in place, your organization is now better positioned to manage VCF updates reliably and securely—even in the most disconnected networks.