Table of Contents
Overview
If the SDDC Manager appliance is not connected to an online or offline depot, you can use the Bundle Transfer Utility to download the bundles to a different computer and then upload them to the SDDC Manager appliance.
Moreover, in cases where an organization comprises numerous VMware Cloud Foundation (VCF) instances or operates within air-gapped networks, the implementation of an offline depot may be advisable. The utilization of an offline depot enhances the experience for air-gapped customers by minimizing the number of steps required to distribute artifacts across multiple VCF instances repeatedly.
RTFM
VCF authenticated downloads configuration update instructions
Offline Download of VMware Cloud Foundation 5.2.x Upgrade Bundles
Download Bundles to an Offline Depot
Connect SDDC Manager to a Software Depot for Downloading Bundles
VMware Cloud Foundation Offline Depot Introduction
Warnings and Assumptions
First, the warning. This whole procedure is not for the faint of heart. If you are comfortable on the command line, you will likely succeed. Hopefully, in the future, this process will be streamlined and we can laugh about this absurd process someday.
Now, on to the assumptions.
- know what VMware Cloud Foundation (VCF) is and what it does
- understand basic virtualization concepts
- working SDDC Manager
- generated your download token with Broadcom support (link)
- understand how to edit files on the command line
I also hope you have a good heartbeat and a greater sense of humor!
Build an Offline Depot
To get started with an offline depot, we need a virtual machine to host the upgrade bundles.
Deploy and Configure a Virtual Machine
I am going to use a minimal Photon OS 5.0 GA image for the virtual machine to build the Offline Depot. The following steps are fairly basic and self-explanatory. I will include screenshots and applicable comments, when needed, for each screen.
My plan is to install the operating system to /dev/sda and then use /dev/sdb for package storage.
For the Network Configuration, I am configuring a static IP address.
Since this virtual machine is running on VMware ESXi, I choose the VMware hypervisor optimized Linux kernel.
Permit root Login for SSH
By default, SSH does not allow root login. However, to make it easier to copy and paste commands during setup, I’ll temporarily enable root access via SSH. I’ll disable it again when I’m done.
Start by opening a console window and logging in as root. Then, let’s edit the sshd_config file to make the necessary changes.
vi /etc/ssh/sshd_config
Search for PermitRootLogin and change the value to yes.
/PermitRootLogin
Save the configuration file.
<Esc>
:wqRestart the sshd.service service and check the status.
systemctl restart sshd.service
systemctl status sshd.service
Update the Virtual Machine
Now that we can log in with SSH, we can update the virtual machine. Log in as root to the virtual machine with SSH.
Update the virtual machine.
tdnf update --assumeyes
Reboot the virtual machine.
rebootApache HTTP Server
Install Apache
Install the Apache Web server.
tdnf install httpd --assumeyesCertificates
Create a temporary directory to prepare the certificate.
mkdir /root/http-certificatesChange to the new directory.
cd /root/http-certificatesGenerate a private key.
openssl genpkey -out /root/http-certificates/server.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048Make a configuration file containing Subject Alternative Name (SAN) information. Modern browsers will not trust the certificate otherwise.
vi offline-depot.cnf[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no
[req_distinguished_name]
C = <country>
ST = <state>
L = <locality>
O = <organization>
OU = <organizational unit>
CN = <fully-qualified domain name>
[req_ext]
subjectAltName = @alt_names
[alt_names]
IP.1 = <ip address>
DNS.1 = <fully-qualified domain name>
DNS.2 = <short name>Generate the Certificate Signing Request (CSR) using the private key and the configuration file.
openssl req -new -key /root/http-certificates/server.key -out /root/http-certificates/request.csr -config /root/http-certificates/offline-depot.cnfHave the Certificate Signing Request (CSR) signed by your Certificate Authority (CA). This will depend on your organization’s policy. Once the certificate request is signed, upload the machine, intermediate (if applicable), and root certificates, in PEM format and encoded in Base64, to the /root/http-certificates directory.
Combine the files, top-to-bottom, starting with the <machine certificate>, <intermediate certificate>, then <root certificate> into a file called server.crt. In my lab, I do not have an intermediate signing authority, so I will skip that.
cat machine.crt >> server.crt
cat intermediate.crt >> server.crt
cat root.crt >> server.crt
Open the file, remove any spaces between the certificates if necessary, and then save it. Move the server key and certificate to the /etc/httpd/conf directory.
mv /root/http-certificates/server.* /etc/httpd/confFix the permissions on the two files.
chmod 0400 /etc/httpd/conf/server.key && chown root:root /etc/httpd/conf/server.keyConfigure Apache HTTP Server
Edit the /etc/httpd/conf/httpd.conf file, setting the ServerName and ServerAdmin directives accordingly.
vi /etc/httpd/conf/httpd.confShow line numbers or use the line-number,character reference mapping natively in vi.
:set numberServerAdmin is on line 213 and ServerName is on line 222.
Remove the octothorpe (#) from the following lines.
Enable ssl_module on line 150.
LoadModule ssl_module /usr/lib/httpd/modules/mod_ssl.so
Enable socache_shmcb_module on line 93.
LoadModule socache_shmcb_module /usr/lib/httpd/modules/mod_socache_shmcb.so
On line 522.
Include conf/extra/httpd-ssl.conf
Save and close the file.
<esc>
:wqConfigure Basic Authentication
According to apache.org, htpasswd is used to create and update the flat-files used to store usernames and password for basic authentication of HTTP users.
Generate a user and password for use with the offline depot. If more than one user will be created, remove the -c after the initial file creation.
htpasswd -c /etc/httpd/conf/.htpasswd <username>
Protect the file by modifying the file’s permissions.
chown apache /etc/httpd/conf/.htpasswd
chmod 0400 /etc/httpd/conf/.htpasswd
Add <Directory> Directives
Open and edit the /etc/httpd/conf/extra/httpd-ssl.conf file.
vi /etc/httpd/conf/extra/httpd-ssl.confGo to the end of the file.
<shift> + gInsert the following lines after the </VirtualHost> (line 290) directive (use :set number in vi to view line numbers). Notice you will need the token id from the Broadcom Support portal for this. There are five locations highlighted in bold to change.
<Directory /etc/httpd/html/<token id>/PROD/COMP/SDDC_MANAGER_VCF>
# Basic Auth (VCF 4.x and 5.0)
AuthType Basic
AuthName "Basic Authentication"
AuthUserFile /etc/httpd/conf/.htpasswd
require valid-user
</Directory>
<Directory "/etc/httpd/html/<token id>/PROD/COMP/SDDC_MANAGER_VCF/Compatibility/VxrailCompatibilityData.json">
# VxRail VVS Cookie Validation (VCF 5.0)
<If "%{HTTP:Cookie} == 'ngssosession=ngsso-token' ">
Require all granted
</If>
</Directory>
<Directory /etc/httpd/html/<token id>/PROD/vsan/hcl>
<If "%{HTTP:X-vmw-esp-clientid} == 'offline-depot-hcl-vcf' ">
Require all granted
</If>
</Directory>
# These Alias statements are needed only for VCF 5.1.0.0.
Alias /products/v1/bundles/lastupdatedtime /etc/httpd/html/<token id>/PROD/vsan/hcl/lastupdatedtime.json
Alias /products/v1/bundles/all /etc/httpd/html/<token id>/PROD/vsan/hcl/all.jsonSave and close the file.
<esc>
:wqStart and Enable Apache HTTP Server
After making modifications to configuration files, it’s a good idea to test them.
httpd -tAssuming that the everything is good, now, start the server.
systemctl start httpd.serviceCheck the status of the server, ensuring the the service is active (running).
systemctl status httpd.serviceNow, enable the service so that it runs at startup.
systemctl enable httpd.service
Configure iptables
To allow web traffic to the server, add an ALLOW rule to iptables and save it.
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPTCheck that the rule is configured as expected.
iptables -LIf the rules look good and are in the correct order, save the configuration.
iptables-save > /etc/systemd/scripts/ip4save 
Open a web browser and navigate to the offline depot URL or IP address.
https://offline-depot.domain.comor
https://<ip address>When you see the following, you can assume the server is up and running.
Clean Up Tasks
Remove the directory used to generate the certificates.
rm -rf /root/http-certificatesRemove the default index file if it exists.
rm -rf /etc/httpd/html/index.htmlIf PermitRootLogin was used for SSH, it is best to set that back to no.
vi /etc/ssh/sshd_config
Optional – cron
Some admins may cringe at the following configuration item, but this is the best way I know to ensure it’s done and not forgotten. (Okay, so maybe I have control issues?)
When downloading or moving downloaded bundles to the DocumentRoot (/etc/httpd/html), the file permissions need to be fixed. This can be done manually using the following commands or using cron to schedule these commands and not have to think about it.
chown apache -R /etc/httpd/html/
find /opt/lampp/htdocs -type d -exec chmod 0500 {} \;
find /opt/lampp/htdocs -type f -exec chmod 0400 {} \;Install the Cron Daemon.
tdnf install -y cronieEdit crontab.
crontab -eInsert the commands, one per line.
* * * * * chown apache -R /etc/httpd/html/ >/dev/null 2>&1
* * * * * find /opt/lampp/htdocs -type d -exec chmod 0500 {} ; >/dev/null 2>&1
* * * * * find /opt/lampp/htdocs -type f -exec chmod 0400 {} ; >/dev/null 2>&1Test crontab file syntax.
crontab -T /etc/cron.d/dailyjobsIf No syntax issues were found in the crontab file, start the crontab service.
systemctl start crond.serviceOptional – Use Additional Storage Device
Since the size of the offline bundles can vary, I would like to use an additional storage device to store the bundles.
Start out listing the available storage devices.
lsblkFrom the listing, I can see that I have an additional disk available to use.
I need to add a partition to make the disk usable so I will use fdisk.
fdisk /dev/sdbYou can use the letter m for help, but we are going to add a single partition. Use the letter n to add a new partition. I am also going to use the default partition number of 1 and the other defaults.
Command (m for help): n
Partition number (1-128, default 1):
First sector (34-1048575966, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-1048575966, default 1048573951):
Created a new partition 1 of type 'Linux filesystem' and of size 500 GiB.When returned back to Command (m for help): use the letter w to write the changes.
Now, when viewing the available block devices, there should be a new partition available.
lsblk
Now that I know my disk is partitioned, I need to format it with a file system.
mkfs -t ext4 /dev/sdb1I can check that the filesystem was created using the -f switch.
lsblk -f
Notice the UUID in the output? That’s the device UUID, not the partition UUID. We will need the partition UUID to update the fstab file so that our disk automounts. The device UUID can be helpful if there are many disks and partitions listed.
blkid
Open the fstab file and add the following line.
vi /etc/fstabPARTUUID=<UUID> /etc/httpd/html ext4 rw,nouser 0 0Save and close the file.
<esc>
:wqMake sure there are no files currently in the /etc/httpd/html directory. If there are, move them to somewhere else. Mount all the devices using the fstab file to test.
mount -aVerify the file systems and mount points.
lsblk -f
Now, when we download the install bundles to /etc/httpd/html, the files will be on our new disk. This will help to prevent our OS disk from filling up and causing trouble down the line.
Offline Depot Bundles Download
For this section, I am going to use the same virtual machine that was just configured above. In an air-gapped network, an additional virtual machine may be required. The offline bundles must be downloaded from an internet connected computer and then transferred over to the air-gapped offline depot virtual machine. In my case, I am cutting out the additional virtual machine and just using the one. The steps will be the same up until the file transfer.
Install tar
On the Photon 5 OS virtual machine, install tar.
tdnf install -y tarDownload the Bundle Transfer Utility
Download the bundle transfer utility (lcm-tools-prod.tar.gz) from support.broadcom.com and transfer it to the offline depot virtual machine to /opt/vmware/vcf/lcm/lcm-tools.
mkdir -p /opt/vmware/vcf/lcm/lcm-tools
cd /opt/vmware/vcf/lcm/lcm-tools
Extract the file.
tar -xvf lcm-tools-prod.tar.gzChange into the bin directory.
cd bin
Make the lcm-bundle-transfer-util file executable.
chmod u+x lcm-bundle-transfer-util
Edit the application-prod.properties file
Prerequisite step required here. Ensure you have already generated your unique token and have it available (VCF Authenticated Downloads). The next few steps are going to require editing the application-prod.properties file.
Open the application-prod.properties file to begin editing.
vi /opt/vmware/vcf/lcm/lcm-tools/conf/application-prod.propertiesLet’s get some line numbers to make this easier.
:set numberLines 25, 26, 27, 39, and 43 need to be edited. I will include the original text as well as updated sample text. If desired, make a backup of the entire file before editing.
Line 25
lcm.depot.adapter.host=depot.vmware.comlcm.depot.adapter.host=dl.broadcom.comLine 26
lcm.depot.adapter.remote.rootDir=/PROD2Replace <downloadToken> with the token you received from the Broadcom Support portal.
lcm.depot.adapter.remote.rootDir=/<downloadToken>/PRODLine 27
lcm.depot.adapter.remote.repoDir=/evo/vmwlcm.depot.adapter.remote.repoDir=/COMP/SDDC_MANAGER_VCFLine 39
lcm.depot.adapter.remote.lcmManifestDir=/evo/vmw/lcm/manifestlcm.depot.adapter.remote.lcmManifestDir=/COMP/SDDC_MANAGER_VCF/lcm/manifestLine 43
lcm.depot.adapter.remote.bundletransferconfig.repoDir:/evo/vmw/obtulcm.depot.adapter.remote.bundletransferconfig.repoDir:/COMP/SDDC_MANAGER_VCF/obtuSave the file.
<esc>
:wqSet up the Offline Depot
Create a plain text password file and make it readable to only the user running the bundle transfer utility. (yes, this is real)
vi ~/obtu-user-passwordchmod 400 ~/obtu-user-passwordPrepare to download the bundles required to upgrade VMware Cloud Foundation.
cd /opt/vmware/vcf/lcm/lcm-tools/binHere is the syntax.
./lcm-bundle-transfer-util --setUpOfflineDepot -sv vcf-source-version --offlineDepotRootDir offline-depot-root-dir --offlineDepotUrl url:port --depotUser user-name --depotUserPasswordFile path-to-password-fileHere is an example.
./lcm-bundle-transfer-util --setUpOfflineDepot -sv 5.2.0.0 --offlineDepotRootDir /etc/httpd/html --offlineDepotUrl https://offline-depot.aaronrombaut.com:443 --depotUser [email protected] --depotUserPasswordFile ~/obtu-user-passwordHere is a documentation example. One thing to note is that the --sourceVersion or -sv parameter is supposed to define the version that will be downloaded. However, while I am trying to download patches for 5.2.1.0, the tool will report back that there are no new bundles. If I specify the version as 5.2.0.0, I will get back the updates I am looking for.
./lcm-bundle-transfer-util --setUpOfflineDepot -sv 5.0.0.0 --offlineDepotRootDir /var/www --offlineDepotUrl https://10.123.456.78:8282 --depotUser [email protected] --depotUserPasswordFile ../vmw-depot Here is a sample list of bundles that will be downloaded when you specify the source version as 5.2.0.0. Take note of the bundle names—although the product version is listed as 5.2.1.0, the list also includes patches for SDDC Manager 5.2.1.1 and 5.2.1.2.
This can be a bit confusing. I initially tried using 5.2.1.2 as the value for the -sv (source version) parameter, but that consistently returned no results. Using 5.2.0.0 instead resolved the issue and returned the expected bundles.
-------------------------------------------------------------------------------------------------------------------------------------------------
Bundle | Product Version | Bundle Size | Bundle Component | Bundle Type
-------------------------------------------------------------------------------------------------------------------------------------------------
bundle-133762 | 5.2.1.0 | 606.4 MB | ESX_HOST-8.0.3-24280767 | PATCH
bundle-133763 | 5.2.1.0 | 8895.2 MB | NSX_T_MANAGER-4.2.1.0.0-24304122 | PATCH
bundle-202281 | 5.2.1.0 | 2364.8 MB | SDDC_MANAGER_VCF-5.2.1.1-24397777 | PATCH
bundle-214558 | 5.2.1.0 | 2375.9 MB | SDDC_MANAGER_VCF-5.2.1.2-24690695 | PATCH
bundle-133760 | 5.2.1.0 | 2364.8 MB | SDDC_MANAGER_VCF-5.2.1.0-24307856 | PATCH
bundle-133761 | 5.2.1.0 | 0.0 MB | SDDC_MANAGER_VCF-5.2.1.0-24307856 | PATCH (Drift)
bundle-133765 | 5.2.1.0 | 18817.0 MB | VCENTER-8.0.3.00300-24305161 | PATCH
bundle-130870 | 5.2.1.0 | 4238.5 MB | NSX_ALB-22.1.7-24190832 | INSTALL
bundle-133764 | 5.2.1.0 | 11636.7 MB | NSX_T_MANAGER-4.2.1.0.0-24304122 | INSTALL
bundle-133766 | 5.2.1.0 | 11832.3 MB | VCENTER-8.0.3.00300-24305161 | INSTALL
-------------------------------------------------------------------------------------------------------------------------------------------------
Once the downloads complete, copy the files to the document root or if they were downloaded directly, view the directory.
ls -la /etc/httpd/html/Notice your directory will use your token id where I have highlighted the <download token>.
root@offline-depot [ /opt/vmware/vcf/lcm/lcm-tools/bin ]# tree /etc/httpd/html
/etc/httpd/html
└── <download token>
└── PROD
├── COMP
│ └── SDDC_MANAGER_VCF
│ ├── bundles
│ │ ├── bundle-130870.tar
│ │ ├── bundle-133760.tar
│ │ ├── bundle-133761.tar
│ │ ├── bundle-133762.tar
│ │ ├── bundle-133763.tar
│ │ ├── bundle-133764.tar
│ │ ├── bundle-133765.tar
│ │ ├── bundle-133766.tar
│ │ ├── bundle-202281.tar
│ │ └── bundle-214558.tar
│ ├── Compatibility
│ │ └── VmwareCompatibilityData.json
│ ├── deltaFileDownloaded
│ ├── deltaFileDownloaded.md5
│ ├── index.v3
│ ├── lcm
│ │ └── manifest
│ │ └── v1
│ │ └── lcmManifest.json
│ ├── manifests
│ │ ├── bundle-130870.manifest
│ │ ├── bundle-130870.manifest.sig
│ │ ├── bundle-133760.manifest
│ │ ├── bundle-133760.manifest.sig
│ │ ├── bundle-133761.manifest
│ │ ├── bundle-133761.manifest.sig
│ │ ├── bundle-133762.manifest
│ │ ├── bundle-133762.manifest.sig
│ │ ├── bundle-133763.manifest
│ │ ├── bundle-133763.manifest.sig
│ │ ├── bundle-133764.manifest
│ │ ├── bundle-133764.manifest.sig
│ │ ├── bundle-133765.manifest
│ │ ├── bundle-133765.manifest.sig
│ │ ├── bundle-133766.manifest
│ │ ├── bundle-133766.manifest.sig
│ │ ├── bundle-202281.manifest
│ │ ├── bundle-202281.manifest.sig
│ │ ├── bundle-214558.manifest
│ │ └── bundle-214558.manifest.sig
│ └── tmp
│ ├── index.v3
│ └── lcmManifest.json
└── vsan
└── hcl
├── all.json
└── lastupdatedtime.json
13 directories, 39 filesSet the correct file permissions to the files or if the cron job was set, this will likely already have taken place.
chown apache -R /etc/httpd/html/Connect SDDC Manager to the Offline Depot
Note: Make sure the TLS certifcates are configured for both the Offline Depot and the SDDC Manager. Otherwise, it is likely a Secure protocol communication error will be logged. You can tail the lcm-debug.log on SDDC Manager for more details if there are issues connecting to the depot. In the UI, I received the message, “Failed to connect to VMware depot with the provided user credentials. Cause: {0}.” Turns out the credentials are correct, the lack of trust between the two servers is actually the cause.
tail -f /var/log/vmware/vcf/lcm/lcm-debug.log
In the navigation pane, click Administration > Depot Settings.
Connect SDDC Manager to the Offline Depot by clicking the Set Up button.
Enter the appropriate details for the offline depot.
You can view the lcm-debug.log to view what’s going on in the background since the Tasks won’t populate. Be patient here as the Bundle Management populates.
Conclusion
By following this guide, you’ve successfully set up an Offline Depot for VMware Cloud Foundation (VCF), enabling patch and bundle management in air-gapped or restricted environments. This configuration allows SDDC Manager to pull upgrade bundles from a secure, local source without requiring internet access, ensuring compliance, control, and operational continuity across isolated VCF deployments.
While the setup process is intricate and requires command-line proficiency, the end result significantly streamlines bundle distribution, especially in environments with multiple VCF instances. With the Offline Depot in place, your organization is now better positioned to manage VCF updates reliably and securely—even in the most disconnected networks.
Leave a Reply