Overview

If the SDDC Manager appliance is not connected to an online or Offline Depot, you can use the VCF Download Tool to download the bundles to a different computer and then upload them to the SDDC Manager appliance.

Moreover, in cases where an organization comprises numerous VMware Cloud Foundation (VCF) instances or operates within air-gapped networks, the implementation of an Offline Depot may be advisable. The utilization of an Offline Depot enhances the experience for air-gapped customers by minimizing the number of steps required to distribute artifacts across multiple VCF instances repeatedly.

RTFM

VCF authenticated downloads configuration update instructions

Set Up an Offline Depot Web Server for VMware Cloud Foundation

Use the VMware Cloud Foundation Download Tool to Download Install Binaries

Using the VCF Download Tool

Download Install Binaries to an Offline Depot

Download Install Binaries from an Online or Offline Depot

Warnings and Assumptions

First, the warning. This whole procedure is not for the faint of heart. If you are comfortable on the command line, you will likely succeed. Hopefully, in the future, this process will be streamlined and we can laugh about this absurd process someday.

Now, on to the assumptions.

• know what VMware Cloud Foundation (VCF) is and what it does
• understand basic virtualization concepts
• working SDDC Manager
• generated your download token with Broadcom support (link)
• understand how to edit files on the command line

I also hope you have a good heartbeat and a greater sense of humor!

Build an Offline Depot

To get started with an Offline Depot, we need a virtual machine to host the upgrade bundles.

Deploy a Virtual Machine

I am going to use a minimal Photon OS 5.0 GA image for the virtual machine to build the Offline Depot. The following steps are fairly basic and self-explanatory. I will include screenshots and applicable comments, when needed, for each screen.

My plan is to install the operating system to /dev/sda and then use /dev/sdb for package storage.

For the Network Configuration, I am configuring a static IP address.

I could not figure out how to put two Nameserver values here, so I left just the one. I tried comma separated and space separated, but neither worked. I will add the additional DNS server later.

Since this virtual machine is running on VMware ESXi, I choose the VMware hypervisor optimized Linux kernel.

Configure the Operating System

Before hosting the VCF Offline Depot, prepare the operating system by updating packages, installing required components, provisioning storage for repository content, creating and securing the Apache document root, opening the firewall for HTTPS, and—if needed—configuring DNS and temporarily enabling root SSH access for initial setup tasks.

• Temporarily enable root SSH access.
• (Optional) Configure an additional DNS server.
• Update the system
• Install required packages
• Attach and mount additional storage
• Open Firewall ports
• Configure TCP keepalive parameters

Temporarily enable root SSH access

Direct SSH login as root is disabled by default. For the initial setup, enable root SSH access temporarily to simplify copying commands, then disable it again when finished. From a console session as root, edit /etc/ssh/sshd_config to permit root login, reload the SSH service, complete the required steps, and finally restore the original setting and reload the service once more.

Start by opening a console window and logging in as root. Then, let’s edit the sshd_config file to make the necessary changes.

vi /etc/ssh/sshd_config +32
Shift + A

Modify the parameter to be yes.

Save and close the configuration file.

Esc
:wq

Restart the sshd.service service and check the status.

systemctl restart sshd.service
systemctl status sshd.service

(Optional) Configure an additional DNS server

Modify the 50-static-en.network file in the /etc/systemd/network/ directory.

vi /etc/systemd/network/50-static-en.network +7Shift + A

On line 7, DNS, add an additional entry, but no more than two!

  1 [Match]
  2 Name=eth0
  3 
  4 [Network]
  5 IPv6AcceptRA=no
  6 Address=172.16.5.2/24
  7 DNS=172.17.0.10 172.17.0.11
  8 Gateway=172.16.5.254

Save and close the configuration file.

Esc
:wq

Restart the systemd-networkd.service service.

systemctl restart systemd-networkd.service

Check the DNS Servers configuration for Link 2 (eth0).

resolvectl status

Update the System

Update the system.

tdnf update --assumeyes

Reboot the system.

reboot

Install required packages

There are a few packages that are needed to configure the Offline Depot.

tdnf install httpd tar tree --assumeyes

Partition and format additional storage

Identify the new disk (example: /dev/sdb).

lsblk -f

Partition the entire disk.

fdisk /dev/sdb

Add a new partition — n.

Welcome to fdisk (util-linux 2.38).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x2d39bf94.

Command (m for help): n

Partition type is primary — p.

Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): 
Using default response p.

Partition number 1.

Partition number (1-4, default 1):

First sector is default.

First sector (2048-2147483647, default 2048):

Last sector is default.

Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2147483647, default 2147483647): 

Summary of operations completed.

Created a new partition 1 of type 'Linux' and of size 1024 GiB.

When returned back to Command (m for help): use the letter w to write the changes.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Format the disk with a file system.

mkfs -t ext4 /dev/sdb1

Attach and mount additional storage

Create a mount point that will be used for the apache DocumentRoot directive. This directory will contain all the install binaries and metadata.

mkdir -p /var/www/html

Retrieve the partition UUID.

UUID=$(blkid -s PARTUUID -o value /dev/sdb1)

Add the disk to fstab.

echo "PARTUUID=${UUID} /var/www/html ext4 defaults 0 2" >> /etc/fstab

Make sure there are no files currently in the /var/www/html directory. If there are, move them to somewhere else.

Mount and verify.

mount -a

df -h /var/www/html

Open Firewall Ports

Append an allow rule for https traffic to iptables.

iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

Verify the iptables rules.

iptables -L

Make the rules persistent.

iptables-save > /etc/systemd/scripts/ip4save

This opens the firewall port, 443, but Apache is not configured to server https content yet, so the webpage is not accessible.

Configure TCP keepalive parameters

TCP keepalive is a mechanism that sends a small, empty packet every so often just to keep the connection alive.

Edit or create the user’s .ssh configuration.

vi ~/.ssh/config

Add the following parameters to the file. The host can be specified if needed.

Host *
    ServerAliveInterval 60
    ServerAliveCountMax 5

ServerAliveInterval 60: every 60 seconds, the SSH client sends a keepalive packet.

ServerAliveCountMax 5: if 5 keepalive packets go unanswered (i.e., 5 minutes of silence), the SSH session ends cleanly instead of hanging forever.

Save and close the file.

Esc
:wq

This keeps the SSH connection active and recovers gracefully if the remote host truly disappears.

Configure the Apache Server

Certificates

Create a temporary directory to prepare the certificate.

mkdir /root/http-certificates

Change to the new directory.

cd /root/http-certificates

Generate a private key.

openssl genpkey -out /root/http-certificates/server.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048

Make a configuration file containing Subject Alternative Name (SAN) information. Modern browsers will not trust the certificate otherwise.

vi offline-depot.cnf

[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
C = <country>
ST = <state>
L = <locality>
O = <organization>
OU = <organizational unit>
CN = <fully-qualified domain name>

[req_ext]
subjectAltName = @alt_names

[alt_names]
IP.1 = <ip address>
DNS.1 = <fully-qualified domain name>
DNS.2 = <short name>

Generate the Certificate Signing Request (CSR) using the private key and the configuration file.

openssl req -new -key /root/http-certificates/server.key -out /root/http-certificates/request.csr -config /root/http-certificates/offline-depot.cnf

Have the Certificate Signing Request (CSR) signed by your Certificate Authority (CA). This will depend on your organization’s policy. Once the certificate request is signed, upload the machine (machine.crt), intermediate (if applicable), and root (root.crt) certificates, in PEM format and encoded in Base64, to the /root/http-certificates directory.

Combine the files, top-to-bottom, starting with the <machine certificate>, <intermediate certificate>, then <root certificate> into a file called server.crt.

cat machine.crt >> server.crt
cat intermediate.crt >> server.crt
cat root.crt >> server.crt

Open the file, remove any spaces between the certificates if necessary, and then save it. Move the server key and certificate to the /etc/httpd/conf directory.

mv /root/http-certificates/server.* /etc/httpd/conf

Fix the mode and permissions on the two files.

chmod 0400 /etc/httpd/conf/server.crt && chown root:root /etc/httpd/conf/server.crt

chmod 0400 /etc/httpd/conf/server.key && chown root:root /etc/httpd/conf/server.key

Configure the httpd.conf file

This /etc/httpd/conf/httpd.conf file is the main configuration file for the Apache Web server.

sed -i 's|#LoadModule ssl_module|LoadModule ssl_module|' /etc/httpd/conf/httpd.conf

sed -i 's|#LoadModule socache_shmcb_module|LoadModule socache_shmcb_module|' /etc/httpd/conf/httpd.conf 

sed -i 's|#Include conf/extra/httpd-ssl.conf|Include conf/extra/httpd-ssl.conf|' /etc/httpd/conf/httpd.conf

Configure the httpd-ssl.conf file

sed -i 's|DocumentRoot "/etc/httpd/html"|DocumentRoot "/var/www/html"|' /etc/httpd/conf/extra/httpd-ssl.conf

Replace the ServerAdmin email address.

sed -i 's|ServerAdmin [email protected]|ServerAdmin [email protected]|' /etc/httpd/conf/extra/httpd-ssl.conf

Replace the ServerName with the actual host name for the server.

sed -i 's|ServerName www.example.com:443|ServerName depot.rainpole.io:443|' /etc/httpd/conf/extra/httpd-ssl.conf

Open and edit the /etc/httpd/conf/extra/httpd-ssl.conf file.

vi /etc/httpd/conf/extra/httpd-ssl.conf +290

Insert the following lines before the </VirtualHost> tag (use :set number in vi to view line numbers).

<Directory /var/www/html/PROD/COMP>
    AuthType Basic
    AuthName "Basic Authentication"
    AuthUserFile /etc/httpd/conf/.htpasswd
    require valid-user
</Directory>
<Directory /var/www/html/PROD/metadata>
    AuthType Basic
    AuthName "Basic Authentication"
    AuthUserFile /etc/httpd/conf/.htpasswd
    require valid-user
</Directory>
<Directory "/var/www/html/PROD/COMP/Compatibility/VxrailCompatibilityData.json">
    # VxRail VVS Cookie Validation (VCF 5.0)
    <If "%{HTTP:Cookie} == 'ngssosession=ngsso-token' ">
    Require all granted
    </If>
</Directory>
<Directory /var/www/html/PROD/vsan/hcl>
    Require all granted
</Directory>
    # Those Alias statements are needed only for VCF 5.1.0.0.
    Alias /products/v1/bundles/lastupdatedtime /var/www/html/PROD/vsan/hcl/lastupdatedtime.json
    Alias /products/v1/bundles/all /var/www/html/PROD/vsan/hcl/all.json
    # Needed only if UMDS downloads are presented
<Directory /var/www/html/umds-patch-store>
    Require all granted
</Directory>

Save and close the file.

Esc
:wq

Configure Basic Authentication

Generate a user and password for use with the Offline Depot. If more than one user will be created, remove the -c after the initial file creation.

htpasswd -c /etc/httpd/conf/.htpasswd <username>

Protect the file by modifying the file’s permissions.

chown apache /etc/httpd/conf/.htpasswd
chmod 0400 /etc/httpd/conf/.htpasswd

---

Enable and Start the httpd Service

Before enabling and starting the service, perform a quick configuration test of the Apache configuration:

apachectl configtest

If Syntax OK is returned, the configuration is valid and the service should start successfully.

Enable and start the service:

systemctl enable httpd.service --now

Verify the service status:

systemctl status httpd.service

At this point, it’s also a good idea to reboot the virtual machine to validate the configuration. After the reboot, confirm that the httpd service starts automatically and remains in a healthy state.

Clean Up Tasks

Remove the directory used to generate the certificates.

rm -rf /root/http-certificates

Remove the default index file if it exists.

rm -rf /var/www/html/index.html

If PermitRootLogin was used for SSH, it is best to set that back to no.

vi /etc/ssh/sshd_config

Use the VCF Download Tool to Download Install Binaries

For this section, I am going to use the same virtual machine that was just configured above. In an air-gapped network, an additional virtual machine may be required. The offline bundles must be downloaded from an internet connected computer and then transferred over to the air-gapped Offline Depot virtual machine. In my case, I am cutting out the additional virtual machine and just using the one. The steps will be the same up until the file transfer.

Create a Download Token File

We need to create a text file to store the download token. This token is the obtained from the Broadcom Support Portal.

vi ~/download-token.txt

Download and Ready the VCF Download Tool

Download the VCF Download Tool (lcm-download-tool-<version>.<build number>.tar.gz) from support.broadcom.com and transfer it to the Offline Depot virtual machine.

scp vcf-download-tool-9.0.1.0.24962179.tar.gz root@<Offline Depot FQDN>:/root/

Make a new directory to extract the files into.

mkdir ~/vcf-download-tool
tar -xvf vcf-download-tool-9.0.1.0.24962179.tar.gz -C ~/vcf-download-tool/

Change into the vcf-download-tool/bin directory.

cd vcf-download-tool/bin

Download Install Binaries

Here is the command to run to download the install binaries to the web server’s root directory (/var/www/html/). Note here that --vcf-version is 9.0.1 at the time of this writing, please check release notes for updated versions when they are available.

(ref: VCF Download Tool Binaries Commands and Options)

./vcf-download-tool binaries download --vcf-version 9.0.1 --automated-install --depot-store /var/www/html --depot-download-token-file ~/download-token.txt

Answer the question to enable Customer Experience Improvement Program (CEIP).

*********Welcome to VMware Cloud Foundation Download Tool***********

Version: 9.0.1.0.24962179
Do you want to enable CEIP(Y/N)?

If all goes well, metadata and binaries will start to download.

Validating depot credentials.
Depot credentials are valid.
Downloading unified release manifest file.
Successfully downloaded unified release manifest file.
Downloading product version catalog file.
Successfully downloaded product version catalog file.
Downloading vSAN HCL file.
Successfully downloaded vSAN HCL file.
Downloading compatibility data.
Successfully downloaded compatibility data.

Fix Ownership and Permissions

Since all the work has been performed as the root user, all the directories and files are owned by root. The Apache web server is run by the user apache, so ownership and permissions must be changed before connecting the VCF Installer to the Offline Depot.

Change the ownership of the /var/www/html directory.

chown apache -R /var/www

Set the proper permissions on the directories and files.

find /var/www/html -type d -exec chmod 0500 {} \;

find /var/www/html -type f -exec chmod 0400 {} \;

Test the permissions with the following file. This is the file the VCF Installer will try to connect to. Ensure all the directories below /var are owned by apache and the file permissions are correct.

namei -l /var/www/html/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json

Directories should be octal 0500 and will be represented by dr-x------.

Files should be octal 0400 and will be represented by -r--------.

f: /var/www/html/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json
drwxr-xr-x root   root /
drwxr-xr-x root   root var
drwxr-x--- apache root www
dr-x------ apache root html
dr-x------ apache root PROD
dr-x------ apache root metadata
dr-x------ apache root productVersionCatalog
dr-x------ apache root v1
-r-------- apache root productVersionCatalog.json

Secure protocol communication error

When trying to connect to the Offline Depot, if the VCF Installer is not secured with TLS certificates (this is common when first installed), then there will be an issue connecting.

Unfortunately, there is not a way to trust or add certificates to the VCF Installer in the UI. This will have to be accomplished on the command line.

1. Make an ssh connection to the VCF Installer appliance with the vcf user.

2. Switch to the root user with su.

3. Obtain the Offline Depot certificate.

openssl s_client -connect <offline-depot-host>:443 -showcerts </dev/null 2>/dev/null | openssl x509 -outform PEM > /tmp/offline-depot.crt

4. Use the Java keytool to import the certificate into the trust chain.

keytool -import -trustcacerts -file /tmp/offline-depot.crt -keystore /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts -alias <offline-depot-fqdn>

5. Enter the keystore password as changeit. (Yes, that is the actual password!) When asked to Trust this certificate? [no], replace with yes.

6. Restart the VCF Installer services.

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

When asked if sure to continue, press the letter y. Wait a few minutes before trying to test the connection.

Connect VCF Installer to Offline Depot

(ref: Connect to an Offline Depot to Download Binaries)

tail -f /var/log/vmware/vcf/lcm/lcm-debug.log | grep -i 'Secure protocol communication error'

1. In a web browser, log in to the VMware Cloud Foundation Installer appliance administration interface: https://installer_appliance_FQDN. Enter the admin@local user and password you provided when you deployed the appliance and then click Log In.

2. Click Depot Settings and Binary Management.

3. Click Configure to connect to the offline depot.

4. Enter the following information for the offline depot:

• FQDN or IP address
• Port
• User name
• Password

5. Click Configure.

You are now ready to download binaries.

The Depot Settings should update as shown below.

Back on the VCF Installer homepage, there should now be a message that shows the Offline depot connection is active.

Troubleshooting

Path not found

The following error, Path not found – /metadata/productVersionCatalog/v1/productVersionCatalog.json
404 Not Found HTTP/1.1, is usually due to the web server not serving files or serving the wrong files.

Try navigating to the web server and see what comes up. When I tested, I saw the default, It works! message. This indicated to me that I forgot to update the DocumentRoot directive or forgot to restart the apache service after updating. Once I checked the apache configuration and updated, I refreshed the browser to be greeted by this.

In this case, this is actually a good indicator the VCF Installer will connect.

Another possibility is that a permission is not set correct.

namei -l /var/www/html/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json

Look at /var/www, it is still set to root.

f: /var/www/html/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json
drwxr-xr-x root   root /
drwxr-xr-x root   root var
drwxr-x--- root   root www
dr-x------ apache root html
dr-x------ apache root PROD
dr-x------ apache root metadata
dr-x------ apache root productVersionCatalog
dr-x------ apache root v1
-r-------- apache root productVersionCatalog.json

Modify the owner of the directories.

chown -R apache /var/www

Set the proper permissions on the directories and files.

find /var/www/html -type d -exec chmod 0500 {} \;

find /var/www/html -type f -exec chmod 0400 {} \;

Now when checking the permissions, apache owns the /var/www/ directory.

f: /var/www/html/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json
drwxr-xr-x root   root /
drwxr-xr-x root   root var
drwxr-x--- apache root www
dr-x------ apache root html
dr-x------ apache root PROD
dr-x------ apache root metadata
dr-x------ apache root productVersionCatalog
dr-x------ apache root v1
-r-------- apache root productVersionCatalog.json

Internal error while validating credentials

Here is another error that may show up.

This occurs when the .htaccess file is either not found or doesn’t have the correct permissions. Try following the error_log to see if this is the case.

tail -f /var/log/httpd/error_log

For this error, I modified the /etc/httpd/conf/httpd.conf file, but didn’t modify anything in the /etc/httpd/conf/extra/httpd-ssl.conf.

Remember when I changed the DocumentRoot to a different directory? It needs to be changed here as well. Look at the error above and compare it to the DocumentRoot in this configuration file. They match, but that’s not where I am hosting the binaries. Also, I see the error, “server certificate does NOT include an ID which matches the server name”, which of course is because the configuration file is still referencing a generic www.example.com. Let’s update both lines and restart the httpd.service service.

Unable to execute VCF Download Tool

Check the permissions of the folders using “ls -la”

Run the following to change permissions to allow both read and execute

chmod -R 755 /etc/httpd/html/vcf-download-tool

run “ls -la” again to verify new permissions

Then you will be able to execute the download

Conclusion

By following this guide, you’ve successfully set up an Offline Depot for VMware Cloud Foundation (VCF), enabling patch and bundle management in air-gapped or restricted environments. This configuration allows SDDC Manager to pull upgrade bundles from a secure, local source without requiring internet access, ensuring compliance, control, and operational continuity across isolated VCF deployments.

While the setup process is intricate and requires command-line proficiency, the end result significantly streamlines bundle distribution, especially in environments with multiple VCF instances. With the Offline Depot in place, your organization is now better positioned to manage VCF updates reliably and securely—even in the most disconnected networks.