Nested VCF Lab: Networking

Table of Contents

Overview

The networking component for the nested VCF lab is very important. There are certain things that must be configured for a successful deployment. These settings do not apply in a production scenario where the physical hosts are connected to Top-of-Rack Switches. Doing so could open up a serious security situation.

On the physical host.

The Virtual Switch (vSwitch0) should have the Maximum Transmission Unit (MTU) set to 9000 (also known as Jumbo Frames)

The Security settings (Promiscuous mode, MAC address changes, and Forged transmits) on vSwitch0 should all be set to Accept

Nested Firewall

I use a Firewalla Gold for a home firewall. This is a great firewall…for home use. I use this primarily for home, but I also want to be able to have a home lab that closely resembles an Enterprise network. Unfortunately, this device does not have a way to easily adjust the MTU or have native Border Gateway Protocol (BGP) support. For this reason, I end up installing a pfSense firewall on my physical ESXi host.

For this to work, I add two VMXNET3 Network Adapters. One is on my home lab network, VLAN 92, and the other adapter is configured with a VLAN ID of 4095 and labeled All-VLAN IDs. In VMware, VLAN 4095 enables Virtual Guest Tagging (VGT) and means the virtual machine handles the VLAN traffic.

Lab Physical Diagram

pfSense to Physical ESXi Logical Diagram

Create the pfSense Interfaces

Take the Deployment Parameter workbook and create the interfaces in pfSense. Make sure to adjust the MTU on the LAN interface to 9000 before adjusting the MTU of the sub-interfaces.

Update: The deployment parameters Management Domain Network in VCF 5.1.0 broke the management networks up. Now there is a VM Management Network which is used for all the management guests, such as SDDC Manager, vCenter Server Appliance, and the NSX Managers. Then there is a Management Network which is intended to be used for the ESXi host kernel interfaces (vmk0).

Create a DHCP Scope for the NSX-T Host Overlay Network

DHCP is recommended for the NSX-T Host Overlay network. I use /24 networks, and since there are only four nested hosts, I just open the range up from 100 to 199.

Permit Traffic to the Management Gateway

When I ran the VCF configuration validation, I kept receiving a warning that the gateway was not contactable. I logged into the console of the Cloud Builder appliance and verified that I actually could. So I checked the firewall logs and saw the traffic being blocked. This was also confirmed in the VCF bring-up log.

The VCF bring-up log can be found at:

/var/log/vmware/vcf/bringup/vcf-bringup.log

Cloud Builder Validation

I thought this was quite odd behavior. What administrator opens up these ports or has anything listening on the router interface at these addresses? Huh, interesting…

Anyways, I added an alias in pfSense and allowed those ports on the appropriate interfaces. The next section will provide more detail on the pfSense.

pfSense Configuration

I am using a nested pfSense firewall in my lab due to the shortcomings of my edge firewall, unfortunately. The pains of home labs, I suppose. Either way, pfSense is quite impressive in this role and does a great job without consuming too many resources. Probably the most frustrating aspect of this set up are the firewall rules. This is not a pfSense issue. This is a security detail. Sure, I could open my firewall wide open, but who in their security conscience mind would do such a thing?

Note: The following has been updated for VMware VCF 5.1.0 and may not be appropriate for your version.

Firewall Aliases

Aliases in pfSense are a way to provide a friendly name to a group of either IPs, Ports, or URLs. As you can see in the image below, instead of using IP addresses for my VMware ESXi hosts, I added an alias with the name, ESXiHosts, and provided all the ESXi hosts in my lab. This will save me on my rulesets as well. Now I can reference one line instead of many for the same task. Additionally, referencing by name rather than IP address is easier to read…and in the off chance IP addresses have to be altered in the future, the rulesets will still be accurate and will not need to be altered.

Firewall Rules

The first group is Floating. This has special meaning and are rarely used. I wanted this particular rule, to be able to access TCP/443 (HTTPS) across all the interfaces. One of the differences of the Floating group is that it is bi-directional. Normally, the direction is ingress to an interface.

Nested VCF Lab: Networking

Overview