Nested Labs: Creating Identical Isolated Nested Environments

LifeOfBrianOC and I recently had cause to build out some new lab infrastructure to host nested ESXi VMs. As part of that build we wanted to be able to deploy identical nested testbeds i.e multiple testbeds, each with their own set of nested hosts and supporting infrastructure, each of which would use the same VLAN and IP subnets but still be accessible from external jump boxes via a virtualized router.

Before this we had been using trunk portgroups in our ‘Infrastructure’ vCenter, but that meant that each VLAN needed to exist on the physical network and each testbed needed to be unique in terms of the VLAN and IP Subnet configuration.

We settled on using NSX-T Segments (with Mac learning). Each testbed uses its own isolated NSX-T segment so that VMs connected to those segments can be complete duplicates of each other and not cause conflicts.

First, you need to setup your infrastructure vCenter and NSX-T instances (we used VMware Cloud Foundation for this part), ensuring that the Host Overlay VLAN exists on the physical network that your physical host(s) are connected to.

To setup your isolated segments you first need to set up the IP Discovery, Mac Discovery and Segment Security Profiles that each of the isolated segments will need. The screenshot below shows the required settings. The same profiles can be used by all the segments.

Then you create a segment that is not connected to any gateway (this is what ensures the isolation) specifying the profiles you previously created. Create as many segments as you want testbeds (we have automation that creates a segment and attaches the profiles as we deploy a new testbed). Note: we found that if you change the settings in a profile, VMs attached to that segment don’t necessarily see the benefit of the change. We found it best to recreate the segment after changing the profiles.

Once you have that in place you are ready to start deploying your nested testbeds. With nested ESXi hosts you need to ensure that the Management VMkernel uses a different MAC address to that of the first NIC assigned to the host. By default they are same.

To get around that we deploy our nested hosts, and then alter the MAC address of the first NIC in the VM i.e. after the VMkernel has picked up the original MAC address of the first NIC. You could achieve in a different way if desired, as long as the two are different.

Nested hosts can then be configured as normal i.e. setting VLANs etc on the portgroups within the nested hosts will VLAN tag traffic that is traversing the isolated NSX-T segment.

If you are connecting regular VMs directly to the NSX-T segment you will need to configure in-guest VLAN tagging in order for it communicate with the nested hosts and VMs that are deployed to the nested hosts. In our case we use the VMXNET 3 NIC settings on our Windows Active Directory VMs to configure the VLAN.

You can also do it via PowerShell on the VM if you want

Get-NetAdapterAdvancedProperty Ethernet0 -DisplayName "VLAN ID" | Set-NetAdapterAdvancedProperty -DisplayValue 1611 -RegistryValue 1611

For our Linux VMs we use sub-interface VLAN tagging on our linux boxes to get the traffic flowing correctly. This is an example from our virtualized router.

# eth0 : Primary interface used to uplink to Public Network
# **********************************************************

auto eth0
iface eth0 inet static
address 192.168.15.20
netmask 255.255.255.0
network 192.168.15.0
broadcast 192.168.15.255
mtu 1600
gateway 192.168.15.1

# eth1 : Interface used for Environment VLANs
# **************************************************
auto eth1
iface eth1 inet manual
pre-up ifconfig eth1 mtu 8900

# Management Network
auto eth1.1611
iface eth1.1611 inet static
address 172.16.11.1
netmask 255.255.255.0
mtu 8900

You may need to figure out how to VLAN tag your own VMs based on their individual capabilities, but every OS has a way to do it.

The lab jumpboxes we use to access the nested labs use static routes that specify the virtualized router address used to access any particular environment. Architecturally, that looks like this:

Our virtualised router (lab1-ptr and lab2-ptr in diagram) are simple Debian appliances with two interfaces. The first is connected to the public network, the second is connected to the isolated NSX-T Segment for the nested testbed it is part of. Our jumpbox static routes (one entry per network you want to access) are configured to point to the public network address of that router. You can have multiple jumpboxes to access different testbeds, or you can dynamically alter the static routes on the same jumpbox to get to the desired networks via a different router as required (we do the latter, using automation to flip between them as needed).

Because our physical network is configured with jumbo frames of 9000, we shrunk the MTU size inside the nested labs to 8900 to allow for the overhead of NSX-T overlay traffic at the Infrastructure NSX-T layer.

As you can see from the diagram, we also deploy CloudBuilder onto the isolated segment, so we can successfully do nested VCF deployments in this configuration 👍