Breadcrumb Build – Region A VVD Virtual Infrastructure on VxRail Part 9: Logical Networking for the Shared Edge/Compute VxRail

Welcome to Part 9 of my VVD on VxRail Breadcrumb Build Series

Nearly there folks! Last post for the configuration of VVD on VxRail. While the topic is the same as Post 5 where we did logical networking for the management VxRail, theres’ not quite so much to do on the SEC VxRail, so this wont require quite as much coffee 🙂

Anything in the format [input_value] represents a value from your preparation that you should insert (without the square brackets of course)

NSX Dynamic Routing

Create a Universal Logical Switch for Use as the Transit Network
- Networking & Security -> Logical Switches -> New Logical Switch
  - Name: Universal Transit Network
  - Transport Zone: SEC Universal Transport Zone
  - Replication Mode: Hybrid
Networking & Security -> Logical Switches -> New Logical Switch
- Name: Global Transit Network
- Transport Zone: SEC Global Transport Zone
- Replication Mode: Hybrid

Deploy NSX Edge Devices for North-South Routing

Networking & Security -> NSX Edges -> Add
First ESG
- Name and Desc
  - Install Type: Edge Services Gateway
  - Name: [sec-esg01-edgename]
  - Deploy NSX Edges: Selected
  - Enable High Availability: Deselected
- Settings
  - User Name: admin
  - Password: [sec-nsx-edge-complex-password]
  - Enable SSH access: Selected
  - Enable FIPS Mode: Deselected
  - Enable auto rule generation: Selected
  - Edge Control Level logging: INFO
- Configure Deployment
  - Appliance Size: Large
  - Cluster/Resource Pool: regiona-rp-sddc-edge
  - Datastore: [sec-vsan-datastore]
- Configure Interfaces
  - Name: Uplink01
  - Type: Uplink
  - Connected to: Uplink01
  - Connectivity Status: Connected
  - Primary IP Address: [sec-esg01-uplink01-ip]
  - Subnet Prefix Length: [sec-uplink01-prefix-length]
  - MTU: 9000
  - Send ICMP Redirect: Selected
- Configure Interfaces
  - Name: Uplink02
  - Type: Uplink
  - Connected to: Uplink02
  - Connectivity Status: Connected
  - Primary IP Address: [sec-esg01-uplink02-ip]
  - Subnet Prefix Length: [sec-uplink02-prefix-length]
  - MTU: 9000
  - Send ICMP Redirect: Selected
- Configure Interfaces
  - Name: SEC-UDLR01
  - Type: Internal
  - Connected to: Universal Transit Network
  - Connectivity Status: Connected
  - Primary IP Address: [sec-esg01-utn-ip]
  - Subnet Prefix Length: [sec-utn-prefix-length]
  - MTU: 9000
  - Send ICMP Redirect: Selected
- Configure Interfaces
  - Name: SEC-DLR01
  - Type: Internal
  - Connected to: Global Transit Network
  - Connectivity Status: Connected
  - Primary IP Address: [sec-esg01-gtn-ip]
  - Subnet Prefix Length: [sec-gtn-prefix-length]
  - MTU: 9000
  - Send ICMP Redirect: Selected
- Default Gateway
  - Deselect Configure Default Gateway
- Firewall and HA -> Next

Networking & Security -> NSX Edges -> Add
Second ESG
- Name and Desc
  - Install Type: Edge Services Gateway
  - Name: [sec-esg02-edgename]
  - Deploy NSX Edges: Selected
  - Enable High Availability: Deselected
- Settings
  - User Name: admin
  - Password: [sec-nsx-edge-complex-password]
  - Enable SSH access: Selected
  - Enable FIPS Mode: Deselected
  - Enable auto rule generation: Selected
  - Edge Control Level logging: INFO
- Configure Deployment
  - Appliance Size: Large
  - Cluster/Resource Pool: regiona-rp-sddc-edge
  - Datastore: [sec-vsan-datastore]
- Configure Interfaces
  - Name: Uplink01
  - Type: Uplink
  - Connected to: Uplink01
  - Connectivity Status: Connected
  - Primary IP Address: [sec-esg02-uplink01-ip]
  - Subnet Prefix Length: [sec-uplink01-prefix-length]
  - MTU: 9000
  - Send ICMP Redirect: Selected
- Configure Interfaces
  - Name: Uplink02
  - Type: Uplink
  - Connected to: Uplink02
  - Connectivity Status: Connected
  - Primary IP Address: [sec-esg02-uplink02-ip]
  - Subnet Prefix Length: [sec-uplink02-prefix-length]
  - MTU: 9000
  - Send ICMP Redirect: Selected
- Configure Interfaces
  - Name: SEC-UDLR01
  - Type: Internal
  - Connected to: Universal Transit Network
  - Connectivity Status: Connected
  - Primary IP Address: [sec-esg02-utn-ip]
  - Subnet Prefix Length: [sec-utn-prefix-length]
  - MTU: 9000
  - Send ICMP Redirect: Selected
- Configure Interfaces
  - Name: SEC-DLR01
  - Type: Internal
  - Connected to: Global Transit Network
  - Connectivity Status: Connected
  - Primary IP Address: [sec-esg02-gtn-ip]
  - Subnet Prefix Length: [sec-gtn-prefix-length]
  - MTU: 9000
  - Send ICMP Redirect: Selected
- Default Gateway
  - Deselect Configure Default Gateway
- Firewall and HA -> Next

Anti-Affinity rules for N/S Edges

Select [sec-cluster] -> Configure -> VM/Host Rules -> Add
- Name: anti-affinity-rule-ecmpedges
- Enable rule: Selected
- Type: Separate Virtual Machine
- Add both [sec-esg01-edgename] and [sec-esg02-edgename]

Disable the Firewall Service

Networking & Security -> NSX Edges -> Double Click [sec-esg01-edgename]-> Manage -> Firewall -> Stop -> Publish
Repeat for [sec-esg02-edgename]

Enable and Configure Routing

For [sec-esg01-edgename]
- Networking & Security -> NSX Edges -> Double Click Edge -> Manage -> Routing -> Global Configuration -> Start ECMP
- Dynamic Routing Configuration -> Edit -> Router ID = Uplink01 -> OK – >Publish
- Routing -> Static Routes – Add
  - Network: [sec-udlr-compute-network]/[sec-udlr-compute-network-prefix-length]
  - Next Hop: [sec-udlr-utn-uplink-ip]
  - Interface: SEC-UDLR
  - Admin Distance 210
- Routing -> Static Routes – Add
  - Network: [sec-dlr-compute-network]/[sec-dlr-compute-network-prefix-length]
  - Next Hop: [sec-dlr-gtn-uplink-ip]
  - Interface: SEC-DLR
  - Admin Distance 210
- Publish
- Routing -> BGP -> Edit
  - Enable BGP: Selected
  - Enable Graceful Restart: Selected
  - Enable Default Originate: Deselected
  - Local AS: [sec-esg01-local-as]
- BGP -> Add
  - Neighbour 1 (TOR1)
    - IP Address: [tor1-bgp-ip]
    - Remote AS: [tor1-local-as]
    - Weight: 60
    - Keep Alive Time: 4
    - Hold Down Time: 12
    - Password: [tor1-bgp-password]
  - Neighbour 2 (TOR2)
    - IP Address: [tor2-bgp-ip]
    - Remote AS : [tor2-local-as]
    - Weight: 60
    - Keep Alive Time: 4
    - Hold Down Time: 12
    - Password: [tor2-bgp-password]
  - Neighbour 3 (UDLR)
    - IP Address: [sec-udlr-bgp-protocol-ip]
    - Remote AS: [sec-udlr-local-as]
    - Weight: 60
    - Keep Alive Time: 1
    - Hold Down Time: 3
    - Password: [sec-udlr-bgp-password]
  - Neighbour 3 (DLR)
    - IP Address: [sec-dlr-bgp-protocol-ip]
    - Remote AS: [sec-dlr-local-as]
    - Weight: 60
    - Keep Alive Time: 1
    - Hold Down Time: 3
    - Password: [sec-dlr-bgp-password]
  - Publish
- Routing -> Route Redistribution -> Edit -> Select BGP -> OK
- Route Redistribution -> Add
  - Prefix: Any
  - Learner Protocol: BGP
  - OSPF: Deselected
  - Static Routes: Selected
  - Connected: Selected
  - Action: Permit
- Publish

For [sec-esg02-edgename]
- Networking & Security -> NSX Edges -> Double Click Edge -> Manage -> Routing -> Global Configuration -> Start ECMP
- Dynamic Routing Configuration -> Edit -> Router ID = Uplink01 -> OK – >Publish
- Routing -> Static Routes – Add
  - Network: [sec-udlr-compute-network]/[sec-udlr-compute-network-prefix-length]
  - Next Hop: [sec-udlr-utn-uplink-ip]
  - Interface: SEC-UDLR
  - Admin Distance 210
- Routing -> Static Routes – Add
  - Network: [sec-dlr-compute-network]/[sec-dlr-compute-network-prefix-length]
  - Next Hop: [sec-dlr-gtn-uplink-ip]
  - Interface: SEC-DLR
  - Admin Distance 210
- Publish
- Routing -> BGP -> Edit
  - Enable BGP: Selected
  - Enable Graceful Restart: Selected
  - Enable Default Originate: Deselected
  - Local AS: [sec-esg02-local-as]
- BGP -> Add
  - Neighbour 1 (TOR1)
    - IP Address: [tor1-bgp-ip]
    - Remote AS: [tor1-local-as]
    - Weight: 60
    - Keep Alive Time: 4
    - Hold Down Time: 12
    - Password: [tor1-bgp-password]
  - Neighbour 2 (TOR2)
    - IP Address: [tor2-bgp-ip]
    - Remote AS : [tor2-local-as]
    - Weight: 60
    - Keep Alive Time: 4
    - Hold Down Time: 12
    - Password: [tor2-bgp-password]
  - Neighbour 3 (UDLR)
    - IP Address: [sec-udlr-bgp-protocol-ip]
    - Remote AS: [sec-udlr-local-as]
    - Weight: 60
    - Keep Alive Time: 1
    - Hold Down Time: 3
    - Password: [sec-udlr-bgp-password]
  - Neighbour 3 (DLR)
    - IP Address: [sec-dlr-bgp-protocol-ip]
    - Remote AS: [sec-dlr-local-as]
    - Weight: 60
    - Keep Alive Time: 1
    - Hold Down Time: 3
    - Password: [sec-dlr-bgp-password]
  - Publish
- Routing -> Route Redistribution -> Edit -> Select BGP -> OK
- Route Redistribution -> Add
  - Prefix: Any
  - Learner Protocol: BGP
  - OSPF: Deselected
  - Static Routes: Selected
  - Connected: Selected
  - Action: Permit
- Publish

Deploy SEC UDLR

Networking & Security -> NSX Edges -> Add
- Name and Description
  - Universal Logical (Distributed) Router: Selected
  - Name: SEC-UDLR
  - Deploy Edge Appliance: Selected
  - Enable High Availability: Selected
  - Enable HA Logging: Selected
  - Log Level: INFO
- Settings
  - User Name: admin
  - Password: [sec-nsx-edge-complex-password]
  - Enable SSH access: Selected
  - Enable FIPS Mode: Deselected
  - Edge Control Level logging: INFO
- Configure Deployment
  - Cluster/Resource Pool: regiona-rp-sddc-edge
  - Datastore: [sec-vsan-datastore]
  - Add two appliances
- Configure Interfaces
  - HA Interface Configuration -> Select -> [mgmt-portgroup]
- Configure Interfaces
  - Name: Uplink
  - Type: Uplink
  - Connected to: Universal Transit Network
  - Connectivity Status: Connected
  - Primary IP Address: [sec-udlr-utn-uplink-ip]
  - Subnet Prefix Length: [sec-utn-prefix-length]
  - MTU: 9000
- Default Gateway
  - Deselect Configure Default Gateway
- Finish
Enable SSH
- Double click SEC-UDLR -> Manage -> Firewall
  - Add Rule
    - Name: enableSSH
    - Source: Any
    - Destination: Any
    - Service: SSH
    - Action: Accept

Configure SEC UDLR for Dynamic Routing

Networking & Security -> NSX Edges -> Double Click SEC-UDLR -> Manage -> Routing -> Global Configuration -> Routing Configuration -> Edit -> Enable ECMP -> OK
Dynamic Routing Configuration -> Edit -> RouterID -> Select [Uplink] -> OK -> Publish
Routing -> BGP -> Edit
- Enable BGP: Selected
- Enable Graceful Restart: Selected
- Local AS: [sec-udlr-local-as]
- Add Member for [sec-esg01-edgename]
  - IP Address: [sec-esg01-udlr-ip]
  - Forwarding Address: [sec-udlr-utn-uplink-ip]
  - Protocol Address: [sec-udlr-bgp-protocol-ip]
  - Remote AS: [sec-esg01-local-as]
  - Weight: 60
  - Keep Alive Time: 1
  - Hold Down Time: 3
  - Password: [sec-esg01-bgp-password]
- Add Member for [sec-esg02-edgename]
  - IP Address: [sec-esg02-udlr-ip]
  - Forwarding Address: [sec-udlr-utn-uplink-ip]
  - Protocol Address: [sec-udlr-bgp-protocol-ip]
  - Remote AS: [sec-esg02-local-as]
  - Weight: 60
    Keep Alive Time: 1
    Hold Down Time: 3
    Password: [sec-esg02-bgp-password]
Publish
Route Redistribution -> Edit
- OSPF: Deselected
- BGP: Selected
Route Redistribution -> Route Redistribution Table -> OSPF Entry -> Edit
- Learner Protocol: BGP
Publish

Deploy SEC DLR

Networking & Security -> NSX Edges -> Add
- Name and Description
  - Logical Router: Selected
  - Name: SEC-DLR
  - Deploy Edge Appliance: Selected
  - Enable High Availability: Selected
  - Enable HA Logging: Selected
  - Log Level: INFO
- Settings
  - User Name: admin
  - Password: [sec-nsx-edge-complex-password]
  - Enable SSH access: Selected
  - Enable FIPS Mode: Deselected
  - Edge Control Level logging: INFO
- Configure Deployment
  - Cluster/Resource Pool: regiona-rp-sddc-edge
  - Datastore: [sec-vsan-datastore]
  - Add two appliances
- Configure Interfaces
  - HA Interface Configuration -> Select -> [mgmt-portgroup]
- Configure Interfaces
  - Name: Uplink
  - Type: Uplink
  - Connected to: Global Transit Network
  - Connectivity Status: Connected
  - Primary IP Address: [sec-dlr-gtn-uplink-ip]
  - Subnet Prefix Length: [sec-gtn-prefix-length]
  - MTU: 9000
- Default Gateway
  - Deselect Configure Default Gateway
- Finish
Enable SSH
- Double click SEC-DLR -> Manage -> Firewall
  - Add Rule
    - Name: enableSSH
    - Source: Any
    - Destination: Any
    - Service: SSH
    - Action: Accept

Configure SEC DLR for Dynamic Routing

Networking & Security -> NSX Edges -> Double Click SEC-DLR -> Manage -> Routing -> Global Configuration -> Routing Configuration -> Edit -> Enable ECMP -> OK
Dynamic Routing Configuration -> Edit -> RouterID -> Select [Uplink] -> OK -> Publish
Routing -> BGP -> Edit
- Enable BGP: Selected
- Enable Graceful Restart: Selected
- Local AS: [sec-dlr-local-as]
- Add Member for [sec-esg01-edgename]
  - IP Address: [sec-esg01-dlr-ip]
  - Forwarding Address: [sec-dlr-gtn-uplink-ip]
  - Protocol Address: [sec-dlr-bgp-protocol-ip]
  - Remote AS: [sec-esg01-local-as]
  - Weight: 60
  - Keep Alive Time: 1
  - Hold Down Time: 3
  - Password: [sec-esg01-bgp-password]
- Add Member for [sec-esg02-edgename]
  - IP Address: [sec-esg02-dlr-ip]
  - Forwarding Address: [sec-dlr-gtn-uplink-ip]
  - Protocol Address: [sec-dlr-bgp-protocol-ip]
  - Remote AS: [sec-esg02-local-as]
  - Weight: 60
    Keep Alive Time: 1
    Hold Down Time: 3
    Password: [sec-esg02-bgp-password]
Publish
Route Redistribution -> Edit
- OSPF: Deselected
- BGP: Selected
Route Redistribution -> Route Redistribution Table -> OSPF Entry -> Edit
- Learner Protocol: BGP
Publish

You should now have a network topology that looks a bit like this

And thats it! You’ve now gotten the virtual infrastructure deployed for VVD on VxRail. All thats left to do is summarize…..