Breadcrumb Build – Region A VVD Virtual Infrastructure on VxRail Part 9: Logical Networking for the Shared Edge/Compute VxRail

Welcome to Part 9 of my VVD on VxRail Breadcrumb Build Series

Nearly there folks! Last post for the configuration of VVD on VxRail. While the topic is the same as Post 5 where we did logical networking for the management VxRail, theres’ not quite so much to do on the SEC VxRail, so this wont require quite as much coffee 🙂

Anything in the format [input_value] represents a value from your preparation that you should insert (without the square brackets of course)

NSX Dynamic Routing

  • Create a Universal Logical Switch for Use as the Transit Network
    • Networking & Security -> Logical Switches -> New Logical Switch
      • Name: Universal Transit Network
      • Transport Zone: SEC Universal Transport Zone
      • Replication Mode: Hybrid
  • Networking & Security -> Logical Switches -> New Logical Switch
    • Name: Global Transit Network
    • Transport Zone: SEC Global Transport Zone
    • Replication Mode: Hybrid

Deploy NSX Edge Devices for North-South Routing

  • Networking & Security -> NSX Edges -> Add
  • First ESG
    • Name and Desc
      • Install Type: Edge Services Gateway
      • Name: [sec-esg01-edgename]
      • Deploy NSX Edges: Selected
      • Enable High Availability: Deselected
    • Settings
      • User Name: admin
      • Password: [sec-nsx-edge-complex-password]
      • Enable SSH access: Selected
      • Enable FIPS Mode: Deselected
      • Enable auto rule generation: Selected
      • Edge Control Level logging: INFO
    • Configure Deployment
      • Appliance Size: Large
      • Cluster/Resource Pool: regiona-rp-sddc-edge
      • Datastore: [sec-vsan-datastore]
    • Configure Interfaces
      • Name: Uplink01
      • Type: Uplink
      • Connected to: Uplink01
      • Connectivity Status: Connected
      • Primary IP Address: [sec-esg01-uplink01-ip]
      • Subnet Prefix Length: [sec-uplink01-prefix-length]
      • MTU: 9000
      • Send ICMP Redirect: Selected
    • Configure Interfaces
      • Name: Uplink02
      • Type: Uplink
      • Connected to: Uplink02
      • Connectivity Status: Connected
      • Primary IP Address: [sec-esg01-uplink02-ip]
      • Subnet Prefix Length: [sec-uplink02-prefix-length]
      • MTU: 9000
      • Send ICMP Redirect: Selected
    • Configure Interfaces
      • Name: SEC-UDLR01
      • Type: Internal
      • Connected to: Universal Transit Network
      • Connectivity Status: Connected
      • Primary IP Address: [sec-esg01-utn-ip]
      • Subnet Prefix Length: [sec-utn-prefix-length]
      • MTU: 9000
      • Send ICMP Redirect: Selected
    • Configure Interfaces
      • Name: SEC-DLR01
      • Type: Internal
      • Connected to: Global Transit Network
      • Connectivity Status: Connected
      • Primary IP Address: [sec-esg01-gtn-ip]
      • Subnet Prefix Length: [sec-gtn-prefix-length]
      • MTU: 9000
      • Send ICMP Redirect: Selected
    • Default Gateway
      • Deselect Configure Default Gateway
    • Firewall and HA -> Next
  • Networking & Security -> NSX Edges -> Add
  • Second ESG
    • Name and Desc
      • Install Type: Edge Services Gateway
      • Name: [sec-esg02-edgename]
      • Deploy NSX Edges: Selected
      • Enable High Availability: Deselected
    • Settings
      • User Name: admin
      • Password: [sec-nsx-edge-complex-password]
      • Enable SSH access: Selected
      • Enable FIPS Mode: Deselected
      • Enable auto rule generation: Selected
      • Edge Control Level logging: INFO
    • Configure Deployment
      • Appliance Size: Large
      • Cluster/Resource Pool: regiona-rp-sddc-edge
      • Datastore: [sec-vsan-datastore]
    • Configure Interfaces
      • Name: Uplink01
      • Type: Uplink
      • Connected to: Uplink01
      • Connectivity Status: Connected
      • Primary IP Address: [sec-esg02-uplink01-ip]
      • Subnet Prefix Length: [sec-uplink01-prefix-length]
      • MTU: 9000
      • Send ICMP Redirect: Selected
    • Configure Interfaces
      • Name: Uplink02
      • Type: Uplink
      • Connected to: Uplink02
      • Connectivity Status: Connected
      • Primary IP Address: [sec-esg02-uplink02-ip]
      • Subnet Prefix Length: [sec-uplink02-prefix-length]
      • MTU: 9000
      • Send ICMP Redirect: Selected
    • Configure Interfaces
      • Name: SEC-UDLR01
      • Type: Internal
      • Connected to: Universal Transit Network
      • Connectivity Status: Connected
      • Primary IP Address: [sec-esg02-utn-ip]
      • Subnet Prefix Length: [sec-utn-prefix-length]
      • MTU: 9000
      • Send ICMP Redirect: Selected
    • Configure Interfaces
      • Name: SEC-DLR01
      • Type: Internal
      • Connected to: Global Transit Network
      • Connectivity Status: Connected
      • Primary IP Address: [sec-esg02-gtn-ip]
      • Subnet Prefix Length: [sec-gtn-prefix-length]
      • MTU: 9000
      • Send ICMP Redirect: Selected
    • Default Gateway
      • Deselect Configure Default Gateway
    • Firewall and HA -> Next

Anti-Affinity rules for N/S Edges

  • Select [sec-cluster] -> Configure -> VM/Host Rules -> Add
    • Name: anti-affinity-rule-ecmpedges
    • Enable rule: Selected
    • Type: Separate Virtual Machine
    • Add both [sec-esg01-edgename] and [sec-esg02-edgename]

Disable the Firewall Service

  • Networking & Security -> NSX Edges -> Double Click [sec-esg01-edgename]-> Manage -> Firewall -> Stop -> Publish
  • Repeat for [sec-esg02-edgename]

Enable and Configure Routing

  • For [sec-esg01-edgename]
    • Networking & Security -> NSX Edges -> Double Click Edge -> Manage -> Routing -> Global Configuration -> Start ECMP
    • Dynamic Routing Configuration -> Edit -> Router ID = Uplink01 -> OK – >Publish
    • Routing -> Static Routes – Add
      • Network: [sec-udlr-compute-network]/[sec-udlr-compute-network-prefix-length]
      • Next Hop: [sec-udlr-utn-uplink-ip]
      • Interface: SEC-UDLR
      • Admin Distance 210
    • Routing -> Static Routes – Add
      • Network: [sec-dlr-compute-network]/[sec-dlr-compute-network-prefix-length]
      • Next Hop: [sec-dlr-gtn-uplink-ip]
      • Interface: SEC-DLR
      • Admin Distance 210
    • Publish
    • Routing -> BGP -> Edit
      • Enable BGP: Selected
      • Enable Graceful Restart: Selected
      • Enable Default Originate: Deselected
      • Local AS: [sec-esg01-local-as]
    • BGP -> Add
      • Neighbour 1 (TOR1)
        • IP Address: [tor1-bgp-ip]
        • Remote AS: [tor1-local-as]
        • Weight: 60
        • Keep Alive Time: 4
        • Hold Down Time: 12
        • Password: [tor1-bgp-password]
      • Neighbour 2 (TOR2)
        • IP Address: [tor2-bgp-ip]
        • Remote AS : [tor2-local-as]
        • Weight: 60
        • Keep Alive Time: 4
        • Hold Down Time: 12
        • Password: [tor2-bgp-password]
      • Neighbour 3 (UDLR)
        • IP Address: [sec-udlr-bgp-protocol-ip]
        • Remote AS: [sec-udlr-local-as]
        • Weight: 60
        • Keep Alive Time: 1
        • Hold Down Time: 3
        • Password: [sec-udlr-bgp-password]
      • Neighbour 3 (DLR)
        • IP Address: [sec-dlr-bgp-protocol-ip]
        • Remote AS: [sec-dlr-local-as]
        • Weight: 60
        • Keep Alive Time: 1
        • Hold Down Time: 3
        • Password: [sec-dlr-bgp-password]
      • Publish
    • Routing -> Route Redistribution -> Edit -> Select BGP -> OK
    • Route Redistribution -> Add
      • Prefix: Any
      • Learner Protocol: BGP
      • OSPF: Deselected
      • Static Routes: Selected
      • Connected: Selected
      • Action: Permit
    • Publish
  • For [sec-esg02-edgename]
    • Networking & Security -> NSX Edges -> Double Click Edge -> Manage -> Routing -> Global Configuration -> Start ECMP
    • Dynamic Routing Configuration -> Edit -> Router ID = Uplink01 -> OK – >Publish
    • Routing -> Static Routes – Add
      • Network: [sec-udlr-compute-network]/[sec-udlr-compute-network-prefix-length]
      • Next Hop: [sec-udlr-utn-uplink-ip]
      • Interface: SEC-UDLR
      • Admin Distance 210
    • Routing -> Static Routes – Add
      • Network: [sec-dlr-compute-network]/[sec-dlr-compute-network-prefix-length]
      • Next Hop: [sec-dlr-gtn-uplink-ip]
      • Interface: SEC-DLR
      • Admin Distance 210
    • Publish
    • Routing -> BGP -> Edit
      • Enable BGP: Selected
      • Enable Graceful Restart: Selected
      • Enable Default Originate: Deselected
      • Local AS: [sec-esg02-local-as]
    • BGP -> Add
      • Neighbour 1 (TOR1)
        • IP Address: [tor1-bgp-ip]
        • Remote AS: [tor1-local-as]
        • Weight: 60
        • Keep Alive Time: 4
        • Hold Down Time: 12
        • Password: [tor1-bgp-password]
      • Neighbour 2 (TOR2)
        • IP Address: [tor2-bgp-ip]
        • Remote AS : [tor2-local-as]
        • Weight: 60
        • Keep Alive Time: 4
        • Hold Down Time: 12
        • Password: [tor2-bgp-password]
      • Neighbour 3 (UDLR)
        • IP Address: [sec-udlr-bgp-protocol-ip]
        • Remote AS: [sec-udlr-local-as]
        • Weight: 60
        • Keep Alive Time: 1
        • Hold Down Time: 3
        • Password: [sec-udlr-bgp-password]
      • Neighbour 3 (DLR)
        • IP Address: [sec-dlr-bgp-protocol-ip]
        • Remote AS: [sec-dlr-local-as]
        • Weight: 60
        • Keep Alive Time: 1
        • Hold Down Time: 3
        • Password: [sec-dlr-bgp-password]
      • Publish
    • Routing -> Route Redistribution -> Edit -> Select BGP -> OK
    • Route Redistribution -> Add
      • Prefix: Any
      • Learner Protocol: BGP
      • OSPF: Deselected
      • Static Routes: Selected
      • Connected: Selected
      • Action: Permit
    • Publish

Deploy SEC UDLR

  • Networking & Security -> NSX Edges -> Add
    • Name and Description
      • Universal Logical (Distributed) Router: Selected
      • Name: SEC-UDLR
      • Deploy Edge Appliance: Selected
      • Enable High Availability: Selected
      • Enable HA Logging: Selected
      • Log Level: INFO
    • Settings
      • User Name: admin
      • Password: [sec-nsx-edge-complex-password]
      • Enable SSH access: Selected
      • Enable FIPS Mode: Deselected
      • Edge Control Level logging: INFO
    • Configure Deployment
      • Cluster/Resource Pool: regiona-rp-sddc-edge
      • Datastore: [sec-vsan-datastore]
      • Add two appliances
    • Configure Interfaces
      • HA Interface Configuration -> Select -> [mgmt-portgroup]
    • Configure Interfaces
      • Name: Uplink
      • Type: Uplink
      • Connected to: Universal Transit Network
      • Connectivity Status: Connected
      • Primary IP Address: [sec-udlr-utn-uplink-ip]
      • Subnet Prefix Length: [sec-utn-prefix-length]
      • MTU: 9000
    • Default Gateway
      • Deselect Configure Default Gateway
    • Finish
  • Enable SSH
    • Double click SEC-UDLR -> Manage -> Firewall
      • Add Rule
        • Name: enableSSH
        • Source: Any
        • Destination: Any
        • Service: SSH
        • Action: Accept

Configure SEC UDLR for Dynamic Routing

  • Networking & Security -> NSX Edges -> Double Click SEC-UDLR -> Manage -> Routing -> Global Configuration -> Routing Configuration -> Edit -> Enable ECMP -> OK
  • Dynamic Routing Configuration -> Edit -> RouterID -> Select [Uplink] -> OK -> Publish
  • Routing -> BGP -> Edit
    • Enable BGP: Selected
    • Enable Graceful Restart: Selected
    • Local AS: [sec-udlr-local-as]
    • Add Member for [sec-esg01-edgename]
      • IP Address: [sec-esg01-udlr-ip]
      • Forwarding Address: [sec-udlr-utn-uplink-ip]
      • Protocol Address: [sec-udlr-bgp-protocol-ip]
      • Remote AS: [sec-esg01-local-as]
      • Weight: 60
      • Keep Alive Time: 1
      • Hold Down Time: 3
      • Password: [sec-esg01-bgp-password]
    • Add Member for [sec-esg02-edgename]
      • IP Address: [sec-esg02-udlr-ip]
      • Forwarding Address: [sec-udlr-utn-uplink-ip]
      • Protocol Address: [sec-udlr-bgp-protocol-ip]
      • Remote AS: [sec-esg02-local-as]
      • Weight: 60
        Keep Alive Time: 1
        Hold Down Time: 3
        Password: [sec-esg02-bgp-password]
  • Publish
  • Route Redistribution -> Edit
    • OSPF: Deselected
    • BGP: Selected
  • Route Redistribution -> Route Redistribution Table -> OSPF Entry -> Edit
    • Learner Protocol: BGP
  • Publish

Deploy SEC DLR

  • Networking & Security -> NSX Edges -> Add
    • Name and Description
      • Logical Router: Selected
      • Name: SEC-DLR
      • Deploy Edge Appliance: Selected
      • Enable High Availability: Selected
      • Enable HA Logging: Selected
      • Log Level: INFO
    • Settings
      • User Name: admin
      • Password: [sec-nsx-edge-complex-password]
      • Enable SSH access: Selected
      • Enable FIPS Mode: Deselected
      • Edge Control Level logging: INFO
    • Configure Deployment
      • Cluster/Resource Pool: regiona-rp-sddc-edge
      • Datastore: [sec-vsan-datastore]
      • Add two appliances
    • Configure Interfaces
      • HA Interface Configuration -> Select -> [mgmt-portgroup]
    • Configure Interfaces
      • Name: Uplink
      • Type: Uplink
      • Connected to: Global Transit Network
      • Connectivity Status: Connected
      • Primary IP Address: [sec-dlr-gtn-uplink-ip]
      • Subnet Prefix Length: [sec-gtn-prefix-length]
      • MTU: 9000
    • Default Gateway
      • Deselect Configure Default Gateway
    • Finish
  • Enable SSH
    • Double click SEC-DLR -> Manage -> Firewall
      • Add Rule
        • Name: enableSSH
        • Source: Any
        • Destination: Any
        • Service: SSH
        • Action: Accept

Configure SEC DLR for Dynamic Routing

  • Networking & Security -> NSX Edges -> Double Click SEC-DLR -> Manage -> Routing -> Global Configuration -> Routing Configuration -> Edit -> Enable ECMP -> OK
  • Dynamic Routing Configuration -> Edit -> RouterID -> Select [Uplink] -> OK -> Publish
  • Routing -> BGP -> Edit
    • Enable BGP: Selected
    • Enable Graceful Restart: Selected
    • Local AS: [sec-dlr-local-as]
    • Add Member for [sec-esg01-edgename]
      • IP Address: [sec-esg01-dlr-ip]
      • Forwarding Address: [sec-dlr-gtn-uplink-ip]
      • Protocol Address: [sec-dlr-bgp-protocol-ip]
      • Remote AS: [sec-esg01-local-as]
      • Weight: 60
      • Keep Alive Time: 1
      • Hold Down Time: 3
      • Password: [sec-esg01-bgp-password]
    • Add Member for [sec-esg02-edgename]
      • IP Address: [sec-esg02-dlr-ip]
      • Forwarding Address: [sec-dlr-gtn-uplink-ip]
      • Protocol Address: [sec-dlr-bgp-protocol-ip]
      • Remote AS: [sec-esg02-local-as]
      • Weight: 60
        Keep Alive Time: 1
        Hold Down Time: 3
        Password: [sec-esg02-bgp-password]
  • Publish
  • Route Redistribution -> Edit
    • OSPF: Deselected
    • BGP: Selected
  • Route Redistribution -> Route Redistribution Table -> OSPF Entry -> Edit
    • Learner Protocol: BGP
  • Publish

You should now have a network topology that looks a bit like this

And thats it! You’ve now gotten the virtual infrastructure deployed for VVD on VxRail. All thats left to do is summarize…..

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s