Breadcrumb Build – Region A VVD Virtual Infrastructure on VxRail Part 3: Adjusting the Management VxRail

Welcome to Part 3 of my VVD on VxRail Breadcrumb Build Series

With the default VxRail deployment out of the way, we now need to tweak the setup of the Management VxRail to align it with VVD expectations. From here on in, the VVD build is pretty much completely manual (other than when we use the VxRail Deployment wizard in a later step to deploy the Shared Edge/compute VxRail). This is where Cloud Builder will really come into its own. I still think its a valuable exercise to do a manual build though….you learn a lot and it ultimately helps when things dont go to plan first time with automation.

Anything in the format [input_value] represents a value from your preparation that you should insert (without the square brackets of course)

Enable SSH on VxRail Manager

  • Log onto console of [mgmt-vxrm-vmname]
  • Open xterm
    • vi /etc/ssh/sshd_config
  • Enable PermitRootLogin
  • service sshd restart

Convert Internal VC to External

  • Check VSAN Health (Ensure no Resync)
    • VC -> Home -> Hosts and Clusters -> Choose VxRail Cluster
    • Monitor -> vSAN -> Health
  • Clone/Snap VC/PSC/VxRail Mgr
  • WinSCP to VXRM
  • Upload vvd_vc_conversion.pyc [obtain from DellEMC]
  • python ./vvd_vc_conversion.pyc
    • FQDN: [mgmt-vcenter-fqdn]
    • Administrator@vsphere.local / [sso-admin-password]
    • FQDN [mgmt-psc-fqdn]
  • Confirm Datacenter
  • Choose VxRail Manager VM
  • Reboot

Deploy External PSC for Shared Edge/Compute

  • Deploy OVA directly or use vSphere deployment tool
    • Appliance Target
      • Target: [mgmt-vcenter-fqdn]
      • Port: 443
      • Username: administrator@vsphere.local
      • Password: [sso-admin-password]
    • Appliance VM
      • VM Name: [sec-psc-vmname]
      • Password: [sec-psc-root-password]
    • Datastore: [mgmt-vsan-datastore]
    • Network
      • Network: [mgmt-portgroup]
      • IP version: IPV4
      • IP Assignment: Static
      • System Name: [sec-psc-fqdn]
      • IP Address: [sec-psc-ip]
      • Subnet Mask: [mgmt-mask]
      • Gateway: [mgmt-gateway]
      • DNS: [dns-server]
    • Appliance Configuration
      • NTP: [ntp-ip]
    • SSO Configuration
      • Join an Existing Domain
      • PSC: [mgmt-psc-fqdn]
      • Port: 443
      • SSO Domain: vsphere.local
      • SSO Password: [sso-admin-password]

Join both PSCs to Domain

  • For both PSCs
    • Browse to /PSC
  • Appliance Settings -> Manage -> Join
    • Join Domain -> [ad-domain] ->Reboot
  • For one PSC only
    • Configuration -> Identity Sources -> Add Domain

Replace the Platform Services Controller Certificates

Replace [psc-hostname] with [mgmt-psc-hostname] or [sec-psc-hostname] as you perform for each PSC

  • For each PSC
    • Ensure bash is set a default shell
      • SSH to Box
      • shell
      • chsh -s “/bin/bash” root
    • mkdir -p /root/certs
    • WinSCP files to /root/certs
      • [psc-hostname].1.cer
      • [psc-hostname].key
      • Root64.cer
    • /usr/lib/vmware-vmca/bin/certificate-manager
      • Option 1
      • Option 2
    • Enter paths to files
    • service vami-lighttp restart
    • On associated VC
      • service-control –stop –all
      • service-control –start –all

Update the Platform Services Controller SSO Configuration and Endpoints in Region A

  • DNS Load Balancer Entry
    • Create DNS entry for [psc-lb-virtual-fqdn]
    • Set to IP to [mgmt-psc-ip]
  • For both PSCs , update the Platform Services Controller SSO configuration
    • SSH into PSC
    • cd /usr/lib/vmware-sso/bin/
    • python updateSSOConfig.py –lb-fqdn=[psc-lb-virtual-fqdn]

Update the Platform Services Controller endpoints

  • On one PSC only
    • SSH into PSC
    • cd /usr/lib/vmware-sso/bin/
    • python UpdateLsEndpoint.py –lb-fqdn=[psc-lb-virtual-fqdn] –user=Administrator@vsphere.local

Add Management vSphere Licenses

  • VC -> Home -> Administration -> Licenses
  • On Licenses tab -> Create New Licenses
  • Add and Assign Licenses

Assign vCenter Permissions

  • Administration -> Global Permissions -> Manage -> Add permission -> Add
    • Select [ad-domain] from the Domain drop-down
    • Enter [vcenter-admins-group] in the Search field -> Enter.
    • Select the [vcenter-admins-group] -> Add -> OK.
  • Select Administrator & Propagate to children -> OK

Repoint the Management vCenter to the Load Balanced SSO FQDN

  • SSH to [mgmt-vcenter-ip]
    • cmsso-util repoint –repoint-psc [psc-lb-virtual-fqdn]
    • /usr/lib/vmware-vmafd/bin/vmafd-cli get-dc-name –server-name localhost

Replace the Certificate of the Management vCenter Server

  • SSH to [mgmt-vcenter-ip]
  • mkdir -p /root/certs
  • WinSCP files to /root/certs
    • [mgmt-vcenter-hostname].1.cer
    • [mgmt-vcenter-hostname].key
    • Root64.cer
  • /usr/lib/vmware-vmca/bin/certificate-manager
    • Option 1
    • Enter IP of [mgmt-psc-ip]
    • Option 2
    • Enter path to filenames
  • service vami-lighttp restart
  • cd /root/certs/
  • rm *

Replace vCenter SSL certificate on VxRail Manager

  • Browse to https://[mgmt-vcenter-fqdn]/
    • Click ‘Download trusted root CA certificates’
  • WinSCP download.zip to /tmp on [mgmt-vxrm-ip]
    • unzip download.zip
    • cd /certs/lin
      • File(s) with a digit as the file name extension are the certificate files.
      • Files with “.r” as the file name extension is the corresponding CRL
  • Convert certificate file(s) to DER format via
    • openssl x509 -outform der -in <INPUT_CA> -out <OUTPUT_FILE>
    • If there are more than one certificate file with distinct file name (ignore the extension different), you should convert each of them, and choose a different output file name for each (e.g. newcertfile2).
    • If there are multiple files with the same name, choose the one with the largest number in the extension
      • openssl x509 -outform der -in /tmp/certs/lin/[470497c9.0] -out newcertfile1
      • openssl x509 -outform der -in /tmp/certs/lin/[8d3caf6d.0] -out newcertfile2
      • openssl x509 -outform der -in /tmp/certs/lin/[a0696df2.0] -out newcertfile3
  • cp newcertfile* /var/lib/vmware-marvin/trust/
  • Convert CRL files, use largest digit extension if applicable
    • openssl crl -outform der -in /tmp/certs/lin/[8d3caf6d.r0] -out newcrltfile1
    • openssl crl -outform der -in /tmp/certs/lin/[a0696df2.r1] -out newcrltfile2
  • cp newcrltfile* /var/lib/vmware-marvin/trust/crl
  • Change Permissions on Files
    • cd /var/lib/vmware-marvin/trust/
    • chown tcserver:pivotal newcertfile*
    • chmod 777 newcertfile*
    • cd crl
    • chown tcserver:pivotal newcrltfile*
    • chmod 777 newcrltfile*
  • systemctl restart vmware-marvin
    • Change the permission of the new cert file(s) back to just ‘-rw-r–r–‘ (chmod 644)
      • chmod 644 /var/lib/vmware-marvin/trust/newcertfile*
      • chmod 644 /var/lib/vmware-marvin/trust/crl/newcrltfile*

Rename Components (optional but handy)

  • Rename Distributed Switch to [mgmt-vds]
  • Rename vSAN data store to [mgmt-vsan-datastore]

Set SDDC Deployment Details on the Management vCenter Server

  • Log into [mgmt-vcenter-fqdn]
  • Home -> Global Inventory Lists -> vCenter Servers -> Resources
  • Select [mgmt-vcenter-fqdn]
  • Configure -> Settings -> Advanced Settings -> Edit
  • Add
    • config.SDDC.Deployed.Type: VVD
    • config.SDDC.Deployed.Flavor: Standard
    • config.SDDC.Deployed.vvd_vc_conversion: 4.3.0
    • config.SDDC.Deployed.WorkloadDomain: Management
    • config.SDDC.Deployed.Method: DIY
    • config.SDDC.Deployed.InstanceId: [generated-random-uid]

Configure the Management Cluster

  • Add all ESXi hosts to the [ad-domain]
    • Log into [mgmt-vcenter-fqdn]
    • Navigator -> Hosts and Clusters -> Select Cluster
    • For Each Host
      • Configure -> System -> Authentication Services -> Join Domain
        • [ad-domain]
        • [ad-psc-bind-username] / [ad-psc-bind-password]
  • System -> Security Profile -> Edit (next to Services)
    • Active Directory Service -> Startup Policy -> Start and stop with host
    • SSH -> Startup Policy -> Start and stop with host
  • Configure the VDS
    • Log into [mgmt-vcenter-fqdn]
    • Select [mgmt-vds] -> Configure -> Advanced
    • MTU 9000
    • NSX Portgroups
    • Distributed Port Group -> New Distributed Port Group
      • Uplink01 Static Binding VLAN [mgmt-uplink01-vlan]
      • Uplink02 Static Binding VLAN [mgmt-uplink02-vlan]
  • Change Default for Main Portgroups
    • [mgmt-vds] -> Distributed Port Group -> Manage Distributed Port Groups -> Teaming and failover -> Next
    • All DPGs except Uplink01 and Uplink02 -> Next
    • Route based on physical NIC load -> Next -> Finish
  • Uplink01 -> Edit Settings -> Teaming and Failover
    • dvUplink2 -> Unused uplinks -> OK.
  • Uplink02 -> Edit Settings -> Teaming and Failover
    • dvUplink1 -> Unused uplinks -> OK.
  • Hosts & Clusters (each host)
    • Configure -> VMKernel Adapters -> vMotion Adapter -> Edit -> NIC Settings -> MTU -> 9000
  • Distributed Switch -> Configure -> Resource Allocation -> System Traffic
    • Virtual SAN Traffic: High
    • vMotion Traffic: Low
    • vSphere Replication (VR) Traffic: Low
    • Management Traffic: Normal
    • vSphere Data Protection Backup Traffic: Low
    • Virtual Machine Traffic: High
    • Fault Tolerance Traffic: Low
    • iSCSI Traffic: Low
    • NFS Traffic: Low
  • [mgmt-vds] -> Configure -> Health Check -> Edit
    • Enabled for VLAN and MTU and Teaming and failover
  • Modify vSphere HA
    • Hosts & Clusters -> Cluster -> Configure -> vSphere Availability -> Edit
    • Failures and Responses -> VM Monitoring -> VM Monitoring Only
  • Advanced Options on the ESXi Hosts (All Hosts)
    • Configure -> System -> Advanced System Settings -> Edit
    • Filter – esxAdmins
    • Change Config.HostAgent.plugins.hostsvc.esxAdminsGroup to [sddc-admins]
    • Filter – vsan.swap
    • VSAN.SwapThickProvisionDisabled to 1
    • Filter – ssh
    • UserVars.SuppressShellWarning to 1
    • OK
  • Mount NFS Storage
  • Create any folders you want for VMs and Templates

Anti-Affinity Rules for the PSCs

  • Log into [mgmt-vcenter-fqdn]
  • Select Cluster -> Configure -> VM/Host Rules -> Add
    • Rule name: anti-affinity-rule-psc
    • Separate Virtual Machines
    • Select both PSCs

VM Groups to Define Startup Order

  • VC -> Select Cluster -> Configure -> VM/Host Group -> Add
    • Group Name: Platform Services Controllers
    • Select both PSCs
  • Select Cluster -> Configure -> VM/Host Group -> Add
    • Group Name: vCenter Servers
    • Add [mgmt-vcenter-vmname]
  • Select Cluster -> Configure -> VM/Host Rules -> Add
    • Rule name: SDDC Management Virtual Machines
    • First restart VMs in VM group: Platform Services Controllers
    • Then restart VMs in VM group: vCenter Servers

Create localos admin for Second PSC

  • SSH to [mgmt-psc-ip]
    • grep vxadmin /etc/passwd
    • Get result similar to
      • admin:x:1003:59003::/home/admin:/bin/bash
  • SSH to [sec-psc-ip]
    • groupadd -g 59003 admin
    • useradd admin -u 1003 -g 59003 -d /home/admin -s /bin/bash
    • passwd admin
    • Supply same password as admin on [mgmt-psc-ip]
  • Log into [mgmt-vcenter-fqdn]
  • Administration -> Global Permissions ->Add -> Select localos
    • Add admin
    • Assign Role – VMware HCIA Management

Ok. Now your management VxRail is aligned to VVD. It doesnt look a whole lot different when you login to vCenter (other than the presence of the new PSC) but it’s now ready to accept the NSX deployment. Thats a lengthy enough set of instructions, so I’m covering that in the next part of the series.

r

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s