Breadcrumb Build – Region A VVD Virtual Infrastructure on VxRail Part 3: Adjusting the Management VxRail

Welcome to Part 3 of my VVD on VxRail Breadcrumb Build Series

With the default VxRail deployment out of the way, we now need to tweak the setup of the Management VxRail to align it with VVD expectations. From here on in, the VVD build is pretty much completely manual (other than when we use the VxRail Deployment wizard in a later step to deploy the Shared Edge/compute VxRail). This is where Cloud Builder will really come into its own. I still think its a valuable exercise to do a manual build though….you learn a lot and it ultimately helps when things dont go to plan first time with automation.

Anything in the format [input_value] represents a value from your preparation that you should insert (without the square brackets of course)

Enable SSH on VxRail Manager

Log onto console of [mgmt-vxrm-vmname]
Open xterm
- vi /etc/ssh/sshd_config
Enable PermitRootLogin
service sshd restart

Convert Internal VC to External

Check VSAN Health (Ensure no Resync)
- VC -> Home -> Hosts and Clusters -> Choose VxRail Cluster
- Monitor -> vSAN -> Health
Clone/Snap VC/PSC/VxRail Mgr
WinSCP to VXRM
Upload vvd_vc_conversion.pyc [obtain from DellEMC]
python ./vvd_vc_conversion.pyc
- FQDN: [mgmt-vcenter-fqdn]
- Administrator@vsphere.local / [sso-admin-password]
- FQDN [mgmt-psc-fqdn]
Confirm Datacenter
Choose VxRail Manager VM
Reboot

Deploy External PSC for Shared Edge/Compute

Deploy OVA directly or use vSphere deployment tool
- Appliance Target
  - Target: [mgmt-vcenter-fqdn]
  - Port: 443
  - Username: administrator@vsphere.local
  - Password: [sso-admin-password]
- Appliance VM
  - VM Name: [sec-psc-vmname]
  - Password: [sec-psc-root-password]
- Datastore: [mgmt-vsan-datastore]
- Network
  - Network: [mgmt-portgroup]
  - IP version: IPV4
  - IP Assignment: Static
  - System Name: [sec-psc-fqdn]
  - IP Address: [sec-psc-ip]
  - Subnet Mask: [mgmt-mask]
  - Gateway: [mgmt-gateway]
  - DNS: [dns-server]
- Appliance Configuration
  - NTP: [ntp-ip]
- SSO Configuration
  - Join an Existing Domain
  - PSC: [mgmt-psc-fqdn]
  - Port: 443
  - SSO Domain: vsphere.local
  - SSO Password: [sso-admin-password]

Join both PSCs to Domain

For both PSCs
- Browse to /PSC
Appliance Settings -> Manage -> Join
- Join Domain -> [ad-domain] ->Reboot
For one PSC only
- Configuration -> Identity Sources -> Add Domain

Replace the Platform Services Controller Certificates

Replace [psc-hostname] with [mgmt-psc-hostname] or [sec-psc-hostname] as you perform for each PSC

For each PSC
- Ensure bash is set a default shell
  - SSH to Box
  - shell
  - chsh -s “/bin/bash” root
- mkdir -p /root/certs
- WinSCP files to /root/certs
  - [psc-hostname].1.cer
  - [psc-hostname].key
  - Root64.cer
- /usr/lib/vmware-vmca/bin/certificate-manager
  - Option 1
  - Option 2
- Enter paths to files
- service vami-lighttp restart
- On associated VC
  - service-control –stop –all
  - service-control –start –all

Update the Platform Services Controller SSO Configuration and Endpoints in Region A

DNS Load Balancer Entry
- Create DNS entry for [psc-lb-virtual-fqdn]
- Set to IP to [mgmt-psc-ip]
For both PSCs , update the Platform Services Controller SSO configuration
- SSH into PSC
- cd /usr/lib/vmware-sso/bin/
- python updateSSOConfig.py –lb-fqdn=[psc-lb-virtual-fqdn]

Update the Platform Services Controller endpoints

On one PSC only
- SSH into PSC
- cd /usr/lib/vmware-sso/bin/
- python UpdateLsEndpoint.py –lb-fqdn=[psc-lb-virtual-fqdn] –user=Administrator@vsphere.local

Add Management vSphere Licenses

VC -> Home -> Administration -> Licenses
On Licenses tab -> Create New Licenses
Add and Assign Licenses

Assign vCenter Permissions

Administration -> Global Permissions -> Manage -> Add permission -> Add
- Select [ad-domain] from the Domain drop-down
- Enter [vcenter-admins-group] in the Search field -> Enter.
- Select the [vcenter-admins-group] -> Add -> OK.
Select Administrator & Propagate to children -> OK

Repoint the Management vCenter to the Load Balanced SSO FQDN

SSH to [mgmt-vcenter-ip]
- cmsso-util repoint –repoint-psc [psc-lb-virtual-fqdn]
- /usr/lib/vmware-vmafd/bin/vmafd-cli get-dc-name –server-name localhost

Replace the Certificate of the Management vCenter Server

SSH to [mgmt-vcenter-ip]
mkdir -p /root/certs
WinSCP files to /root/certs
- [mgmt-vcenter-hostname].1.cer
- [mgmt-vcenter-hostname].key
- Root64.cer
/usr/lib/vmware-vmca/bin/certificate-manager
- Option 1
- Enter IP of [mgmt-psc-ip]
- Option 2
- Enter path to filenames
service vami-lighttp restart
cd /root/certs/
rm *

Replace vCenter SSL certificate on VxRail Manager

Browse to https://[mgmt-vcenter-fqdn]/
- Click ‘Download trusted root CA certificates’
WinSCP download.zip to /tmp on [mgmt-vxrm-ip]
- unzip download.zip
- cd /certs/lin
  - File(s) with a digit as the file name extension are the certificate files.
  - Files with “.r” as the file name extension is the corresponding CRL
Convert certificate file(s) to DER format via
- openssl x509 -outform der -in <INPUT_CA> -out <OUTPUT_FILE>
- If there are more than one certificate file with distinct file name (ignore the extension different), you should convert each of them, and choose a different output file name for each (e.g. newcertfile2).
- If there are multiple files with the same name, choose the one with the largest number in the extension
  - openssl x509 -outform der -in /tmp/certs/lin/[470497c9.0] -out newcertfile1
  - openssl x509 -outform der -in /tmp/certs/lin/[8d3caf6d.0] -out newcertfile2
  - openssl x509 -outform der -in /tmp/certs/lin/[a0696df2.0] -out newcertfile3
cp newcertfile* /var/lib/vmware-marvin/trust/
Convert CRL files, use largest digit extension if applicable
- openssl crl -outform der -in /tmp/certs/lin/[8d3caf6d.r0] -out newcrltfile1
- openssl crl -outform der -in /tmp/certs/lin/[a0696df2.r1] -out newcrltfile2
cp newcrltfile* /var/lib/vmware-marvin/trust/crl
Change Permissions on Files
- cd /var/lib/vmware-marvin/trust/
- chown tcserver:pivotal newcertfile*
- chmod 777 newcertfile*
- cd crl
- chown tcserver:pivotal newcrltfile*
- chmod 777 newcrltfile*
systemctl restart vmware-marvin
- Change the permission of the new cert file(s) back to just ‘-rw-r–r–‘ (chmod 644)
  - chmod 644 /var/lib/vmware-marvin/trust/newcertfile*
  - chmod 644 /var/lib/vmware-marvin/trust/crl/newcrltfile*

Rename Components (optional but handy)

Rename Distributed Switch to [mgmt-vds]
Rename vSAN data store to [mgmt-vsan-datastore]

Set SDDC Deployment Details on the Management vCenter Server

Log into [mgmt-vcenter-fqdn]
Home -> Global Inventory Lists -> vCenter Servers -> Resources
Select [mgmt-vcenter-fqdn]
Configure -> Settings -> Advanced Settings -> Edit
Add
- config.SDDC.Deployed.Type: VVD
- config.SDDC.Deployed.Flavor: Standard
- config.SDDC.Deployed.vvd_vc_conversion: 4.3.0
- config.SDDC.Deployed.WorkloadDomain: Management
- config.SDDC.Deployed.Method: DIY
- config.SDDC.Deployed.InstanceId: [generated-random-uid]

Configure the Management Cluster

Add all ESXi hosts to the [ad-domain]
- Log into [mgmt-vcenter-fqdn]
- Navigator -> Hosts and Clusters -> Select Cluster
- For Each Host
  - Configure -> System -> Authentication Services -> Join Domain
    - [ad-domain]
    - [ad-psc-bind-username] / [ad-psc-bind-password]
System -> Security Profile -> Edit (next to Services)
- Active Directory Service -> Startup Policy -> Start and stop with host
- SSH -> Startup Policy -> Start and stop with host
Configure the VDS
- Log into [mgmt-vcenter-fqdn]
- Select [mgmt-vds] -> Configure -> Advanced
- MTU 9000
- NSX Portgroups
- Distributed Port Group -> New Distributed Port Group
  - Uplink01 Static Binding VLAN [mgmt-uplink01-vlan]
  - Uplink02 Static Binding VLAN [mgmt-uplink02-vlan]
Change Default for Main Portgroups
- [mgmt-vds] -> Distributed Port Group -> Manage Distributed Port Groups -> Teaming and failover -> Next
- All DPGs except Uplink01 and Uplink02 -> Next
- Route based on physical NIC load -> Next -> Finish
Uplink01 -> Edit Settings -> Teaming and Failover
- dvUplink2 -> Unused uplinks -> OK.
Uplink02 -> Edit Settings -> Teaming and Failover
- dvUplink1 -> Unused uplinks -> OK.
Hosts & Clusters (each host)
- Configure -> VMKernel Adapters -> vMotion Adapter -> Edit -> NIC Settings -> MTU -> 9000
Distributed Switch -> Configure -> Resource Allocation -> System Traffic
- Virtual SAN Traffic: High
- vMotion Traffic: Low
- vSphere Replication (VR) Traffic: Low
- Management Traffic: Normal
- vSphere Data Protection Backup Traffic: Low
- Virtual Machine Traffic: High
- Fault Tolerance Traffic: Low
- iSCSI Traffic: Low
- NFS Traffic: Low
[mgmt-vds] -> Configure -> Health Check -> Edit
- Enabled for VLAN and MTU and Teaming and failover
Modify vSphere HA
- Hosts & Clusters -> Cluster -> Configure -> vSphere Availability -> Edit
- Failures and Responses -> VM Monitoring -> VM Monitoring Only
Advanced Options on the ESXi Hosts (All Hosts)
- Configure -> System -> Advanced System Settings -> Edit
- Filter – esxAdmins
- Change Config.HostAgent.plugins.hostsvc.esxAdminsGroup to [sddc-admins]
- Filter – vsan.swap
- VSAN.SwapThickProvisionDisabled to 1
- Filter – ssh
- UserVars.SuppressShellWarning to 1
- OK
Mount NFS Storage
Create any folders you want for VMs and Templates

Anti-Affinity Rules for the PSCs

Log into [mgmt-vcenter-fqdn]
Select Cluster -> Configure -> VM/Host Rules -> Add
- Rule name: anti-affinity-rule-psc
- Separate Virtual Machines
- Select both PSCs

VM Groups to Define Startup Order

VC -> Select Cluster -> Configure -> VM/Host Group -> Add
- Group Name: Platform Services Controllers
- Select both PSCs
Select Cluster -> Configure -> VM/Host Group -> Add
- Group Name: vCenter Servers
- Add [mgmt-vcenter-vmname]
Select Cluster -> Configure -> VM/Host Rules -> Add
- Rule name: SDDC Management Virtual Machines
- First restart VMs in VM group: Platform Services Controllers
- Then restart VMs in VM group: vCenter Servers

Create localos admin for Second PSC

SSH to [mgmt-psc-ip]
- grep vxadmin /etc/passwd
- Get result similar to
  - admin:x:1003:59003::/home/admin:/bin/bash
SSH to [sec-psc-ip]
- groupadd -g 59003 admin
- useradd admin -u 1003 -g 59003 -d /home/admin -s /bin/bash
- passwd admin
- Supply same password as admin on [mgmt-psc-ip]
Log into [mgmt-vcenter-fqdn]
Administration -> Global Permissions ->Add -> Select localos
- Add admin
- Assign Role – VMware HCIA Management

Ok. Now your management VxRail is aligned to VVD. It doesnt look a whole lot different when you login to vCenter (other than the presence of the new PSC) but it’s now ready to accept the NSX deployment. Thats a lengthy enough set of instructions, so I’m covering that in the next part of the series.