Pro Tip: NSX Distributed Firewall Rule locked you out of vCenter?

Just recently, while manually installing VVD 4.3 on VxRail I ran into an issue whereby I managed to lock myself out of vCenter due to an errant firewall rule.

It’s easy to do. When setting up a Security Group and adding an IP set, you need first select the IP set to add, then click to add it, then click OK. Sometimes 1 and 3 seem like enough to the intuitive part of us (trust me it isn’t!). If you vCenter isn’t in the DFW Exclusion list yet then you have a recipe for disaster!

Security-Groups

In this case, a single rule which was to allow all the management infrastructure to communicate seamlessly was effectively missing the subnet to apply the rule to. As soon as I changed the default rule in the NSX firewall to Block instead of Allow, my entire rig crashed and burned.

Thankfully NSX Manager is by default excluded from a DFW rule. So you can still get to it to repair the damage via API.

The objective of this is reset the firewall to its default state. Don’t worry though, you’re not going to lose the all the good rules as NSX stores the config every time you change it and you can reload from those backups, fix the errant rule and republish. That said, in a production environment you might want to schedule this and take some extra precautions as there will be a small window where the firewall is in default state.

I like to use Postman to do this and specifically an older than latest version and the newer versions all seem to suffer the ‘Black Screen’ issue. In this case I installed 4.10.7

When you have Postman spun up, click the gear icon in the top right corner, then Settings. On the General tab turn off SSL certificate verification and close the window

Postman-Settings

On the main page, set the Type to Basic Auth and enter the admin username and password for your NSX manager

Select DELETE from the drop down and paste the below into the API call cell (replace the IP below as appropriate) and click Send

https://172.19.5.35/api/4.0/firewall/globalroot-0/config

NSX-Call

The request should return a status of 204. If so you are good. Now log back into VC and go to your NSX Firewall configuration. Click More in the top right corner and Load Saved Configuration

Load-Config

Now Choose the restore point and click Load. In this case the ones Saved By admin are the operations done via API, and the others are the ones done through the GUI.

Choose-Config

The config will now be loaded back into the firewall, but are unpublished. All thats left to do is fix the errant rule and publish. Hey presto, you’re firewall is back in the game. You might need to restart some of the key services / appliances that were effected

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s