Using NSX-T to Test NSX-T and Virtual Machine Recovery with Automation – Practical

Part 1 – Windows SFTP Backup Targets

Part 2a – Using NSX-T to Test NSX-T and Virtual Machine Recovery with Automation – Concept

Part 2b –Using NSX-T to Test NSX-T and Virtual Machine Recovery with Automation – Practical

In Part 2a, the Healthcare organization admins had created several scripts using VMware PowerCLI, PowerShell Core 6, OVF Tool, and NSX-T Policy REST APIs.  Those scripts are located at the following GitHub link for other community admins to consume as well.

The original requirements that were put forth for the admins to provide a design for were:

Requirements:

  1. Use NSX-T to build a production replica network to test restores of the NSX-T Manager and show virtual machines can also be restored and tested on the same network
  2. Use Veeam to restore the following virtual machines:
    1. Backup Server – Will be used to run automation scripts from
    2. Active Directory – Will be needed for DNS purposes
    3. SFTP Server – Hosts the NSX-T backups that restores will be tested from
  3. Deploy a new NSX-T Manager to test the restore process to it
  4. Use automation wherever possible to continue expanding automated techniques

To meet these requirements the admin had designed the following topology to meet these requirements:

finish_topology

  • Standalone Tier-1 Gateway – not connected to any Tier-0 Gateway, preventing northbound communications that would conflict with the production networking
  • Restore Network Segment – Provides a logical network for the restored VMs to attach to
  • Restored Domain Controller – One of the organizations domain controllers that will provide DNS for the replica network and the VMs attached
  • Restored Backup Server – Hosts the PowerShell scripts that are necessary for scripting part of the deployment on the restored NSX-T Manager. Some of the scripts will need to be run from the Production Backup Server and some of them from the Restored Backup Server since there will be no outside communications to the Restore environment other than vCenter Server direct console access
  • Restored SFTP Server – Hosts the backups of the NSX-T Manager
  • Restored NSX-T Manager – Will be used to test its own restores. NSX-T Manager restores requires that the new NSX-T Manager have the same IP address as the production copy.  To test this appropriately, we have to create a copy of the production network and IP addressing
  • vCenter Server B – Manages the Compute Cluster B
  • Compute Cluster B – Provides a non-production host for the restored systems to be placed on that’s not managed by the production vCenter Server A.

For further details on reasonings for this topology, you can take a look at Part 2a referenced at the top of this thread.

With the scripts created, it’s now time for the admin to work through the workflow processes and test that this strategy will meet the requirements in practice.  This is a review of the workflow process:

restores_pic10

Step 1 – Copy scripts to BACKUP-01a – GitHub download and copy

The scripts just need to be pulled down from GitHub and copied to a location on the BACKUP-01a server

Step 2 – Copy NSX-T OVA to BACKUP-01a – Download and copy

Another straightforward step with downloading the NSX-T OVA that’s the exact version of the current NSX-T Manager and copying it to a location on BACKUP-01a

Step 3 – Install PowerShell Core 6, PowerCLI, and OVFTool – Download installs and install

restore_pract_pic2

Step 4 – Perform a Backup of the NSX-T Manager – Native Backup Tool

A pretty simple step by just going into the NSX-T Manager and the Backup & Restore tab and pressing the ‘BACKUP NOW’ button and verifying its completion.

restore_pract_pic1

Step 5 – Backup SFTP-01a, AD-01a, BACKUP-01a – Single Veeam Backup Job

Once all of the components to perform the remaining workflows are done and installed and configured, the backups of the necessary virtual machines, especially the BACKUP-01a machine, can occur.

restore_pract_pic3

Step 6 and 7 – Deploy Testing Tier-1 Gateway and Segment – NSX-T Policy API via PowerCLI

From the BACKUP-01a production server, the admin runs the 01_NSXT_DEPLOY.ps1 to build the Tier-1 Gateway and Segment and then it will start the OVF Tool to deploy the NSX-T Manager OVA file to the Compute Cluster B.

restore_pract_pic4

Tier-1 Gateway has been created, not linked to a Tier-0 Gateway to prevent Northbound connectivity with the overlapping production network and ‘nsxt-restore-segment’ created for the virtual machines and new NSX-T Manager to attach to.

restore_pract_pic5

restore_pract_pic6

The admin can also see that the new NSX-T Manager, connected to the ‘nsxt-restore-segment’ is being deployed.

restore_pract_pic7

Step 8 – Adjust NSX-T CPU/Mem Resources and Power-On – PowerCLI

Once the new NSX-T Manager is deployed, the admin wants to adjust the memory reservation so that they can start the NSX-T Manager without running into memory constraints since the test environment is rather limited.  The deployed NSX-T Manager is in ‘small’ form factor, but still has a 16GB Memory reservation on it.  From the BACKUP-01a production server, the admin runs the 02_NSXT_RESERVATION_ADJUST.ps1 to adjust the memory reservation down to 8GB and then power on the appliance.

restore_pract_pic8

Step 9 – Restore VMs to NSX-T Testing Segment – Veeam Restore Job

To get the virtual machines necessary to help in the NSX-T restore process and to prove that the admins can restore NSX-T and virtual machines from native and Veeam backups respectively, the admin runs a restore entire VM job of the three VMs previously backed up, and…

  • Points the Veeam restores to the Compute Cluster B host
  • Places them on the VM Network
  • Appends ‘_restored’ to each of their VM names
  • Leaves them powered Off. They’re left powered off so that once restored, the admin can adjust their network configurations to be attached to the ‘nsxt-restore-segment’.

restore_pract_pic9

Step 10 – Change Restored VMs networking to NSX-T Testing Segment – vCenter Server network vMotion

The restored VMs can easily be moved in bulk to the ‘nsxt-restore-segment’ by using the Migrate VMs to Another Network option.

restore_pract_pic10

Once the VMs are restored and moved to the ‘nsxt-restore-segment’, they can be powered on and the next step can proceed.

Step 11 – Add NSX-T Restore Config – NSX-T Policy API via PowerCLI

Now that the restored VMs are all added to the ‘nsxt-restore-segment’ and the new NSX-T Manager is online and attached as well, the admin can access these VMs by using the vSphere Client and using a direct console to the BACKUP-01a_restored VM.  It’s critical to run the remaining scripts from that machine, as there is no outside network access to the new NSX-T Manager appliance, as intended.

Consoling into the BACKUP-01a_restored server, the admin can make some checks to see if network connectivity is indeed limited to the ‘nsxt-restore-segment’.  Taking a quick look at the IPCONFIG of the BACKUP-01a_restored server, the admin can see that they cannot PING the default gateway of the network, however they are able to PING the other VMs and the NSX-T Manager (which has the same IP address as the Production NSX-T Manager).

restore_pract_pic11

The admin can also log into the UI of the NSX-T Manager from the BACKUP-01_restored server as well and can see that this is a brand-new deployment with no configurations.

restore_pract_pic12

The admin can also see that the Restore configuration is no longer configured as well.  The next step is to get the configuration for restoring the NSX-T Manager put back into the new NSX-T Manager.  This NSX-T Manager is already the same IP and Name as the production version, which is a requirement for restoration.

restore_pract_pic13

With connectivity to the NSX-T Manager, and confirmation that there’s no configurations, the admin can proceed with running the PowerCLI script to add the Restore Configuration into the NSX-T Manager from script 03_NSXT_RESTORE_CONFIG.ps1.

restore_pract_pic14

A quick run of the script and a refresh of the NSX-T Manager UI, and the admin can see that the SFTP server configuration is back and all of the backups that have been taken are showing up as well.

restore_pract_pic15

After checking the backup files, the admin picks the first one in the list of Available Backups and clicks on the restore button to apply the configuration.  During the restore process, since this is not a full restore and components such as Edge Nodes and Transport Node hosts are not contactable, the admin may get a few error messages that they can skip through.  Once the restore is done, the admin can take a look at the restored configuration and see that the NSX-T Manager configuration matches the production instance and the restore was successfully finished and validated.

restore_pract_pic16

restore_pract_pic17

With a successful test and the requirements accomplished, the admin can now perform the final steps running the last two scripts on the BACKUP-01a production server.  One of the scripts, 04_NSXT_RESTORE_CLEANUP.ps1 will shutdown and then forcibly delete all of the restored virtual machines and the NSX-T Manager.  The last script, 05_NSXT_DEPLOY_CLEANUP.ps1, runs a Policy API REST command to remove the Tier-1 Gateway and Segment to bring the entire deployment back to its original, clean state.

restore_pract_pic18

restore_pract_pic19

restore_pract_pic20

The last 2 posts have shown the Healthcare organization the power of using NSX-T and how it can be used with even a small amount of automated techniques to accomplish several use case examples and provide a real value to the organization that requires them to test their backups.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s