The VMware NSX Platform – Healthcare Series – Part 4.2: Micro-segmentation Practical with Application Rule Manager

Originally this series on Micro-segmentation was only going to cover Log Insight, vRealize Network Insight (vRNI), and VMware NSX.  With the release of VMware NSX 6.3, there is a new toolset within NSX that can be leveraged for quick micro-segmentation planning The Application Rule Manager (ARM) within NSX, provides a new way to help create security rulesets quickly for new or existing applications on a bigger scale than Log Insight, but smaller scale than vRNI.   With that in mind, we’re going to take the previous post using Log Insight, and perform the same procedures with ARM in NSX to create our rulesets using the same basic methodologies.

The Application Rule Manager in VMware NSX leverages real-time flow information to discover the communications both in and out, and between an application workload so a security model can be built around the application.  ARM can monitor up to 30 VMs in one session and have 5 sessions running at a time.  The beauty of ARM is that it can correlate the information that you would typically have to either have or look up in Log Insight to create your rulesets and significantly reduces time to value.  ARM can also show you blocked flows and the rules that are doing the blocking.

Let’s bring back our use case from the previous post and sub in, ARM instead.

Use case – Provide a Zero Trust security model around a Healthcare Organization’s EMR system.  Facilitate only the necessary communications both to the application and between the components of the application.

  • Allow EMR Client Application to communicate with EMR Web/App Server
  • Allow EMR Web/App Server to communicate with EMR Database Server
  • Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.
  • Allow bi-directional communication between the Infrastructure Services and the entire EMR Application

Problem – The Healthcare organization does not have a clear understanding of how the application communicates within and outside the organization.  Organization wants to lock down the EMR application so that only known good workstations can access.

Technology used –   

Windows Clients:

  • Windows 10 – Clinician Desktop – Client-01a (192.168.0.36)
  • Windows 10 – HR Desktop – Client-02a (192.168.0.33)
  • Mac OSX – Unauthorized Desktop – (192.168.0.18)

VMware Products

  • vSphere
  • vCenter
  • NSX and Application Rule Manager

Application in question –

Open Source Healthcare Application:

  • OpenMRS – Open Source EMR system
    • Apache/PHP Web Server
    • MySQL Database Server

Infrastructure Services:

  • NTP

The first thing we need to do to utilize ARM is establish the VM’s that we’re going to look at the flows from.  These systems will represent the session that we will build and run for a period of time to examine what flows are coming in and going out, and between these systems.

Since Infrastructure Services are generally global services that affect more than one system within a data center, we’re going to build the rules that accommodate our NTP virtual machine first.  We’re going to see more flows to other VM’s in the lab, but we’ll be narrowing it down to just the EMR systems.  Then when we’ve got those working, we’ll do the EMR-based rules.

First, we need to establish our naming scheme and methodology to writing our rule sets.  Below is a chart I use when creating my rules for the applications.  It helps me lay things out logically and then apply it quickly in the DFW.

ms_table_pic1

You will notice that I like to nest my Security, Services, and Groups.  I do this because at scale, this makes changing things much simpler and more efficient.  If I have to swap in a service or change a port, I want the changes to be reflected to all affected groupings down the table without having to seek out where else I might need to make a change.  This has consequences as well, you must pay attention to what you’re doing when you make a change as it can have effect on other applications.  This is why I name things the way I do, to ensure that any change you make that you understand the implications.

Now that we have all our custom services and naming convention in place, we can go back to the ARM and start resolving Services, Service Groups, and building out rules.  We’re going to take these flows and squeeze them down even further to create very succinct rules with as few entries as possible.

We’re going to create a session called INFRA MONITOR and add the NTP-01a server into the monitor so we can see all the flows coming in and out of it.

arm_ms_pic1

The amount of time that you collect flows for is up to you.  My advice is the following:

  • Understand the application you’re planning for. If this application is a billing cycle application that does 30 day monthly cycles, ARM my not be the tool to use.  ARM was built to monitor flows in real-time over the maximum of a 7-day period.  If the application you’re looking at has periodic flows that occur over a longer period of time, I suggest using vRealize Network Insight for those applications.  It has a longer retention period for looking up flows.
  • Keep the applications small. ARM has a 30 VM upper limit on monitoring of flows per session, with up to 5 sessions.  While most applications may fall into this range, some applications may have more than 30 machines.  In that case, my suggestion is breaking up the application into smaller chunks and running the chunks in different sessions to keep the numbers down.
  • ARM uses NSX Manager resources. Just like Flow Monitoring, ARM uses resources from the NSX Manager to collect flow data.  Be careful of NSX Manager utilization during this process.  Unlike Flow Monitoring, if you exit the interface, ARM will continue to run until you stop the session collection.

Once we hit OK, the flow collections will begin.

arm_ms_pic2

When we’re comfortable that we’ve captured all the flows from the VM, we can stop the collection and ARM will take the number of flows, de-duplicate the flows down to unique flows for us.

arm_ms_pic3

Now we’re going to analyze our flows.  This is going to resolve any VM names it can as well as resolve the ports to protocols that they could be.

arm_ms_pic4

I’ve hidden other flows to other systems within the lab for simplicity sake for now, and only focused on finding the flows to the EMR system.  You’ll also notice a column that I added into the View Flows tab.  The column is Rule ID.  The Rule ID column will show the applied rule that affects that flow in the View Flows table.

arm_ms_pic5

As you can see from the output, all the flows appear to be hitting the Rule ID of 1001.  If you click on the Rule ID 1001 link, it will pop open the details of the Rule from the DFW.  While this is showing an allowed flow, we will see later, that we can also see flows being blocked by a Rule ID.

arm_ms_pic6

The View Flows tab only resolves to VM names.  We’ll need to take this information and create appropriate Security Group, Service, and Service Group names for our rules.  We’ll click on the gear icon next to the EMR-DB-01a VM and select ‘Create Security Group and Replace’.

arm_ms_pic7

We’ll name the Security Group after the naming scheme you’ll find below of EMR-SG-DB and we’ll statically add in the VM, EMR-DB-01a.

arm_ms_pic8

What we’ll find is that the Security Group now replaces the virtual machine as the Source.  But since we’re both of the VM’s that make up the EMR talking to the same destination, we’ll create an EMR-SG-ALL Security Group and nest the individual VM Security Groups within it.  To do this, we can click on the gear for the Source we just changed, and select ‘Create Security Group and Replace’ same as before but this time we’re going to add the EMR-SG-DB Security Group to the EMR-SG-ALL Security Group we’re going to create.

arm_ms_pic9

We should now see that the Source has changed to the EMR-SG-ALL GROUP.

arm_ms_pic10

For the EMR-WEB-01a VM, we’ll do the same thing.  Create a Security Group called EMR-SG-WEB and add that VM to it.

arm_ms_pic11

We’ll then click on the gear and select ‘Add to existing Security Group and Replace’.  Then select the EMR-SG-ALL group, and this will nest the EMR-SG-WEB Security Group into the main group.

arm_ms_pic12

We repeat this same process for the NTP-01a VM according to the layout above.  Create an INFRA-SG-NTP and add the NTP-01a system.  Create a INFRA-SG-ALL and add the INFRA-SG-NTP Security Group to it.  This will allow us to add more SG’s to the main group later as needed without having to write another rule to do it.  When done, it should look like this.

arm_ms_pic13

You only need to do this to one rule as you’ll see later, we’ll grab both of these flows to write one rules.  We focus in on the Services that we found now.  If you click on the ‘2 Services’ link, you’ll see what ARM resolved the ports to, protocol-wise.

arm_ms_pic14

I like creating my own Services and Service Groups.  This way we know if we make changes, we know what could possibly be impacted.  So with this, we’re going to perform a similar workflow as we did above.  Create a Service and Replace.  Create a Service Group and Replace.  This will add the custom Service we create to the custom Service Group we create.

arm_ms_pic15arm_ms_pic16arm_ms_pic17arm_ms_pic18

Leaving us with this final result.

arm_ms_pic19

We’re now ready to create our rule from the two flows.  Selecting both rules, we right click and select ‘Create Firewall Rule’.

arm_ms_pic20

We’ll see some unresolved items in the list, but we’re going to make modifications so that this one rule covers both flows.  We’re going to:

  • Remove the NTP-01a as the Destination leaving the INFRA-SG-ALL
  • Remove the UDP 123 Service leaving the INFRA-SVG-ALL
  • Remove the vNIC for NTP-01a and add EMR-SG-ALL and INFRA-SG-ALL

arm_ms_pic21arm_ms_pic22

Once we click on OK, we’ll have a new rule created in the ‘Firewall Rules’ tab.

arm_ms_pic23

Double check our rule and make sure it looks correct and then hit ‘Publish’.  We’re then prompted to create a section name and select where to insert it.

arm_ms_pic24

We should get confirmation that the publish was successful and we can go to the ‘Firewall’ interface and verify our work.

arm_ms_pic25

Everything looks good!

Now we can focus on the EMR system.  Talking with our applications team, we have determined that the VM’s in question are:

  • EMR-WEB-01a
  • EMR-DB-01a

Now that we know the names of the VM’s, we can go into the Flow Monitoring section of the NSX Management console and select Application Rule Manager.  From here we can Start New Session.

We’ll select the VM’s we discussed above, EMR-WEB-01a and EMR-DB-01a, and we’ll add them to the session.  This will start collection of flow data from the vNIC’s of these VM’s and post them in the View Flows table.

From here we will create a new session, calling it EMR MONITOR, and add in our VM’s to the session.  Once we hit OK, the collection will start and we can stop the collection when we’re comfortable with the period of time we collected for.  In this instance, I’m going to collect data for 15 minutes.  I have automated tasks running to provide traffic to the EMR so we can see flows that run every 5 minutes, so this should be long enough for this demonstration.

arm_ms_pic26

Once we have stopped the collection, we should see the View Flows table has several flows showing.  ARM will attempt to de-duplicate repeated flows as much as possible.

arm_ms_pic27

Analyze the outputarm_ms_pic28

Now that ARM has analyzed our flow data and matched where it can we can see a few things:

  • Direction – This tells us in what direction the flow came to the source
    • IN – Inbound
    • OUT – Outbound
    • INTRA – Between
  • Source – Resolved to a VM name if the VM falls under the scope of the vCenter/NSX Manager relationship. If an IP address, represents an external IP system that is not resolvable in the vCenter/NSX Manager relationship.
  • Destination – Resolved to a VM name if the VM falls under the scope of the vCenter/NSX Manager relationship. If an IP address, represents an external IP system that is not resolvable in the vCenter/NSX Manager relationship.
  • Service – Resolved to all services that exist within NSX. If more than one service is shown, this means that the user will need to manually pick the service it correlates to because NSX has more than one service definition with that corresponding port number. If you want to create a custom Service or Service Group, which we will

With the information shown we have the following:

  • Inbound 192.168.0.99 > EMR-WEB-01a TCP 8080
    • 168.0.99 is identified as the Management Jumpbox for the infrastructure.
  • Inbound 192.168.0.36 > EMR-WEB-01a TCP 8080
    • 168.0.36 is identified as the Clinician Desktop
  • *Inbound HL7-01a > EMR-DB-01a TCP 3306 – Skipping this for now
  • *Outbound EMR-DB-01a > NTP-01a UDP 123 – Rule ID 1028
  • *Outbound EMR-WEB-01a > NTP-01a UDP 123 – Rule ID 1028
  • Intra EMR-WEB-01a > EMR-DB-01a TCP 3306

What you may notice when you have the Rule ID column unhidden, is that the two flows we already built for above in the Infrastructure section are showing as the Rule ID 1028 which is the Rule ID for that Infrastructure rule.  As I said before, ARM can show you any flow and the corresponding rule in the DFW that it’s hitting.  All of the rest of the flows are hitting the Default Allow Rule ID of 1001 just like Infrastructure before we started.

arm_ms_pic29

What this means for us, is that we’re already covered on Rule ID 1028 for NTP Services and we can hide these two flows.  We’re also going to hide the HL7-01a flow so we can see another interesting feature that ARM can show us.

arm_ms_pic30

This leaves us with three flows we need to write rules for.  Using the same methodology as we did above, we’re going to create our table for naming and apply it to the VM’s, Services, Security Groups, Service Groups for the EMR.

ms_table_pic2

Good news is that we’ve already created Security Groups for the EMR system when we created the Infrastructure Rules.  We just need to ‘Replace with Membership’ each VM with the appropriate Security Group.

arm_ms_pic31

When you select ‘Replace with Membership’, ARM will show you all the Security Groups that the VM belongs to.  You’ll notice the differences when you change EMR-WEB-01a and EMR-DB-01a.

arm_ms_pic32arm_ms_pic33

We do however, need to create IP Sets for the Desktop systems that we’re going to allow access to the EMR through.  We do this by clicking on the gear and selecting ‘Create IP Set and Replace’.

arm_ms_pic34arm_ms_pic35arm_ms_pic36

We then create the EMR-SG-ACCESS Security Group and add these two IP Sets to it and replace.

arm_ms_pic37arm_ms_pic38

We need to resolve our Services now as well.  We have two unresolved ports of 3306 and 8080.  Again, we’re going to refer to the table and create new Services and Service Groups for these.

arm_ms_pic39arm_ms_pic40arm_ms_pic41

There’s no reason to resolve the service for the second IP-based flow, as the services are the same and we’re going to create one rule like we did above for these two flows since they’re fundamentally the same.  Time to swap in Service Groups.

arm_ms_pic42arm_ms_pic43arm_ms_pic44

We’re now left with this output.

arm_ms_pic45

Let’s create our rules.  We’re going to select both of the IP-based rules and combine them.

arm_ms_pic46arm_ms_pic47

Cleaned up it looks like this

arm_ms_pic48

Now to create our INTRA communication rule between the EMR-WEB-01a and EMR-DB-01a VMs.

arm_ms_pic49

Cleaned up it looks like this

arm_ms_pic50

We now have our two rules that we can publish to the DFW for the EMR application.

arm_ms_pic51

We’ll create a new section called ‘EMR’ and place it below our Infrastructure Section.

arm_ms_pic52

Double check our work.

arm_ms_pic53

Things look correct.  Let’s add in our Block Rules so we can do our testing to make sure we did this correctly and it meets the requirements.

arm_ms_pic54

Here’s the list of requirements the customer gave us:

  • Allow EMR Client Application to communicate with EMR Web/App Server
  • Allow EMR Web/App Server to communicate with EMR Database Server
  • Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.
  • Allow bi-directional communication between the Infrastructure Services and the entire EMR Application

With ARM, we can check that these requirements are hitting the correct rules very quickly.  We’re going to create another Monitor session to monitor EMR-WEB-01a, EMR-DB-01a, and NTP-01a.  When we see the flows that come out, we should see the rules that they hit.  This will quickly tell us if things are working correctly.  I’m also going to generate traffic that should hit the block rules because they don’t meet the requirements.  Let’s start!

arm_ms_pic55

arm_ms_pic56

We can see flows hitting the Rule ID’s to the right which is good.  We’re going to stop the collection and do an analysis on the flows captured.

arm_ms_pic57

We can see the flows captured and the Rule ID’s associated.  Nothing appears to be hitting the Default Allow Rule ID 1001 which is good.  The Application is functional and NTP is accepting connections from the VMs.  If we take a look at the first requirement:

  • Allow EMR Client Application to communicate with EMR Web/App Server

arm_ms_pic58arm_ms_pic59

You can see that both of the IP-based flows are captured under Rule ID 1030 which when you click on it, shows the correct DFW rule.  Let’s check the next requirement.

  • Allow EMR Web/App Server to communicate with EMR Database Server

arm_ms_pic60

You can see that by clicking on the Rule ID of 1029 for the EMR-WEB-01a to EMR-DB-01a flow, that that traffic is hitting the correct rule.  Let’s check the next requirement.

  • Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.

arm_ms_pic61

When we look for block based rules, we’re looking for anything hitting a block rule in the DFW.  In this case, Rule ID 1032 is a Block rule.  We seem to have two flows that are hitting that rule.

arm_ms_pic62

When we drill into Rule ID 1032, we see that it is indeed, our Block rule.  Let’s look at the last requirement.

  • Allow bi-directional communication between the Infrastructure Services and the entire EMR Application

arm_ms_pic63

From this picture, we can see that there are two flows from the EMR VMs to the NTP server.  Both are hitting Rule ID 1028.  When we open up that rule we see that it is indeed hitting the correct rule to allow NTP traffic to flow to the NTP server from the EMR.

arm_ms_pic64

This completes our requirements set forth by the customer to meet to secure the EMR application.

As you can see, ARM is very adept at helping simply micro-segmentation with VMware NSX.  Taking the concepts we learned and leveraged with Log Insight, we can remove quite a bit of manual processes involved with resolving services and VM’s IP addresses to their names.  I was able to perform this process in about 15-20 minutes outside of flow capture time from start to finish and that’s what makes ARM another very useful toolset to use for reducing the complexity of micro-segmentation.

Advertisements

2 thoughts on “The VMware NSX Platform – Healthcare Series – Part 4.2: Micro-segmentation Practical with Application Rule Manager

  1. Pingback: Application Rule Manager (ARM) Practical Implementation - Healthcare - The Network Virtualization Blog

  2. Pingback: Application Rule Manager (ARM) Practical Implementation – Healthcare - CNARENA - Social Media | Content Sharing | Social Bookmarking

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s