The VMware NSX Platform – Healthcare Series – Part 4.1: Micro-segmentation Practical

In the previous blog post, we discussed how the concept of micro-segmentation provides a Zero-Trust security model for Healthcare applications.  We also discussed how that model would apply when we talk about security around a Healthcare organization’s EMR/EHR.  In this post we’re going to take those concepts and actually apply them to a Healthcare lab environment to show how we functionally achieve this outcome. With some applications however, organizations are not privy to the details of the communications flows for the application.  In this post, we’ll be leveraging VMware tools to help figure out how the application actually communicates so we can write our rule sets.  We’ll be focusing on using the NSX DFW and Log Insight to create our rules for the application.  The next blog post will be over using Service Composer and vRealize Network Insight and rule building at scale.

Picture1.png

Use case – Provide a Zero Trust security model around a Healthcare Organization’s EMR system.  Facilitate only the necessary communications both to the application and between the components of the application.

  • Allow EMR Client Application to communicate with EMR Web/App Server
  • Allow EMR Web/App Server to communicate with EMR Database Server
  • Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.
  • Allow bi-directional communication between the Infrastructure Services and the entire EMR Application – We’re going to skip this part for now as we’ll add it in later when we expand the use case.

Problem – The Healthcare organization does not have a clear understanding of how the application communicates within and outside the organization.  Organization wants to lock down the EMR application so that only known good workstations can access.

Technology used –   

Windows Clients:

  • Windows 10 – Clinician Desktop (my jump box system)
  • Windows 7 – Non-Clinician Desktop (random system on the network)

VMware Products

  • vSphere
  • vCenter
  • NSX
  • vRealize Log Insight

Application in question –

Open Source Healthcare Application:

  • OpenMRS – Open Source EMR system
    • Apache/PHP Web Server
    • MySQL Database Server

With the EMR system I have deployed now, OpenMRS, it consists of two systems; the Web/App server and the Database server.  The Web/App server runs queries and application-specific functions against the Database server.

I’m not going to go through how to deploy and install the VMware NSX Platform.  Suffice it to say, that’s covered very well in many other blog posts and deployment is rather trivial.  In this environment, I have 3 ESXi servers in my Compute1 Cluster.  All three hosts are prepared with the VMware NSX Distributed Firewall (DFW) software bundle.  Once the NSX DFW has been deployed, all virtual machines that reside on those hosts will be covered by the Layer 2-4 stateful Distributed Firewall that NSX provides at the virtual network card level of each virtual machine. To start the process of locking down the communications between the systems we need to first come up with our methodology for doing so.  Since the NSX Manager is connected to the vCenter Server, we can use vCenter-type objects to build our rules, in this case we’re going to use VM names.  When we move these systems to different networks and the rules still work, this will make more sense and show the agility of the VMware NSX Platform.

Picture2.png

For applications we’re unsure of how they interact, there are tools and methods to help build your rule sets for NSX DFW.  I going to use vRealize Log Insight and also vRealize Network Insight.  Both provide very granular ways to help build your rule sets and offer slightly different approaches overall.  When you write your rules you can leverage either the NSX DFW or Service Composer.

First let’s start with the methodology I use to build rules using NSX DFW and Log Insight

  • Create NSX Security Groups and Security Tags for each of the different ‘tiers’ of the application in question. The Security Tags will be applied to the VMs of each tier, and the Security Tag will be the criteria which places the VMs into the Security Group.  There are many different ways to included VMs in a Security Group and tags is just one.  We’ll be looking at more in future posts.  The IP addresses are there for reference purpose, not as a criterion for inclusion. 
    • Security Group – Application
      • EMR-SG-WEB
      • EMR-SG-DB
    • Security Tag – Application
      • EMR-ST-WEB
        • EMR-WEB-01a – 192.168.0.25
      • EMR-ST-DB
        • EMR-DB-01a – 192.168.0.27
  • Create an NSX Security Group for the entire application. This will be used to nest the different tier Security Groups into.
    • Security Group
      • EMR-SG-APP
        • EMR-SG-WEB
        • EMR-SG-DB
  • Create firewall rules in the NSX DFW interface. We start with very general rules for the entire application.  Once we take a look at the flows within Log Insight, we can write more specific rules for the application that will only comprise of the necessary ports and protocols for the application to function.
    • Allow All Inbound Log
      • Rule ID 1008 – DFW
    • Allow All Outbound Log
      • Rule ID 1007 – DFW
    • Block All Inbound Log
      • Rule ID 1006 – DFW
    • Block All Outbound Log
      • Rule ID 1005 – DFW

Security Groups

Picture3.png

Security Tags

Picture4.png

DFW Rules

Picture5.png

This methodology allows us to see all the traffic coming in and out and between the application and pipes it all to our logging application.  As the application generates traffic, we’ll be able to use either Log Insight or Network Insight to see it.  We simply apply the Security Tag to VMs and they’re placed into the appropriate Security Groups and subsequently the NSX DFW rules are applied.

Now that we have the application traffic being logged and the rules placed into the NSX DFW, we will bring up vRealize Log Insight and take a look at the traffic patterns.

Picture6.png

As we can see, we’re getting traffic connection hits on the two rules we should be getting hits on, the 1007 and 1008 rules which are the Allow All Inbound/Outbound Log rules.  This is exactly what we should be seeing.  When we dig into the connections and do an Interactive Analysis on each of the hits we’re getting on these two rules we see the following in the Field Table:

picture7

We can see that our rules are working and when traffic is generated to the application, we’re seeing the connections being established within the application and to the application.  With Log Insight we’re constrained to only seeing the IP address of the logs and not the DNS names. We can now pick out the connections to start writing our more specific rules:

  • 192.168.0.99 > 192.168.0.25 over TCP 8080
    • 192.168.0.99 is the Clinician Desktop that’s accessing the EMR WEB server. Since we were unsure who was accessing the EMR and we want to lock it down so only specific machines can access, we’ll leverage an IP Set here since this machine is outside the NSX environment.
  • 192.168.0.25 > 192.168.0.27 over TCP 3306
  • 192.168.0.27 > 192.168.0.25 over TCP 3306
    • This is an exampled of a stateful TCP flow. The application established a connection over 3306 to the database server and the database server responded back on 3306.  Since the NSX DFW is stateful, we can write one rule to allow the application server to talk to the database over 3306 and the database can response back on the same port without needing to open any ports or writing any rules for the database server for that communication to occur.

We’ll quickly add a new Security Group and IP Set for the Jump Box system so we can build a rule that only allows this system to communicate with the EMR.  I have another Windows 7 desktop called CWS-01a, 192.168.0.33, that we will attempt connection to the EMR from and show how we can restrict who can actually hit the EMR application from the web interface.

Picture8.png

Before we write our rules, the port 8080 does not match anything in the NSX Services list specifically for this application.  However, there is a listing for 3306, MySQL.  Being that this is the most critical Healthcare application, I recommend creating your own Services and Service Groups within NSX to accommodate these ports regardless of whether or not they’re in the Services list.  This provides a visual cue to anyone looking at modifying a service in NSX.  They will immediately know that this service is in use by the EMR and that they should be very careful making changes or removing it from NSX.

Services and Service Groups behave in a similar fashion to Security Groups.  You simply create the Services you want and add them as members of the Service Group.  You can find a post a did a while back on how to go through this process here.  I’ve already went ahead and built and added them.

picture9

picture10

When we take all this information and put it into NSX DFW rules we get the following configurations:

Picture11.png

Let’s break down this output and explain this in simple terms.

  • Rule 1 – Restrict EMR Access
    • EMR-SG-ACCESS (IP SET = 192.168.0.99) > EMR-SG-WEB over EMR-SVG-WEB (TCP 8080) is Allow and Applied To EMR-SG-WEB

Functionally we are allowing a system outside the NSX environment, my jump box with an IP of 192.168.0.99, access to the EMR-WEB-01a system over TCP port 8080.  Were applying this rule only to the EMR-SG-WEB Security Group.  This means that only the EMR-WEB-01a will get this rule in its firewall.  This reduces adding rules that don’t need to apply to a VM where they’re not needed.  The power of NSX in this scenario is that if the EMR system adds another WEB server that needs to talk to the DB, we can simply apply the EMR-ST-WEB Security Tag and that system will be immediately put into the EMR-SG-WEB Security Group and the above NSX DFW rules will apply to that machine too!  Consistently applying security through operational simplicity is just one of the many benefits NSX provides.

We then applied a similar rule to only allow EMR-WEB-01a to communicate with EMR-DB-01a over 3306 and applied those rules to each of the Security Groups to reduce rule sprawl.  Going back to the discussion about stateful firewalling from above, we only needed to add this one rule and when the WEB system communicates over 3306 to the DB, the response communication back to the WEB system is not necessary.

Finally, we have the last 4 rules for logging inbound and outbound communications for the entire application.  Now that we have more granular rules and have reduced the number of open ports down to the 2 necessary, we can turn off our Allow All rules and ensure the application is functioning.  At this point we’re not seeing any communication in and/or out of the application to any other systems so we can comfortably test the application.  Let’s go back to our use cases and requirements to see if we fulfilled them properly:

  • Allow EMR Client Application to communicate with EMR Web/App Server
    • Built EMR-SG-ACCESS > EMR-SG-WEB over 8080
  • Allow EMR Web/App Server to communicate with EMR Database Server
    • Built EMR-SG-WEB > EMR-SG-DB over 3306
  • Block any unknown communications except the actual application traffic and restrict access to the application to only a Clinician Desktop system running the EMR Client Application.
    • Restricted access to only 192.168.0.99
      • Test attempt to login from another system, 192.168.0.33, and verify that the access is blocked and hitting Rule ID 1006 that blocks unauthorized access to the application.
      • Test attempt to SSH from Web to DB
  • Allow bi-directional communication between the Infrastructure Services and the entire EMR Application – We’re going to skip this part for now as we’ll add it in later when we expand the use case.
    • We’re going to follow up on this requirement in the next post as we expand the environment out further to include more applications.

I’ve turned off the Allow All rules and we can now test our results to ensure the application works and we fulfill the requirements asked. Disabling the rules turns them grey.

picture12

Allow EMR Client Application to communicate with EMR Web/App Server – Rule 1009Allow EMR Web/App Server to communicate with EMR Database Server – Rule 1010

Picture13.png

picture15picture14

We can access the application from the Clinician Desktop as expected and application is working as we would not be able to login and perform a patient lookup.

Let’s see if the block requirements are working properly.

Block any unknown communications except the actual application traffic and restrict access to the application to only a Clinician Desktop system running the EMR Client Application.  Rule – 1006

  • Restricted access to only 192.168.0.99
    • Test attempt to login from another system, 192.168.0.33, and verify that the access is blocked and hitting Rule ID 1006 that blocks unauthorized access to the application.
    • Test attempt to SSH from Web to DB

Picture16.png

Trying to SSH from WEB to DB

Picture17.png

Let’s look in Log Insight to confirm

Picture18.png

Looks like we have covered all the bases with our rule sets we built.  The application is functional and the appropriate systems are able to access the EMR application.

I know this post was fairly long for an application that only had two ports, but in the next post we’re going to be adding more complexity to this environment.  The fundamentals are sound as we’ll be applying them to the rest of the applications we introduce.  Remember, the Healthcare EMR is connected to many ancillary systems in a Healthcare Organization, it doesn’t just function on its down. We’ll be adding in systems that will talk to our EMR and showing how to do micro-segmentation at scale using vRealize Network Insight and then leveraging Service Composer with NSX.  This will setup the foundation for subsequent posts in the series.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s