The VMware NSX Platform – Healthcare Series – Part 4.1: Micro-segmentation Practical

In the previous blog post, we discussed how the concept of micro-segmentation provides a Zero-Trust security model for Healthcare applications.  We also discussed how that model would apply when we talk about security around a Healthcare organization’s EMR/EHR.  In this post we’re going to take those concepts and actually apply them to a Healthcare lab environment to show how we functionally achieve this outcome. With some applications however, organizations are not privy to the details of the communications flows for the application.  In this post, we’ll be leveraging VMware tools to help figure out how the application actually communicates so we can write our rule sets.  We’ll be focusing on using the NSX DFW and Log Insight to create our rules for the application.  The next blog post will be over using Service Composer and vRealize Network Insight and rule building at scale.


Use case – Provide a Zero Trust security model around a Healthcare Organization’s EMR system.  Facilitate only the necessary communications both to the application and between the components of the application.

  • Allow EMR Client Application to communicate with EMR Web/App Server
  • Allow EMR Web/App Server to communicate with EMR Database Server
  • Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.
  • Allow bi-directional communication between the Infrastructure Services and the entire EMR Application – We’re going to skip this part for now as we’ll add it in later when we expand the use case.

Problem – The Healthcare organization does not have a clear understanding of how the application communicates within and outside the organization.  Organization wants to lock down the EMR application so that only known good workstations can access.

Technology used –   

Windows Clients:

  • Windows 10 – Clinician Desktop (my jump box system)
  • Windows 7 – Non-Clinician Desktop (random system on the network)

VMware Products

  • vSphere
  • vCenter
  • NSX
  • vRealize Log Insight

Application in question –

Open Source Healthcare Application:

  • OpenMRS – Open Source EMR system
    • Apache/PHP Web Server
    • MySQL Database Server

With the EMR system I have deployed now, OpenMRS, it consists of two systems; the Web/App server and the Database server.  The Web/App server runs queries and application-specific functions against the Database server.

I’m not going to go through how to deploy and install the VMware NSX Platform.  Suffice it to say, that’s covered very well in many other blog posts and deployment is rather trivial.  In this environment, I have 3 ESXi servers in my Compute1 Cluster.  All three hosts are prepared with the VMware NSX Distributed Firewall (DFW) software bundle.  Once the NSX DFW has been deployed, all virtual machines that reside on those hosts will be covered by the Layer 2-4 stateful Distributed Firewall that NSX provides at the virtual network card level of each virtual machine. To start the process of locking down the communications between the systems we need to first come up with our methodology for doing so.  Since the NSX Manager is connected to the vCenter Server, we can use vCenter-type objects to build our rules, in this case we’re going to use VM names.  When we move these systems to different networks and the rules still work, this will make more sense and show the agility of the VMware NSX Platform.


For applications we’re unsure of how they interact, there are tools and methods to help build your rule sets for NSX DFW.  I going to use vRealize Log Insight and also vRealize Network Insight.  Both provide very granular ways to help build your rule sets and offer slightly different approaches overall.  When you write your rules you can leverage either the NSX DFW or Service Composer.

First let’s start with the methodology I use to build rules using NSX DFW and Log Insight

  • Create NSX Security Groups and Security Tags for each of the different ‘tiers’ of the application in question. The Security Tags will be applied to the VMs of each tier, and the Security Tag will be the criteria which places the VMs into the Security Group.  There are many different ways to included VMs in a Security Group and tags is just one.  We’ll be looking at more in future posts.  The IP addresses are there for reference purpose, not as a criterion for inclusion. 
    • Security Group – Application
      • EMR-SG-WEB
      • EMR-SG-DB
    • Security Tag – Application
      • EMR-ST-WEB
        • EMR-WEB-01a –
      • EMR-ST-DB
        • EMR-DB-01a –
  • Create an NSX Security Group for the entire application. This will be used to nest the different tier Security Groups into.
    • Security Group
      • EMR-SG-APP
        • EMR-SG-WEB
        • EMR-SG-DB
  • Create firewall rules in the NSX DFW interface. We start with very general rules for the entire application.  Once we take a look at the flows within Log Insight, we can write more specific rules for the application that will only comprise of the necessary ports and protocols for the application to function.
    • Allow All Inbound Log
      • Rule ID 1008 – DFW
    • Allow All Outbound Log
      • Rule ID 1007 – DFW
    • Block All Inbound Log
      • Rule ID 1006 – DFW
    • Block All Outbound Log
      • Rule ID 1005 – DFW

Security Groups


Security Tags


DFW Rules


This methodology allows us to see all the traffic coming in and out and between the application and pipes it all to our logging application.  As the application generates traffic, we’ll be able to use either Log Insight or Network Insight to see it.  We simply apply the Security Tag to VMs and they’re placed into the appropriate Security Groups and subsequently the NSX DFW rules are applied.

Now that we have the application traffic being logged and the rules placed into the NSX DFW, we will bring up vRealize Log Insight and take a look at the traffic patterns.


As we can see, we’re getting traffic connection hits on the two rules we should be getting hits on, the 1007 and 1008 rules which are the Allow All Inbound/Outbound Log rules.  This is exactly what we should be seeing.  When we dig into the connections and do an Interactive Analysis on each of the hits we’re getting on these two rules we see the following in the Field Table:


We can see that our rules are working and when traffic is generated to the application, we’re seeing the connections being established within the application and to the application.  With Log Insight we’re constrained to only seeing the IP address of the logs and not the DNS names. We can now pick out the connections to start writing our more specific rules:

  • > over TCP 8080
    • is the Clinician Desktop that’s accessing the EMR WEB server. Since we were unsure who was accessing the EMR and we want to lock it down so only specific machines can access, we’ll leverage an IP Set here since this machine is outside the NSX environment.
  • > over TCP 3306
  • > over TCP 3306
    • This is an exampled of a stateful TCP flow. The application established a connection over 3306 to the database server and the database server responded back on 3306.  Since the NSX DFW is stateful, we can write one rule to allow the application server to talk to the database over 3306 and the database can response back on the same port without needing to open any ports or writing any rules for the database server for that communication to occur.

We’ll quickly add a new Security Group and IP Set for the Jump Box system so we can build a rule that only allows this system to communicate with the EMR.  I have another Windows 7 desktop called CWS-01a,, that we will attempt connection to the EMR from and show how we can restrict who can actually hit the EMR application from the web interface.


Before we write our rules, the port 8080 does not match anything in the NSX Services list specifically for this application.  However, there is a listing for 3306, MySQL.  Being that this is the most critical Healthcare application, I recommend creating your own Services and Service Groups within NSX to accommodate these ports regardless of whether or not they’re in the Services list.  This provides a visual cue to anyone looking at modifying a service in NSX.  They will immediately know that this service is in use by the EMR and that they should be very careful making changes or removing it from NSX.

Services and Service Groups behave in a similar fashion to Security Groups.  You simply create the Services you want and add them as members of the Service Group.  You can find a post a did a while back on how to go through this process here.  I’ve already went ahead and built and added them.



When we take all this information and put it into NSX DFW rules we get the following configurations:


Let’s break down this output and explain this in simple terms.

  • Rule 1 – Restrict EMR Access
    • EMR-SG-ACCESS (IP SET = > EMR-SG-WEB over EMR-SVG-WEB (TCP 8080) is Allow and Applied To EMR-SG-WEB

Functionally we are allowing a system outside the NSX environment, my jump box with an IP of, access to the EMR-WEB-01a system over TCP port 8080.  Were applying this rule only to the EMR-SG-WEB Security Group.  This means that only the EMR-WEB-01a will get this rule in its firewall.  This reduces adding rules that don’t need to apply to a VM where they’re not needed.  The power of NSX in this scenario is that if the EMR system adds another WEB server that needs to talk to the DB, we can simply apply the EMR-ST-WEB Security Tag and that system will be immediately put into the EMR-SG-WEB Security Group and the above NSX DFW rules will apply to that machine too!  Consistently applying security through operational simplicity is just one of the many benefits NSX provides.

We then applied a similar rule to only allow EMR-WEB-01a to communicate with EMR-DB-01a over 3306 and applied those rules to each of the Security Groups to reduce rule sprawl.  Going back to the discussion about stateful firewalling from above, we only needed to add this one rule and when the WEB system communicates over 3306 to the DB, the response communication back to the WEB system is not necessary.

Finally, we have the last 4 rules for logging inbound and outbound communications for the entire application.  Now that we have more granular rules and have reduced the number of open ports down to the 2 necessary, we can turn off our Allow All rules and ensure the application is functioning.  At this point we’re not seeing any communication in and/or out of the application to any other systems so we can comfortably test the application.  Let’s go back to our use cases and requirements to see if we fulfilled them properly:

  • Allow EMR Client Application to communicate with EMR Web/App Server
    • Built EMR-SG-ACCESS > EMR-SG-WEB over 8080
  • Allow EMR Web/App Server to communicate with EMR Database Server
    • Built EMR-SG-WEB > EMR-SG-DB over 3306
  • Block any unknown communications except the actual application traffic and restrict access to the application to only a Clinician Desktop system running the EMR Client Application.
    • Restricted access to only
      • Test attempt to login from another system,, and verify that the access is blocked and hitting Rule ID 1006 that blocks unauthorized access to the application.
      • Test attempt to SSH from Web to DB
  • Allow bi-directional communication between the Infrastructure Services and the entire EMR Application – We’re going to skip this part for now as we’ll add it in later when we expand the use case.
    • We’re going to follow up on this requirement in the next post as we expand the environment out further to include more applications.

I’ve turned off the Allow All rules and we can now test our results to ensure the application works and we fulfill the requirements asked. Disabling the rules turns them grey.


Allow EMR Client Application to communicate with EMR Web/App Server – Rule 1009Allow EMR Web/App Server to communicate with EMR Database Server – Rule 1010



We can access the application from the Clinician Desktop as expected and application is working as we would not be able to login and perform a patient lookup.

Let’s see if the block requirements are working properly.

Block any unknown communications except the actual application traffic and restrict access to the application to only a Clinician Desktop system running the EMR Client Application.  Rule – 1006

  • Restricted access to only
    • Test attempt to login from another system,, and verify that the access is blocked and hitting Rule ID 1006 that blocks unauthorized access to the application.
    • Test attempt to SSH from Web to DB


Trying to SSH from WEB to DB


Let’s look in Log Insight to confirm


Looks like we have covered all the bases with our rule sets we built.  The application is functional and the appropriate systems are able to access the EMR application.

I know this post was fairly long for an application that only had two ports, but in the next post we’re going to be adding more complexity to this environment.  The fundamentals are sound as we’ll be applying them to the rest of the applications we introduce.  Remember, the Healthcare EMR is connected to many ancillary systems in a Healthcare Organization, it doesn’t just function on its down. We’ll be adding in systems that will talk to our EMR and showing how to do micro-segmentation at scale using vRealize Network Insight and then leveraging Service Composer with NSX.  This will setup the foundation for subsequent posts in the series.

The VMware NSX Platform – Healthcare Series Part 3 – Micro-Segmentation Concept

When using an application-based policy approach, security is a critical part to the application workload.  Security is just as important as how much CPU or RAM you give an application workload.  The VMware NSX Platform introduces 3 primary use cases when it comes to security for application workloads.  We’re going to focus on the first use case:  Micro-segmentation and how it relates to Healthcare organizations.


A quick background on why Micro-segmentation is important and security trends in modern data centers. In most modern data centers, there has been a large uptick in the amount of traffic that occurs between systems rather than inbound-and-outbound of systems.  This is referred to as East-West traffic within the data center, versus North-South traffic in and out of the data center.  In the hardware-base world, security for East-West traffic is sometimes done either by sending the traffic from their applications to their perimeter firewalls or by purchasing hardware appliances to put inside the data center between the applications. This form of security is through isolation and segmentation of the applications.  You can do this at the entire application level through concepts such as Trust Zones, achieving what some call macro-segmentation, but when we place security at the workload level we achieve what’s called, micro-segmentation.  Micro-segmentation facilitates a Zero-Trust security model. Zero-Trust means that unless communications between systems are explicitly trusted, it’s implicitly untrusted.  The application of Micro-segmentation using a hardware-based approach creates a two-fold problem:

Lack of interior controls and security – If an organization simply does nothing, no use of micro-segmentation, to secure the East-West traffic within their data center, the perimeter firewall becomes the single point of entry and security to their environment from North-South.  This type of defense still provides protection, however once an attacker is able to break through that perimeter, unfettered access around the inside of the environment is rather easy.  With no lateral controls, the attack surface is enormous for attackers to work with.


Operationally infeasible and lack of scale – If an organization puts in hardware appliances or use the external firewall to facilitate Micro-segmentation for both East-West and North-South traffic firewalling, those systems become operationally difficult to manage.  Hardware appliances are costly and only scale to a point.  Multiple user interfaces and policies that don’t scale as you add new workloads or even modify existing workloads into your data center and certainly don’t provide mobility in a virtualized environment.  As you add new applications you may need to add more firewalls.  If a workload needs to move, you may need to move or change the rules associated with that application.  And what happens when that application needs to talk to another application?  All those rules need to change as well.  While this can reduce the attack surface of the application, it’s operationally infeasible to support and lacks scaling as a long term option for customers.


How does the VMware NSX Platform provide a business value around these issues? The VMware NSX Platform uses a software-defined micro-segmentation approach applied at the Virtual Machine workload level to facilitate a Zero-Trust security model.  This security is built into the vSphere hypervisor creating a distributed and scale-out firewalling architecture.  This architecture provides kernel-level performance and scales as your organization and workload requirements increase without the need to add more specific hardware appliances to the environment.


Let’s focus on Healthcare customers specifically.  A recent study by the Ponemon Institute and IBM for 2016, shows that a security breach and exposure of a patient health record is now averaging $355 per record.  The average total cost to an organization in the US was $7.01 million and the average number of records breached was around 29,000.  Traditional methods of security, like those listed above, are no longer sufficient to prevent attacks.  While there is no ‘silver bullet’ to security, Healthcare organizations can provide a layered approach to security that helps reduce their attack surface overall and reduce the potential for exposure. The VMware NSX Platform, through the use of Micro-segmentation, allows Healthcare organizations to accomplish this.

The VMware NSX Platform can provide an application-based security policy around the critical and patient information sensitive applications within the data center.  This provides us the ability to effectively control all communication paths both in and out of the application, thus reducing the attack surface of that application immensely.

The EMR/EHR system for Healthcare organizations, represents a mission-critical application for the organization and houses the majority of patient record information.  For this example, we’re going to look at a typical installation of an Electronic Medical/Health Records system, EMR/EHR, and how traffic both in and out and between the servers within the application are secured. They can be comprised of several Windows/Linux and Appliance-based systems. Below is a typical example of the layout of an EMR/EHR system server architecture.  Most consist of a client application that connects to the Application Server which has a Database Server connection where the data is stored.


Let’s take a look at traditional security approaches to East-West and North-South traffic isolation, first.  You’ll see below that for North-South traffic, the end user workstation could traverse through either a perimeter firewall or an internal data center firewall before it gets to the presentation layer of the EMR/EHR.  Also from an East-West perspective, to secure communications between the servers within the application and Shared Services, the traffic patterns will need to traverse through those same firewalls to either allow or block the communications necessary for the application to function. This creates a hair pinning effect that is operationally inefficient.


With the Physical Firewall Policy, we’re now sending all the traffic through the external firewall to do the segmentation for the applications.  This firewall could also be an internal firewall between the applications.  Nevertheless, the premise stands.  Sending all the traffic through that firewall will not scale-out as your workloads increase and this is just one application in this environment.  Most Healthcare organizations have hundreds of applications they need to secure.

With the VMware NSX Platform, we instantiate a stateful, Layer 2-4, firewall at the Virtual Machine virtual network card level, which allows us to create security policies based on the application, that can secure the application in the host itself, rather than traversing to an external firewall.  This reduces the dependency on the external and internal physical firewalls for allowing and disallowing of traffic both in and out and between the EMR/EHR system and provides a much more operationally efficiency configuration for both network and operational resources.


As you can see, the VMware NSX Platform has Security Policies created for each of the different applications, in this case the EMR App Server, the EMR DB Server, and the Infrastructure Services Servers.  Through micro-segmentation, we can setup NSX Security Policies that only allow the traffic that needs to occur within the application, to actually occur.  This enforcement is done in the hypervisor with no need to traverse to a hardware Firewall device to secure the workloads.  What we see here is:

  • The EMR Client Application initiates a connection to the EMR App Server.
  • The EMR App Server allows inbound communications to occur with the EMR Client Application and also allows communication to the EMR DB server.
  • The EMR DB Server only allows connections inbound from the EMR App Server. This functionally secures the EMR application to only allow the communications that are necessary for the application to function, and the EMR Client Application to connect to the system securely.
  • The EMR App and DB Servers are also allowing both inbound-and-outbound, communications to the Infrastructure Services servers.

Using the VMware VMware NSX Platform, Healthcare organizations can implement security at a much more granular level that provides a simple way to secure Healthcare organizations application workloads and reduce their attack surface.  Security is now implemented at the virtual machine workload level using the Application-Based Policy control.  This new model, scales as the application workload scale in the data center environment while still providing the same security posture consistently.

The VMware NSX Platform – Healthcare Series Part 2 – Environment

To get started with the series, we need to establish the environment that we’re going to use to apply the NSX platform concepts on and the applications we’re going to use.  Access to real world applications is a great way to break out of the typical test application bubble and really apply these concepts.  One of the requirements I had for this environment was that all the software had to be in use somewhere in production and the applications needed to obviously be Healthcare applications.  Getting access to a full EMR, PACS, HR, or other Healthcare-related systems would be very difficult and costly.  However, after some serious digging I found open source alternatives that are in real-world usage today.  These applications can facilitate the points of how Healthcare application integrate with each other and are representative of typical applications in a Healthcare organization.

With that in mind, using a combination of open source software, VMware products, and 3rd party partner products, we’ll be able to showcase multiple use cases and how the NSX platform can help customers move toward a Software-Defined Datacenter and drive application-based policy through the stack.

Open Source Healthcare Applications:

  • OpenMRS – EMR system
    • Apache/PHP Web Server
    • MySQL Database Server
  • DCM4CHEE – PACS (Picture and Archiving Communication System)
    • Apache/PHP Web Server
    • MySQL Database Server
  • OrangeHRM – HRHIS (Human Resource for Health Information System)
    • Apache/PHP Web Server
    • MySQL Database Server
  • Weasis – DICOM (Digital Imaging and Communications in Medicine) system
    • Runs on a Windows-based system
  • DTKv – DICOM Emulator
    • Runs on a Windows-based system
    • Used to simulate a modality for the PACS system
      • Modality – MRI, CAT Scan, etc.
  • Mirth Connect – HL7 Interface Engine
    • Apache/Java
    • MySQL Database Server
  • DHIS – Health Information Management System/Data Warehouse
  • i2b2 – Informatics and Data Warehouse

VMware Applications:

  • vSphere
  • vCenter
  • NSX
  • vRealize Log Insight
  • vRealize Network Insight
  • vRealize Operations Manager
  • vRealize Automation
  • vCloud Air – Advanced Networking Services
  • Site Recovery Manager
  • vSphere Replication

3rd Party Applications:

  • Palo Alto
    • Panorama
    • V1000MH
  • Trend Micro
  • EMC Virtual Isilon

Infrastructure Servers:

  • Active Directory
  • DNS
  • NTP

This list subject to change and will grow over time, but provides the foundational basics we need to setup a true re-creation of what a Healthcare organization would be using in their own organizations. We’ll be using the new nested system I built to provide the environment to test in.

Starting with a conceptual design we have the following for our environment:


As we continue with the series, we’ll add in a second site for discussions around Business Continuity and Disaster Recovery, Metro-Pooling, Hybrid-Cloud use cases.

The VMware NSX Platform – Healthcare Series Part 1 – Intro

At VMware, I’m part of a dedicated team of professionals that are focused solely on Healthcare organizations.  My role is around providing NSX specialist support to my customers.  If you’re in the Healthcare field, you know that Healthcare has a vastly different approach when it comes to business goals than say Financial or Retail.  Healthcare customers are focused on patient care, driving down costs, security of patient information, and how fast they can innovate their services to their patients. Healthcare also brings up interesting and different challenges from a business and use case perspective.  We believe that the NSX platform can help organization with the business goals that matter to them, and more specifically we’re going to examine how Healthcare organizations can benefit.

The NSX platform is one of the three pillars in VMware’s Software-Defined Data Center (SDDC) approach.  When extended to our Compute virtualization platform, vSphere, and our Software Defined Storage platform, VSAN, the NSX platform provides the software networking portion of the infrastructure that allows an organization to drive Application-based policy through the entire infrastructure stack.


Similar to how vSphere disrupted the hardware compute business by removing dependency on specific hardware to run application workloads, the NSX platform is doing the same for the networking hardware layer. We’re not saying the hardware layer is not important and/or irrelevant, but driving application-based policies through hardware intelligence is something that can often times require specific and even proprietary hardware to accomplish.  With a software-defined approach, customers can choose the hardware of their choice and budget, and layer on the same hardware intelligence through software products and gain speed and agility, but not at the cost of security.  This means the focus of organizations moves to the application where it should be, and less on the hardware-based infrastructure.

When looking at a Healthcare organization and asking how the NSX platform can help achieve their goals, we need to examine the NSX use cases and find out how they apply to a Healthcare organization.  The NSX platform provides several outcomes for organizations:



As you can see, the NSX platform is truly a strategic platform, and not simply a tactical solution for one business goal a customer wishes to accomplish.  The NSX platform extends into multiple areas of the infrastructure and customers are seeing tremendous benefits.  While some customers often start with a single use case, we’re seeing our customers immediately find other use cases for the product.  Over the next series of blog posts, we will be digging into each of the use cases for the NSX platform listed above and how these use cases are applying to Healthcare organizations.

New Home Lab Build

In previous iterations of my home lab, I’ve built on the premise of keeping things as close to a ‘production’ system as I could.  This meant running physical equipment and setting up scenarios with that line of thinking. This grew from two physical hosts that I bought back in 2010 and then grew into the three vSAN nodes that I added in 2014 and Cisco 3750G switches that I added in as well.  I managed to build out a dual-DC type of configuration and it worked great for quite a while.

Since joining VMware in late 2014, my lab has been satisfactory to keep my skills current.  As I made the transition to the Networking and Security Business Unit, NSBU, at VMware in January of 2016, the lab has been in need of some updating.  Recently I made a decision to build out a new home lab, this time consisting of high-end pieces and going with nesting.  I sold off my vSAN nodes, moved one of my old servers to be my wife’s new desktop, it’s still a badass for what she needs, and I got rid of the loud and power-hungry 3750G’s.  The result has been a dramatic reduction in heat and noise in my office which is where I had to put my lab as I’m not fortunate to have basement where I can this gear.

As I started going down the path of deciding what I wanted to build, I knew it had to be powerful, quiet, and as power-efficient as possible.  I wanted maximum capabilities to run multiple topologies and basically be a on-premises version of VMware’s OneCloud environment we have to demo and testing.  With my new job in the NSBU and working with the VMware NSX platform, there are so many different use cases that I wanted to be able to setup and test out that my Healthcare customers may run into.  With some back and forth with my homeDC peer, Erik Bussink, I came up with a build that does just that.

The build consists of the following:

The best thing about this was I was able to sell all the other parts of my home lab and not have to add too much more cash to build this.  I got a really good deal on the NVMe drive off eBay as those can run as much as $2500 by themselves.

The rest of my lab consists of the following:

The result has been a new high-powered and fully capable lab system that meets all my requirements.