VMworld 2019 Sessions and looking forward

It’s been well over a year since I last posted anything on my blog, but since moving into the Networking and Security Business Unit at VMware and my girls getting older, I’ve been focusing on other commitments.  That being said, I have been compiling a list of things that I plan to start working on and blogging over the next several months to bring new content back to the blog.  Stay tuned, I have a massive list that I want to get posted.

Now for a bit of shameless self-promotion of my own sessions as well as ones from my peers. 🙂

We’re now just a few short weeks away from VMworld US 2019.  There are well over 200 different sessions for security including Breakout Sessions, Deep Dives, Self-Paced and Expert-led Hands-on labs, Meet the Expert, and many more.  Discussing with my peers, their sessions are filling up so if you haven’t signed up for all of your sessions take a look at the ones below and get them scheduled.

Below are the sessions that I will be presenting with one of my customers, CHRISTUS Health, my partner in EUC crime, Graeme Gordon, and last but not least my deep dive on NSX-T Guest Introspection (Endpoint Protection) that came out recently with the NSX-T 2.4 release.

• Agentless Anti-Virus with NSX-T Guest Introspection Deep Dive [SAI1986BU]

SPEAKERS – Geoff Wilmington, Senior Technical Product Manager, VMware

Thursday, August 29, 09:00 AM – 10:00 AM

• NSX-T and Horizon: Better Security and Performance for VDI [ADV1313BU]

SPEAKERS – Graeme Gordon, Senior Staff EUC Architect, VMware and Geoff Wilmington, Senior Technical Product Manager, VMware

Tuesday, August 27, 05:00 PM – 06:00 PM

• Healthy Expansion: CHRISTUS Health’s Virtualization Journey [SAI3510BUS]

SPEAKERS – Brandon Rivera, Enterprise Infrastructure Architect, CHRISTUS Health and Geoff Wilmington, Senior Technical Product Manager, VMware

Monday, August 26, 02:30 PM – 03:30 PM

If you’re looking for other VMware security product related deep dive sessions, take a look at these from some of my peers that I highly recommend you attend.  These folks are all amazing presenters and their content is top-notch.

• NSX-T Advanced Security and Networking Service Insertion Deep Dive [SAI2781BU]

SPEAKERS – Stijn Vanveerdeghem, Senior Technical Product Manager, VMware

Wednesday, August 28, 03:30 PM – 04:30 PM

• Apply Consistent Security Across VMs, Containers, and Bare Metal [SAI1017BU]

SPEAKERS – Ganapathi Bhat, Sr Technical Product Manager, VMware

Wednesday, August 28, 09:30 AM – 10:30 AM

• NSX-T Firewall Design and Deployment Best Practices [SAI1529BU]

SPEAKERS – Anthony Burke, Solutions Architect, VMware and Dale Coghlan, Staff Solution Architect, VMware

Wednesday, August 28, 08:00 AM – 09:00 AM

• AppDefense Reference Architecture Deep Dive [SAI3196BU]

SPEAKERS – Kevin Berger, Director, Security Engineering, VMware and Chris Corde, Senior Director of Product Management, VMware

Wednesday, August 28, 01:00 PM – 02:00 PM

Last but not least, you’re definitely going to want to check out this session.  I won’t go into too many details, but Ray has some seriously cool stuff to show off.

• Using VMware NSX Policy and Analytics to Address Key Challenges in Security [SAI3071BU]

SPEAKERS – Ray Budavari, Senior Staff Technical Product Manager, VMware

Wednesday, August 28, 02:00 PM – 03:00 PM

If we’ve never met, please don’t hesitate to come up and say ‘Hi’.

 

Advertisements

The VMware NSX Platform – Healthcare Series – Part 7.1: Secure End-User VDI Practical

In the last post we broke down each of the six use cases that the VMware NSX platform can provide for Secure End-User computing environments.  Each of these use cases providing a Healthcare organization a different business value depending on their needs.

As we start to break each use case down, the first one we’ll be looking at is around Micro-segmentation of a Virtual Desktop Infrastructure (VDI) environment that a Healthcare organization may run.

Speaking with my Healthcare customers running VDI systems, there wasn’t many security-based controls to protect traffic between each desktop.  These systems were fully allowed to communicate with each other regardless of an actual need to.  Not only that, but those VDI systems could scale up or down as demanded.  The security posture must be able to scale up and down in a similar fashion.  These use cases for NSX are quite simple and easy to implement.  By blocking unnecessary communications between these systems, we can ensure that if one desktop is compromised, could transition and compromise other desktops.  Several times in discussions, my Healthcare customers were using one or more technologies as they transitioned from one to the other or as part of a legacy acquisition.  We’ll take the top two VDI software providers, Citrix and Horizon and show that it doesn’t matter which one a customer is using, both can be secured in an identical fashion.

Healthcare organizations also had different types of uses for example, clinicians and HR users.  There could be a need to create separate pools for each of these different types of users.  There are other ways to address this problem with other products and utilizing a single pool but in this case, we’re focusing on customers with this model.  In a later blog we’ll also explore how Identity-based Firewall in NSX can help as well.

Let’s look at our use case.

Use case – Augment an existing VDI deployment using VMware Horizon with NSX and secure traffic between each system and providing isolation for desktop pools

• Block all VDI to VDI
• Isolate HR and Clinician Desktop pool from each other
• Maintain the existing infrastructure as much as possible and without major changes 

Technology used –

Windows systems:

• External Client-01a – Windows 10 – HR User
• External Client-02a – Windows 10 – Clinician User
• EMR-VDI-01a > EMR-VDI-02a (HZN)
• HR-VDI-01a > HR-VDI-02a (CTX)
• Allow EMR-VDI pool to expand and contract based on Clinician need

VMware Products

• vSphere
• vCenter
• NSX
• Horizon View

Citrix Products

• XenDesktop 

Applications in question –

Open Source Healthcare Application:

• OpenMRS – Open Source EMR system
  • Apache/PHP Web Server
  • MySQL Database Server

Open Source HR Application:

• IceHRM – Open Source HR system
  • Apache/PHP Web Server
  • MySQL Database Server

Environment –

In this environment, the customer has a Citrix VDI environment and a Horizon VDI environment.  They are in the middle of a transition from one platform to another, however security needs to remain consistent.  The HR VDI systems need to be able to access the HR application, and the Clinicians need to be able to access the EMR application.  Neither should be able to talk to the other application and neither pool of desktops should be able to talk to each other.

We also need to provide a security posture for when the EMR Desktop pool needs to expand due to Clinician need.

We’ll start by building out our NSX grouping constructs and putting in each of the different objects into their respective containers.  We’ll also mock up our firewall rule sets

NSX Groupings:

The groupings are shown here for use in our rule sets.  Each of these NSX Security Groups will enable the DFW rule sets to scale as the environment scales without the need to make modifications to the DFW rules.  If more VDI systems are created, they will simply land in their appropriate Security Group, and the DFW policy will be applied.  We’ll be exploring the concept of automation for VDI in another post.

HR Application Communications:

EMR Application Communications:

VDI Communications:

The rule sets are simple to segment out each of the different VDI systems from their appropriate application servers.  Also, by nesting all VDI VMs into one Security Group, we can keep the block communications between the systems to one rule.  With all the necessary components to create our security policy, we can start putting it into the NSX DFW.

All the rules are in place; we can now perform verification to see that our rules are working and the requirements have been achieved.

HR users can get to their HR application and EMR users cannot get to the HR application

EMR users can get to their EMR application and HR users cannot get to the EMR application

VDI desktops are not allowed to talk to each other

Now that we’ve verified that the HR user and the Clinician can access their VDI desktops and the relevant applications they need, we’ll scale up the EMR VDI systems by adding one more desktop and attempt to connect to the newest desktop the same as we did in the previous verification.

Currently, there are only two VDI desktop machines in our pool.  We’ll increase that to 3 as a customer may need a new desktop should more clinicians need to login during rush periods, where not enough desktops are online already.  The main consideration here, is that we want to keep the same security posture as above, regardless of how many desktops we have in the pool.

With the pool increased, our dynamic criteria has added the newly built machine into the Security Group as it should.  All the rules associated with this Security Group will now apply to this newly built VDI desktop as well.  Let’s verify.

Our pings to the other VDI systems, including the new one we just built drop as we’d expect them to drop.  This fulfills the requirements of the customer.

Micro-segmentation for Secure End-User starts with securing VDI desktops.  Regardless of the implementation technologies, Horizon or Citrix, NSX can provide a security posture surrounding the VDI desktops and the application systems they require access to.  In future posts I’ll be showing how NSX can provide ever further value for the other Secure End-User use cases.

The VMware NSX Platform – Healthcare Series – Part 7: Secure End User Concept

As more and more organizations look to bring the edge computing closer to the data center, this can include the desktop systems that end users utilize to access the organizations systems.  Bringing these systems can bring in vulnerabilities and exposures that would typically be constrained by the physical desktop device.  When these systems are virtualized, a new security posture is required to protect the critical data and assets in the data center.

When visiting most Healthcare organizations during my pre-sales days, I found that many of them were using some form of Virtual Desktop Infrastructure (VDI) or Remote Desktop Session Host (RDSH) technology to present the applications to their clinicians.  Regardless of the overarching technology providing these services, Horizon or even Citrix, the VMware NSX platform has several business values for securing Healthcare customers running these platforms.

Revisiting the nine NSX use cases we previously identified:

The use case of Secure End User with NSX can be further broken down into six unique use cases.  A Healthcare organization does not have to use all six of them, but can start with one or more as their needs dictate.  Let’s break down each use case and explain when each can meet a business need.

In the majority of Secure End user use cases, we can apply the concept of micro-segmentation to provide granular security for a Healthcare organization by protecting East-West traffic between VDI desktops, or implementing Identity-based Firewall, separation of desktop pools, even providing 3^rd party integrations like Agent-less Anti-Virus or Anti-Malware into the NSX platform.  Protecting the desktop or RDS hosts is straightforward, but we can even apply the same security concepts to protect the infrastructure that manages the VDI or RDSH environments.

Micro-segmentation

From the virtual desktop standpoint, micro-segmentation provides a means in which we can control East-West traffic between the desktop systems.  It also means, that if an organization has the need to have separate pools of desktops or even RDSH systems, NSX can provide security within each pool and between each pool, separately.  In Healthcare environments, there may be a need for external coders to provide services for the organization.  A new desktop pool, specifically for those external coders could be created and secured with NSX to only allow access to necessary systems.

Edge Services

NSX is a platform product.  This means that it has capabilities that span more than just security.  NSX can also provide NAT and Load Balancing services for the Edge management components of a VDI and RDSH infrastructure.  This added benefit helps customers reduce the complexity of having multiple interfaces in which to manage their infrastructure servers.  Healthcare systems require high availability and maximum uptime for their patient-facing systems.  The NSX Edge can be put into high availability and provide Load Balancing services to meet this use case, without the additional costs of 3^rd party products.  These features come in even the standard version of NSX.

Network Virtualization

The ability to create logical networking constructs dynamically, is a principal use case with NSX.  NSX can faithfully recreate production networks, even with the same IP addressing, and isolate each network from talking with each other.  For Healthcare organizations where application uptime means patient care, the ability to quickly spin up these network reproductions can mean that copies of production applications can be placed into the isolated copy network and things like upgrades and security changes can be tested, prior, to deployment into the production workloads.

Protecting VDI Infrastructure

There’s no doubt that the virtual desktops and RDSH servers are key to a VDI deployment, but the back-end management components that provide the means to ‘spin up’ these desktops and servers, can also be protected by NSX.  These systems provide desktop interfaces for clinicians and hospital staff.  If these staff are unable to access the applications and systems they need to perform their jobs, it could directly affect patient care for the organization.  The back-end systems which facilitate these desktops are just as critical as the desktops themselves.

Protecting Desktop Pools

NSX provides 3^rd party partners with the ability to plug into the NSX framework using NetX or EPSec APIs.  These APIs provide the partners the ability to integrate products such as:  Next-Gen Firewalls, Intrusion Detection and Prevention solutions, as well as Anti-Virus and Anti-Malware products.  By integrating with NSX, these products can remove the need for traditional in-guest agent approaches for these products.  Doing so can greatly impact the overall performance and resource requirements of each of the ESXi hosts these services reside on.

User-based Access Control

Regardless of whether a Healthcare organization uses one or all the use cases in their environment, each use case provides a unique value and layered approach to securing virtualized desktops or remote session hosts.  With the proximity these systems now have to the internal data center systems, their protection is very important to ensure a compromise or attack on one of them, doesn’t allow further access in the data center and to vital patient information.

Over the next several blog posts, we’ll dive deep into each of these concepts and show how to practically apply these use cases to common scenarios that a Healthcare organization may run into.

The VMware NSX Platform – Healthcare Series – Part 6: DMZ Anywhere Practical

Continuing our discussion on the topic of Healthcare and the DMZ use case, we’re going to put these concepts into actual practice.  With Healthcare systems, patients want access to their information quickly and not necessarily within the four walls of a Healthcare organization.  This means that this information needs to be provided to Internet-facing devices for secure access.  Below is the layout we’re going to use as a typical layout with Internet-facing EMR Patient Portal for customers using traditional methods.

Traditional Model

For this post, we’re going to use a physical Perimeter Firewall, and an NSX Edge Services Gateway (ESG) as the Internal Firewall to separate the DMZ systems from the Internal data center systems.

In our concept post, we talked about how NSX can help augment an existing DMZ approach to simplify the restrictions of communications between systems that reside there.  For Healthcare providers, the EMR Internet-facing Web Servers should not allow communications between themselves.  If one Web Server is compromised, lateral movement must be restricted.  Traditional approaches to restrict intra-server traffic between the EMR Web Servers would require blocking the communication at Layer 2, using MAC addresses.  With NSX, we can instantiate a firewall at the virtual machine layer, regardless of the servers being on the same Layer 2 network, and restrict the Web Servers from talking to each other without needing to know the MAC addresses or by sending the intra-server traffic through an external firewall to block.  This same concept of East-West Micro-Segmentation, is covered in previous posts and is the same concept we can apply for DMZ workloads.

Let’s lay out the requirements from the customer for the first use case.

VMware NSX – DMZ Augment Model

Use case – Augment the existing DMZ to remove communications between DMZ systems.

• Block all EMR Web Servers from talking to each other
• Maintain the existing infrastructure as much as possible and without major changes

Technology used –

Windows Clients:

• Windows 10 – Management Desktop – Jumpbox-01a (192.168.0.99)

VMware Products

• vSphere
• vCenter
• NSX
• Log Insight

Application in question –

Open Source Healthcare Application:

• OpenMRS – Open Source EMR system
  • Apache/PHP Web Server
  • MySQL Database Server

Let’s start things off like we normally do, with the layout of our methodology for writing our rule sets.

 

When we put in NSX, we can write one rule and get the following result.

The rule is very simple to write. We simply add any DMZ systems to a Security Group and add that Security Group as both the Source and Destination and apply a Block.

Once this rule is in place, any virtual machines we place into the DMZ-SG-ALL Security Group, will be blocked from talking to each other.  Let’s verify this is working.

As we can see, the Web Servers are no longer allowed to talk to each other.  We have produced a similar result with less complexity and more scalability and operational ease without changing the existing infrastructure at all.

For the next use case, collapsing the traditional hardware DMZ back into the data center, the goal is to remove the need for the NSX Edge Services Gateway to provide an Internal Firewall and use the NSX Distributed Firewall (DFW) to handle access between the DMZ and the internal data center systems.

VMware NSX – DMZ Anywhere (Collapsed) Model

You may notice that the ESG is still in place.  That’s because the Internal data center is running on VXLAN and there still needs to be an off-ramp router to get to the physical network.  However, we have disabled the ESG’s firewall to demonstrate removing the Internal Firewall separation and allowing the DFW to handle the restrictions.

Let’s lay out the use case and the requirements from the customer.

Use case – Collapse the existing DMZ back into the data center while still maintaining the same security posture as when it was isolated.

• Restrict External Patients to connect only to the EMR DMZ Web Servers
• Restrict Internal Clinicians to connect only to the internal EMR Web Server
• Allow all EMR Web Servers to connect to the EMR DB Server
• Block all EMR Web Servers from talking to each other
• Maintain DMZ isolation of the EMR System from the HR System

Technology used –

Windows Clients:

• Windows 10 – Clinician Desktop – Client-01a (192.168.0.36)
• Windows 10 – HR Desktop – Client-02a (192.168.0.33)
• iPad – External Patients – (External IP)

VMware Products

• vSphere
• vCenter
• NSX
• Log Insight

Application in question –

Open Source Healthcare Application:

• OpenMRS – Open Source EMR system
  • Apache/PHP Web Server
  • MySQL Database Server
• IceHRM – HRHIS (Human Resource for Health Information System)
  • Apache/PHP Web Server
  • MySQL Database Server

Let’s start things off like we normally do, with the layout of our methodology for writing our rule sets.  I’m not going to go through how to get these flows.  Please reference one of my previous posts around using Log Insight, Application Rule Manager, and vRealize Network Insight to gather this information.

 

A few things of note.  We created an RFC 1918 IP Set in these groupings.  We did so, so that we can restrict only External IP addresses access to the EMR DMZ Web Servers.  We don’t want our internal Clinicians connecting to them.  By blocking the entire 1918 range set, we should never get a connection from an internal system to the DMZ systems.  To do this, we create an IP Set with all three RFC 1918 ranges in it.  We create a Security Group with this IP Set put into the Inclusion Criteria.  Then we write a rule that blocks these ranges above an ANY rule to filter the types of traffic that should hit the DMZ Web Servers.

 

Let’s put our rules in the appropriate places on the appropriate firewalls and do some testing to verify the traditional method is working properly.

NSX Edge Services Gateway Firewall Policy

This rule is in place to allow the EMR DMZ Web Servers to talk to the backend Database only.  We have to use an IP Set here because the DMZ Web Servers are outside the scope of NSX and do not have a firewall applied to them yet.  However, we can control what talks to the EMR-SG-DB Security Group from the physical environment.

Physical Firewall Policy

We’re going to forward our DMZ Web Servers through our Physical Firewall to accept traffic on TCP 8080.  With this change we should be able to access our OpenMRS EMR system from the Internet.  Let’s verify.

As you can see from the address bar, we’re able to hit one of the DMZ Web Servers from the Internet.  I’m using an iPad to demonstrate that it doesn’t matter the device at this point.  We can also verify that our NSX ESG Firewall is being hit by the DMZ Web Servers as well.  Using Log Insight, we can verify this quickly.

We can see that the DMZ Servers are hitting our rule and that the destination being hit is 172.16.20.11, which is the EMR-DB-01a server.

Let’s put our rules for inside the data center into the NSX DFW.

This type of configuration represents how we’d have to build our rule sets to accommodate a segregated DMZ environment.  Let’s verify that our EMR DMZ and Internal EMR Web Servers can still hit the EMR DB and that our Clinician Desktop and HR Desktops cannot browse to their respective systems.

Clinician Desktop to Internal EMR

HR Desktop to HRHIS

We’ve confirmed that all the rules in place are working and the traditional approach still works.  Let’s collapse those two Web Servers back into the data center and show how we can still provide a similar security posture, without the physical hardware isolation.

To do this we’re going to need to move back into our data center the two EMR DMZ Web Servers.  I’m going to create a new VXLAN network for them to live on that mimics their physical VLAN configuration inside the data center so we can still keep network isolation.  Keeping the same network doesn’t technically matter since we can still control the traffic, but most production Healthcare organizations would want to refrain from having to change IP addresses of their production systems if they can help it.

As you can see, the EMR-DMZ-WEB-01a/02a machines are now inside the Compute cluster in my data center.  They’re also on their same layer 2 network as they were before in hardware isolation.

We’ve disabled the Firewall on the ESG as well.

And here is our now modified DFW rule sets to accommodate a collapsed DMZ environment similar to the hardware isolated configuration.

So, here’s what did we added/changed:

• We added our RFC1918 Security Group so that any internal systems would not connect to the DMZ Web Servers.
• We also created a PERIMETER-IPSET for the Physical Firewall. This is because the ports for the EMR DMZ Web Servers are being NAT’d through the Perimeter Firewall so communications to the EMR DMZ Web Servers appear to come from an interface on that device.  Since that interface is on RFC1918 network, we add it to the RFC1918 Security Group as an Excluded host address.
• Added DMZ Security Tags so that any new systems that are built can have the DMZ-ST-ALL Security Tag applied, which will put them into the DMZ-SG-ALL Security Group and block intra-server communications immediately.

Now that all of our changes in architecture are in place, we can go through and verify that all the requirements are being accounted for.  Let’s revisit the requirements.

Use case – Collapse the existing DMZ back into the data center while still maintaining the same security posture as when it was isolated.

• Restrict External Patients to connect only to the EMR DMZ Web Servers

We can see that our External device from an IP of 172.221.12.80 is connecting to our EMR-DMZ-WEB-01a server.  We can also see that the Web Server is also talking to the backend EMR-DB-01a server.

• Restrict Internal Clinicians to connect only to the internal EMR Web Server

Here we can see that our Internal Clinician Desktop has the ability to connect to the Internal EMR Web Server but when they attempt to connect to one of the DMZ Servers, they’re blocked.

• Allow all EMR Web Servers to connect to the EMR DB Server

This requirement appears to be functioning as expected as well.

• Block all EMR Web Servers from talking to each other

A quick cURL to the Web Servers shows that Internal and External are not communicating with each other.  Also, from EMR-DMZ-WEB-02a to EMR-DMZ-WEB-01a we’re not getting a connection either.

• Maintain DMZ isolation of the EMR System from the HR System

Another attempt to cURL to the HRHIS System shows that the EMR-DMZ-WEB-01a server is not able to communicate to the HRHIS System.  This completes the requirements set forth by the customer.  The patient information access is now limited to only from the EMR system and compromise of any adjacent system within the Healthcare organization, will not allow communications between those systems and the EMR.  We have effectively reduced the attack surface and added defense-in-depth security with minimal efforts.

As we look back, there are several ways to architect a DMZ environment.  Traditional hardware isolation methods can still be augmented to remove massive infrastructure changes to an existing DMZ.  Customers looking to remove the hardware isolation altogether, can do so by collapsing the DMZ environment back into the data center and still maintain the same level of control over the communications both in and out of DMZ systems.  With NSX, the DFW and its ability to control security from an East-West perspective can be overlaid on top of any existing architecture.  This software-based approach helps provide security around a Healthcare organization’s most critical externally-facing patient systems and help reduce exposure from adjacent threats in the data center.

 

 

 

 

The VMware NSX Platform – Healthcare Series – Part 5: DMZ Anywhere Concept

Healthcare organizations are being asked to expose Internet-based services and applications to their patients more than ever.  With Healthcare, exposure of PHI and PII is of the utmost concern.  With the perimeter of the Healthcare organization needing to be as secure as possible, exposing external systems and applications to the Internet falls under this scope as well.  Traditional DMZ approaches are hardware-centric, costly, and operationally difficult to use in most modern datacenters.  With VMware NSX, we can take the concept of the DMZ, and augment a current DMZ approach, or even collapse the DMZ back inside the data center while still providing a robust security posture necessary for Internet-facing applications.

Let’s revisit the nine NSX use cases we identified previously.

DMZ Anywhere is a use case that our customers are looking at that augments traditional hardware-based approaches and leverages the Distributed Firewall capabilities to segment how traffic is allowed to flow between systems anywhere in the data center.  Let’s be clear, VMware NSX is not in the business of replacing a hardware perimeter firewall system.  But with NSX, you can fundamentally change how you design the DMZ environment once you’re inside the perimeter firewall to provide a much easier to manage and scalable solution overall.  You can review previous posts on how to Micro-segmentation works here.  https://vwilmo.wordpress.com/category/micro-segmentation/

Let’s take a quick look at traditional approaches to building a DMZ environment with physical devices.

Traditional hardware-based approaches can leverage either Zone-based logical firewalling or actual physically independent firewalls to separate out a specific section called the DMZ for Internet-facing applications to sit in. These zones are built to only allow specific sets of communication flows from the Internet-facing systems to their backend components. The systems are typically on their own separate networks.  Typical applications exposed to the Internet are web-based applications for major systems.  These types of systems can comprise of several Web servers, all of which can be used to provide multi-user access to the application.

If customers want to keep the same traditional approaches using zone-based Firewalling, NSX can help block movement for the virtual systems that reside within the DMZ from East-West movement.  In most cases, the systems that sit in the DMZ are Web-based systems.  These types of systems typically do not require communications between the Web servers, or even between disparate applications.

In the above examples, all the DMZ Servers can instantiate a conversation bi-directionally with each other.  This is inherently insecure and the only way to secure these is to send all the East-West traffic through the firewall.  When you add more systems, you add more rules. This problem continues to compound itself the larger the DMZ gets.  What if you have multiple networks and systems in the DMZ?  That will require significantly more rules and more complexity.  If you need to scale out this environment, it becomes even more operationally difficult.  How can NSX plug into this scenario and help reduce this complexity and also provide a similar level of security?

With NSX, we can provide the East-West firewalling capabilities in both scenarios to secure the applications from each other from compromise.  If one system is breached, the attack surface for movement laterally, is removed as the systems don’t even know the other systems exist.

Putting in NSX, we’re now blocking the systems from talking to each other without changing any aspect of the underlying infrastructure to do so.  We’re placing an NSX firewall at the virtual machine layer and blocking traffic.  As you can see, NSX can be made to fit nearly any DMZ infrastructure architecture.

Here we have our Electronic Medical Records application that has an Internet-facing Patient Access Portal.  With a traditional approach, the Patient Portal may be on separate hardware, situated between two sets of hardware Firewalls, or one set of Internally Zoned, Firewalls, and on a completely different logical network.  The backend systems that are required for the DMZ EMR systems are situated behind another internal firewall along with the rest of the systems in the data center, in this case, share infrastructure systems and the EMR backend database system.  Neither of these systems should have contact with the Internal HR Web or DB Server.  If they did, compromise from the DMZ environment could allow an attacker access to other sensitive internal systems like the HR system.

Now let’s look how NSX can change the traditional design of a DMZ and collapse it back into the data center but will allow the same levels of security as traditional methods, but with a software-based focus.

Using NSX in this approach, we’re doing the same thing we did when we augmented the existing hardware approach by placing a software-based Firewall on each Virtual Machine in the data center.  This fundamentally means, that every VM, has its own perimeter and we can programmatically control how each of those VM’s talk or don’t talk to each other.  This approach could enable a Healthcare organization to pull back the hardware isolation for their DMZ back into their data center compute clusters and apply DMZ-level security to those specific workloads hereby collapsing the isolation into software constructs versus hardware ones.  In the collapsed DMZ model, we have no separate infrastructure to support a DMZ environment, we simply control the inbound traffic from the perimeter through the physical firewall as we would normally do, but apply VM-level security using NSX between the systems that would’ve been separated out.  The DMZ EMR Web Servers are still restricted access to the HR system even though they technically live next to each other within the Internal data center.

Let’s contrast a software-based approach versus traditional hardware methods.

Hardware-based

• For Zone-based firewalling leveraging a single hardware appliance, this is much less of an issue. Some organizations purchase at multiple Firewalls at the perimeter for HA configurations.  If they leverage a separation of their DMZ using two sets of Firewalls, that means they’ll need to purchase at least 4 Firewalls to perform this configuration.
  • New features and functions with hardware products can be tied to the hardware itself. Want these new items?  That could require a new hardware purchase.
• Scale
  • Hardware-based scaling is generally scale-up. If the Firewall runs out of resources or begins to be over-utilized, it could require moving to larger Firewalls overall to accommodate. This means a rip and replace of the existing equipment.
• Static
  • A hardware-based DMZ is very static. It doesn’t move within the data center and the workloads have to be positioned in accordance to the network functions it provides.  In modern data centers, workloads can exist anywhere and on any host in the data center.  They can even exist between data centers.  Uptime is critical for Healthcare providers as is maintaining data security.  Wherever the workload may end up, it requires the same, consistent security policy.
• Cost
  • Buying multiple hardware Firewalls is not cheap. If the organization needs to scale up, ripping and replacing the existing Firewalls for new ones can be costly and incur downtime.  For Healthcare organizations, downtime affects patient care.  Some DMZ architectures have separate hardware to run only the workloads in the DMZ environment.  This separates out the Management of that environment from the internal data center environment.  It also means that, when architecting a hardware-based DMZ, you may end up with compute resources that costly and underutilized.  A concept that totally goes against virtualization in general and leads to higher operating costs in the data center and wasted resources.
• Operationally difficult
  • If the customer is going with the multiple Firewall method, this means that to configure the allowed and disallowed traffic, the customer would need to go into two sets of Firewalls to do this. Hardware Firewalls for the DMZ will require MAC addresses for all the workloads going into them.  DMZ networks may be a few networks, but usually Web Servers exist on the same logical network.  Healthcare systems can have massive Internet-facing infrastructures to provide for their patients.

Software-based

• By placing the Firewall capabilities within the ESXi kernel, we’re able to ensure security simply by virtue of the workload residing on any host that is running the vSphere hypervisor. When it comes to new features and functions, where you might need to upgrade proprietary Firewall hardware, NSX is tied to any x86 hardware and new features simply require an update to the software packages reducing the possibility of ripping and replacing hardware.  For Healthcare customers, this reduces or eliminates the downtime required to keep systems up-to-date where downtime is a premium.
• Scale
  • The nature of NSX being in every hypervisor means Firewall scales linearly as you add more hypervisors to a customer environment. It also means, that instead of having to purchase large physical Firewalls for all your workloads, the DFW will provide throughput and functionality for whatever your consolidation ratio is on your vSphere hosts.  Instead of a few physical devices supporting security for 100s-1000s of virtual machines, each host with the vSphere hypervisor supports security for the VMs residing on it.  With a distributed model that scales as you add hosts, this creates a massive scale platform for security needs.  Physical Firewalls with high bandwidth ports are very expensive, and generally don’t have nearly as many ports as you can have in a distributed model across multiple x86 hardware platforms.
• Mobility
  • Hardware-based appliances are generally static. They don’t move in your data center although the workloads you’re trying to protect may.  These workloads, when virtualized, can moved to any number of hosts within the data center and even between data centers.  With NSX, the Firewall policy follows the virtual workload no matter the location.  Healthcare providers care about uptime, the ability to move sensitive data systems around to maintain uptime, while maintaining security, is crucial.
• Cost-effective
  • Software-based solution only need to be licensed for the hosts that the workloads will reside on. No need to purchase licensing for hosts where protected workloads may never traverse to.  With Healthcare organizations, they can focus on the workloads that house their patient’s sensitive data and the systems that interact with them.
  • No need to spend money on separate hardware just for a DMZ. Collapse the DMZ workloads back to the compute environments and reduce wasted resources.
• Operationally easier
  • By removing another configuration point within the security model, NSX can still provide the same level of security around DMZ workloads even if they sat on the same host as a non-DMZ workload. All of this, while keeping them logically isolated versus physically isolated.  With NSX, there’s no reason to use multiple networks to segment DMZ traffic and the workloads on those segments.  NSX resolves the IP and MAC addresses so that rule and policy creation is much simpler and can be applied programmatically versus traditional manual methods.

When it comes to DMZ architecture, traditional hardware approaches that have been followed in the past, can be too static and inflexible for modern workloads.  Healthcare customers need uptime and scale as medical systems that house patient data are not getting smaller and patient requirements for access to their information continues to grow.  With NSX, we can augment a current DMZ strategy, or even collapse their physical DMZ back into their virtual compute environment and still provide the same levels of security and protection as hardware-based approaches, at a lower cost and easier to maintain.

The VMware NSX Platform – Healthcare Series – Part 4.3: Micro-segmentation Practical – vRealize Network Insight

In the last post, we showed the methodology to building out Micro-segmentation rules for the OpenMRS EMR application in the test environment.  This was a straightforward process using the VMware NSX Distributed Firewall and vRealize Log Insight. The process became even simpler when we leveraged the new NSX 6.3 feature – Application Rule Manager. More importantly it gives us a starting point for applications to provide a Zero-Trust security posture.

As we continue the series, we’re going to expand the Healthcare organization’s environment to include other systems that are typically running.  As Healthcare professionals, we know that while the EMR is a critical application, it’s not solely an independent system.   There are many systems within the Healthcare organization that have connections to and pull information from the EMR system, and there are other systems inside that environment as well beyond clinical-facing.  All of these systems require the same amount of security as the EMR system does.  In this post, we’re going to leverage the VMware NSX Platform, our foundational methodology for micro-segmenting we built out in the first post, and vRealize Network Insight to help build our NSX DFW rules for several new systems we’re going to be adding in.  This will help complete the environment build out and conclude the Micro-segmentation NSX use case for the series.

Let’s start with a layout of the environment and the systems we added in to show just how complex this type of environment could be.

• We have our EMR system as we had in the previous posts.
• We’re now going to add a DCM4CHEE PACS system that our OpenMRS EMR can forward events to. Our PACS system has 2 modalities, a CT and MRI scanner that talk to the PACS system, simulated using the DTKv modality emulator.  These systems simulate physical devices out in a clinical setting.
• We’re introducing an HL7 Interface Engine, Mirth Connect, that is pulling data from our EMR and sending that information via SFTP over to our Data Warehouse DB server for processing into a population health application. When the SFTP job is complete, the HL7 system will be sending an email notification of completion to our hMail, email server.
• Finally, we’re going to connect the systems to the organizations Active Directory, DNS, and NTP servers for the shared infrastructure based services these systems require.

 The Healthcare organization has asked that we expand the Zero Trust security model using NSX to their entire environment.  They have given us some details about the applications and requirements on what we need to accomplish.  Given the size of the environment and number of applications now in-scope, the customer has asked for a scalable way to operationalize Micro-segmentation.  The customer has added several new integrations to the EMR and between other application systems in the infrastructure.  We need to find out what the applications are, verify with Application Teams, block/allow as necessary per Application Team.

 Expanded Use case – Provide a Zero Trust security model using Micro-segmentation around a Healthcare organization’s data center applications.  Facilitate only the necessary communications both to the applications and between the components of the applications.

• Allow EMR Client Application to communicate with EMR Web/App Server
• Allow EMR Web/App Server to communicate with EMR Database Server
• Allow PACS Client Application to communicate with PACS Web/App Server
• Allow PACS Web/App Server to communicate with PACS Database Server
• Allow PACS Web/App Server to communicate with EMR Web/App Server
• Allow PACS Modalities (CT Scan/MRI) to communicate with the PACS Web/App Server
• Allow HL7 Server to communicate with whatever systems it requires
  • This system is the ‘bridge’ between ancillary applications and the EMR system
• Allow HRHIS Client Application to communicate with HRHIS Web/App Server
• Allow HRHIS Web/App Server to communicate with HRHIS Database Server
• Allow EMAIL messages to be sent as necessary. Certain applications are emailing their status updates.
• Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.
• Block any unknown communications except the actual application traffic and restrict access to the HRHIS application to only a HRHIS Desktop system running the HRHIS Client Application.
• Allow bi-directional communication between the Infrastructure Services and all applications that require access to those services

Problem – The Healthcare organization still does not have a full understanding of how their applications communicate within and outside the organization. We have some details listed above, but nothing about the flows or ports and protocols to restrict traffic to only necessary communications.  With the expanded use case, the Healthcare organization thinks it will be difficult to operationalize this security model.

Technology used –   

Windows Clients:

• Windows 10 – Clinician Desktop – Client-01a (192.168.0.36)
• Windows 10 – HR Desktop – Client-02a (192.168.0.33)
• Windows 10 – Jumpbox-01a – (192.168.0.99)

VMware Products:

• vSphere
• vCenter
• NSX
• vRealize Network Insight

Applications in question –

Open Source Healthcare Applications:

• OpenMRS – Open Source EMR system
  • Apache/PHP Web Server
  • MySQL Database Server
• DCM4CHEE – PACS (Picture and Archiving Communication System)
  • Apache/PHP Web Server
  • MySQL Database Server
• IceHRM – HRHIS (Human Resource for Health Information System)
  • Apache/PHP Web Server
  • MySQL Database Server
• Weasis – DICOM (Digital Imaging and Communications in Medicine) system
  • Runs on a Windows-based system
• DTKv – DICOM Emulator
  • Runs on a Windows-based system
  • Used to simulate a modality for the PACS system
    • Modality – MRI, CAT Scan, etc.
  • Mirth Connect – HL7 Interface Engine
    • Apache/Java/MySQL Server
  • Data Warehouse DB server

Infrastructure Applications:

• LDAP
• DNS
• NTP

Enterprise Applications

• hMail Email Server

After sitting down with each Application owner and asking questions about their application, we were able to at least identify the virtual machines associated with each application.  Using vRealize Network Insight (vRNI from here on), we’re going to map out the flows of all the applications that are in-scope here. Now that we have all the Applications defined, we’ll go ahead and build all the constructs within NSX for use with our rules.

If you’ve heard me speak before, you’ll know that I harp on creating naming schemes that are intuitive.  What do I mean by that?  Intuitive, in that just looking at the name will tell you the meaning of whatever you’re looking at.  In any setting, making changes to rules, services, ports, groups, etc. can have a profound impact on the infrastructure.  In the Healthcare setting, this could mean patient lives.  It’s important that we adopt a naming scheme, and adhere to it, even if the services already exist within NSX.  I’ve settled on these examples for this blog post:

INFRA Analysis and Rule Building:

 Similar to the last post with ARM, we’re going to start with Infrastructure services.  They are services that all of these systems depend upon and represent rules that will encompass the entire environment.  When we open up vRNI we’re going to look at everything within the vCenter for Site 1.  You can go more granular as you wish but with my environment being nested I have to look at bit higher.  I’m also only going to look at flows over the last 7 days.  In a production deployment, you’d want to examine each application and understand it.  If it’s a payroll processing application that does month-end processing you may want to look at the last 30 days.  When we change the micro-segments to ‘by Application’ we can see all of our custom applications we created and their flows both intra and inter the virtual environment.

Servers:

• Active Directory – AD-DNS-01a
• DNS – AD-DNS-01a
• NTP – NTP-01a

These are the 3 servers in scope for the infrastructure.  We’ll need to create our structure for writing our rules.  We’re going to combine all of these servers into one Infrastructure Grouping as necessary.  To start building our rules we’ll need to ‘Plan Security’ using vRNI for the VM’s.

We’ll start by examining the flows of the AD-DNS-01a VM to other VMs.  When we log into vRNI the top option on the left side menu is ‘Plan Security’.  When we select that, we get a pop-up box that allows us to choose several Entities to plan security around as well as to examine the flows over a duration of time.  vRNI will allow you to examine flows from the last 30 days.  Again, understand the application and the necessary time that would be needed to realize the flows.

When we select Analyze, we’ll be given a report of information about the VM in question.

We’ll need to change a few settings to bring into view, the information we’re looking for.  In this case, we’re going to change the ‘Group By’ to ‘by VM’.  We’ll also need to change the ‘Also show groups for’ to ‘Other Virtual’.  This will show us all VM’s that are in-scope for this planning.

From this view, we can see all of the ‘Other Virtual’ machines that the AD-DNS-01a server talks to.  If we want to dig into the flows we’ll need to look at to build our rules, we’ll focus on the AD-DNS-01a wedge.  Double-clicking on that wedge brings us to this screen.

This screen shows up all of the flows captured over the last 7 days both to and from AD-DNS-01a as well as the port and protocol information.  We can use filters to break down the flows into smaller groups and dig into them specifically.  In this case, we want to take a look at what vRNI is recommending we do to create NSX Distributed Firewall (DFW) rules.  To see that, we’ll select the ‘Recommended Firewalls Rules’ tab.

vRNI has shown us that it’s recommending we create 17 rules for our flows.  Let it be said, that Infrastructure Services will share the bulk of the flows in this lab.  Nearly every VM needs access to some or all of the services.  It will show very granular rules for us to build.  The thing to understand, is that these are ‘recommendations’, not absolutes.  Taking a look at the recommended rules, it looks like we might be able to combine a few and make things a bit simpler.  To do that, we’re going to take these rules and extract them from vRNI.

In the upper right corner of vRNI, there’s an option to ‘Export as CSV’.  This will take all of those recommendations and put them in a format we can modify to our liking.

vRNI has also given us some recommended Security Group structures as well.  You can choose to use this structure or not.  In my case, I’m going with the structure I created above.  I’ve gone through and replaced the groups with the custom ones.

 

We see a few Destinations in this output that don’t give us an explicit location, DC-Physical and Internet.  How do we handle those? What we need to do is go back to the wheel depiction, and hover over the DC-Physical portion.  For the Internet flows, we’re going to block any communications to the Internet.

When we do this, we can see the flows just between AD-DNS-01a and DC Physical.  If we want to see the what the Destination of these flows are, we can click on the green line.  This will open up those flows.  When we get into this screen, we’re going to select the DC-Physical to AD-DNS-01a tab.

From this output, we’re given the port and protocol, the number of flows, and the amount of traffic that’s been sent over the 7-day period.  If we drill into the ‘Count of Flow’ of ‘3’, this will give us the information we need.

We now see that the 3 flows are coming from:

• 168.0.33 – Clinician Desktop
• 168.0.36 – HR Desktop
• 168.0.99 – Management Jumpbox

Now that we have this information, we can create IP Set’s to match these sources, and add them to our rules.  Since we’re not going to specifically allow any Internet flows, we don’t have to worry about creating any rules specifically for this traffic.  The Block All rules will catch anything not explicitly allowed.

Let’s look at our vRNI output, albeit modified.

So, you’ll notice a few things are different with the original output from vRNI.

• Subbed in custom Security Groups with appropriate naming
• Added a column for Combined Security Groups and Combined Service Groups. This will help us tremendously slim down the number of rules we need to write overall.  Remember, vRNI will give us very granular rules one by one.  We can consolidate those down to bigger groups and smaller numbers of rules.
• Split all Services out if multiples were showing per line and created Services in NSX.
• Color-coded things so that you can see where we can group Security Groups together to make one rule to cover each line instead of a single line for each.

What does this all mean?  Well this is what things will look like for our rules now:

Let’s move to the rules for NTP.  We’re going to change the VM to NTP-01a and run the Analyze again.

We’ll get a similar output that we got for the AD-DNS-01a VM.  When we drill into the NTP-01a wedge that will show us more details.

We can see that there are 10 flows both incoming and outgoing, and vRNI is recommending 10 rules for us to build to accommodate.  Let’s drill into the ‘Recommended Firewall Rules’ tab.

Now this set of ‘Recommended Firewall Rules’ is much simpler than the AD-DNS-01a ones.  One of these rules, the rule for DNS is already covered in the rules built above.  The rest are 9 rules from different Sources going to the same Destination.  We can reduce these rules down to one rule.

As you can see, we’ve significantly reduced the number of rules we’ll need to cover all the recommended rules just by combining Security Groups and nesting.  This will make our DFW policies much more efficient.  Let’s take this info and put it into our tables so we can visualize what all these groupings have inside.

Now that we have our structure in order, we start building our Security Groups, Security Tags, Services, and Service Groups in NSX.  I’m not going to go through the process of creating the objects within NSX.  The processes are very straightforward and my previous blogs discuss this very process.  Here are the results.

Security Tags:

IP Sets:

Security Groups:

Services:

Service Groups:

Let’s build the rules that we had planned above now.  Again, I’ve already shown how to build rules in previous posts, so I’m not going to go through that here either. Here’s what we end up with in the DFW.

DFW Rules:

All in all, we were able to take 27 recommendations and bring that all down to 4 Allow rules, and 2 Block rules that cover all of our Infrastructure Services.  Let’s move onto the next application and requirements, our EMR.

EMR Analysis and Rule Building:

 Requirements to meet:

• Allow EMR Client Application to communicate with EMR Web/App Server
• Allow EMR Web/App Server to communicate with EMR Database Server
• Allow PACS Web/App Server to communicate with EMR Web/App Server
• Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.

In the previous post we went through and put in our rules for the EMR application.  I have since removed those rules so we can leverage vRNI to show us what to put in.  Since we’ve added several new systems that may communicate with the EMR, we’ll be making some changes anyway.

Similar to what I did in the Infrastructure Services section, I’m going to select the EMR-WEB-01a VM and then show the micro-segments ‘by VM’.

We’re going to dig into the VM and look at the recommended rules just like we did for the Infrastructure VMs.

What we will notice is that there are a few different rules for different connections. We see new connections from the PACS-WEB-01a VM to our EMR-WEB-01a server.  This would imply that these two systems shared information over TCP 9696 and TCP 2575.  We also see TCP 8080 and TCP 22 connections from DC-Physical which means that external systems are accessing the EMR-WEB-01a VM.  Lastly, we see that the EMR-WEB-01a system talks to its EMR-DB-01a system across TCP 3306 like we’d discovered before.  Since these devices are from outside the NSX environment, we’ll need to look at the Flows section to see what these IP addresses are.

These seem to correlate with at least one known IP address.  The 192.168.0.99 address was my Jumpbox system from the last blog post, but this 192.168.0.36 and 192.168.0.33 systems are unfamiliar.  Talking with the Application Team we learned:

• The 192.168.0.18 system is an outside system that is accessing the EMR-WEB-01a via SSH on TCP 22, and also accessing the EMR itself, over TCP 8080.
• The 192.168.0.36 system is a Clinician Desktop
• The 192.168.0.99 Jumpbox should have access for management purposes to the EMR.
• The TCP 9696 is an integration plugin that now exists between the PACS and EMR system to route data to each. This port is the DICOM MPPS port.
• The TCP 2575 is also part of the integration plugin that now exists between the PACS and EMR systems. This port is used by the EMR system to send Radiology orders to the PACS system.

One of our requirements was to only allow Clinician Desktops access to the EMR, and block any other connections that are not identified. If we don’t explicitly allow that communication, our catch-all Blocks will stop this communication.

Extracting these rules from the interface gives us this format in CSV.

We’re going to swap in our groupings and remove any already covered rules like NTP and DNS.  We need to figure these DC-Physical flows first.  We’re going to go back to the Flows tab and filter down the results to only show the 8080 and 22 flows respective of the recommended rule.

From this page, we’re seeing:

• Two unknown IP address flows from 192.168.0.18 on 8080 and 22.
• One unknown IP address flow from 192.168.0.24 on 8080.
• One known IP address flow from 192.168.0.99 (Jumpbox) on 8080.

Looking at these, only one of these is a proper flow that we should account for.  The others have been identified as not necessary.  Now we can finish up our rules.

Legend

Yellow with dots = Already covered by another rule

Blues – Rules we’ll write

Let’s move onto EMR-DB-01a and get our recommended rules.  Same process as before.  We’re going to analyze the VM named EMR-DB-01a.

 

This brings us to those output.

This leaves us with only a couple of rules for our EMR.  Now that we know the communications that are necessary and are known good, we’ll go ahead and write our rules. Let’s lay out what those rules should look like:

Most of our groupings have already been created, so we’ll finish out whatever is left.

Security Tags:

Security Groups:

Services:

With all this info, we build our DFW rules.

DFW Rules:

 This completes the EMR section of the DFW.  We can move to the next application and it’s requirements, the PACS system.

PACS Analysis and Rule Building:

PACS applications have physical devices, modalities, that connect to them to send the images and information.  In this case, we have an MRI and CT Scanner emulator that’s playing that role in this scenario.  While they are VM’s themselves, they are outside the NSX environment so we’ll have to use IP Set’s to accommodate, similar to Clinician physical desktops.  The requirements are similar to all the other applications.

Requirements to meet:

• Allow PACS Client Application to communicate with PACS Web/App Server
• Allow PACS Web/App Server to communicate with PACS Database Server
• Allow PACS Modalities (CT Scan/MRI) to communicate with the PACS Web/App Server

We’ll start by running the analysis against PACS-WEB-01a and then on PACS-DB-01a.

Let’s clarify our DC-Physical flows:

Talking with the Application owners, the Modalities are shown to be the systems connecting via on port TCP 6060.  The Application owners have shown that this is the DICOM Echo port for the modalities.  The other system is our Jumpbox, as usual.

We’re all squared away on the PACS-WEB-01a server.  Let’s work on the PACS-DB-01a.

We’re going to start noticing a trend here.  The recommended rules are going to start being covered by previous recommendations as we continue down the list of applications.  This is good because this is simplifying our rule sets overall.

Looks like every rule that’s recommended here will be covered in a previous rule.  Let’s build the groupings and our rules.

Security Tags:

IP Sets:

Security Groups:

Services:

DFW Rules:

This completes our rule build out for the PACS system.  Let’s move to the HL7 integration engine, Mirth Connect.

HL7 Analysis and Rule Building:

 Requirements to meet:

• Allow HL7 Server to communicate with whatever systems it requires
  • This system is the ‘bridge’ between ancillary applications and the EMR system

Talking with the Application Team for the HL7 system, we learned that it’s all running on one server, HL7-01a.  With that being the case, there will be no intra-group communications necessary for this application to function.  However, the HL7 system talks with many other systems in the infrastructure.  It’s a broker of communications between disparate systems.  In our case, we’re going to take a look at what other systems this one brokers communications between.

Let’s dig into the DC-Physical side so we can understand the IP-based flows to build out our full rule sets.

We see a port 22 connection talking with the Application owners, we’ve determined that this connection is for the HL7 system to SFTP a file pulled from a MYSQL query.  Let’s clean these up a bit and classify them.

The green identified flow, we’re going to put that into the EMAIL Group as that’s accessing the EMAIL server, specifically.  Now that we’re all cleaned up, we can build out our rules and groupings.

Security Tags:

Security Group:

Services:

Service Groups:

DFW Rules:

EMAIL Analysis and Rule Building:

 Requirements to meet:

• Allow EMAIL messages to be sent as necessary. Certain applications are emailing their status updates.

As with the HL7 system, the EMAIL server is running on one server so there’s no need for any intra-group communications.  Let’s analyze the flows.

As usual, we need to dig into the DC-Physical flows to get the IP addresses of these systems.

It appears that we have the two Desktop machines connecting to the EMAIL-01a server over IMAP and we have a NETBIOS-DGM port that’s hitting the Broadcast network of the VLAN it’s on.  Let’s start making our groupings.

Let’s refine this.

Security Tags:

Security Groups:

Services:

DFW Rules:

Let’s finish up with our HR System now.

HRHIS Analysis and Rule Building:

 Requirements to meet:

• Allow HRHIS Client Application to communicate with HRHIS Web/App Server
• Allow HRHIS Web/App Server to communicate with HRHIS Database Server
• Block any unknown communications except the actual application traffic and restrict access to the HRHIS application to only a HRHIS Desktop system running the HRHIS Client Application.

Again, we’re going to follow the same process to finish our build out.

Looks like our Clinician’s desktop has been attempting access to the HR system.  There doesn’t appear to be much traffic, but that’s definitely a system we don’t want having access to the HR web site.  We’ll be blocking this traffic off from that system.

These are pretty simple rules.  Half of them are covered already in Infrastructure Services.  A quick analysis run against the DB:

A quick glance will show that there are no recommendations that aren’t already covered by another rule we’ve accounted for.  We’re all set to finalize our last application.

Let’s build our groupings and rule sets.

Security Tags

Security Groups:

Services:

DFW Rules:

With the rules all in place, we will go through and check our communications with our applications and verify they’re all still working correctly per the requirements given by the customer.

Let’s bring back the requirements we needed to fulfill and demonstrate how we’re accomplishing these as specified.

• Allow EMR Client Application to communicate with EMR Web/App Server
• Allow EMR Web/App Server to communicate with EMR Database Server

Verified – Able to log into the EMR and pull up a patient record.

• Allow PACS Client Application to communicate with PACS Web/App Server
• Allow PACS Web/App Server to communicate with PACS Database Server
• Allow PACS Web/App Server to communicate with EMR Web/App Server

Verified – Able to log into the PACS system and run a DICOM Echo from the PACS system to the EMR.

• Allow PACS Modalities (CT Scan/MRI) to communicate with the PACS Web/App Server

Verified – The PACS modality physical emulators are able to DICOM Echo to the PACS system successfully.

• Allow HL7 Server to communicate with whatever systems it requires
  • This system is the ‘bridge’ between ancillary applications and the EMR system

Verified – We’re seeing successful connections from the HL7-01a system to EMR-DB-01a, runs a MySQL query, outputs it to a file on the HL7-01a server, and then SFTP’s it to DW-DB-01a for import.  Then, it emails the ‘dwftpservice’ Email account and provides the status.

• Allow HRHIS Client Application to communicate with HRHIS Web/App Server
• Allow HRHIS Web/App Server to communicate with HRHIS Database Server

Verified – We’re able to access the IceHRM, HRHIS system, and login.  We’re also able to pull up Employee data, all from the HR Desktop system.

• Allow EMAIL messages to be sent as necessary. Certain applications are emailing their status updates.

Verified – This is the same confirmation email that was sent from the HL7-01a system.

• Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.

• Block any unknown communications except the actual application traffic and restrict access to the HRHIS application to only a HRHIS Desktop system running the HRHIS Client Application.

Verified – We’ve attempted access to the HRHIS system from the Clinician Desktop.  You can see that we’re not allowed access from this machine.

• Allow bi-directional communication between the Infrastructure Services and all applications that require access to those services

That completes our verifications.  As you can see we were able to use vRNI to do Micro-segmentation at scale and get to a Zero-Trust model faster than traditional methods.

The VMware NSX Platform – Healthcare Series – Part 4.2: Micro-segmentation Practical with Application Rule Manager

Originally this series on Micro-segmentation was only going to cover Log Insight, vRealize Network Insight (vRNI), and VMware NSX.  With the release of VMware NSX 6.3, there is a new toolset within NSX that can be leveraged for quick micro-segmentation planning The Application Rule Manager (ARM) within NSX, provides a new way to help create security rulesets quickly for new or existing applications on a bigger scale than Log Insight, but smaller scale than vRNI.   With that in mind, we’re going to take the previous post using Log Insight, and perform the same procedures with ARM in NSX to create our rulesets using the same basic methodologies.

The Application Rule Manager in VMware NSX leverages real-time flow information to discover the communications both in and out, and between an application workload so a security model can be built around the application.  ARM can monitor up to 30 VMs in one session and have 5 sessions running at a time.  The beauty of ARM is that it can correlate the information that you would typically have to either have or look up in Log Insight to create your rulesets and significantly reduces time to value.  ARM can also show you blocked flows and the rules that are doing the blocking.

Let’s bring back our use case from the previous post and sub in, ARM instead.

Use case – Provide a Zero Trust security model around a Healthcare Organization’s EMR system.  Facilitate only the necessary communications both to the application and between the components of the application.

• Allow EMR Client Application to communicate with EMR Web/App Server
• Allow EMR Web/App Server to communicate with EMR Database Server
• Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.
• Allow bi-directional communication between the Infrastructure Services and the entire EMR Application

Problem – The Healthcare organization does not have a clear understanding of how the application communicates within and outside the organization.  Organization wants to lock down the EMR application so that only known good workstations can access.

Technology used –   

Windows Clients:

• Windows 10 – Clinician Desktop – Client-01a (192.168.0.36)
• Windows 10 – HR Desktop – Client-02a (192.168.0.33)
• Mac OSX – Unauthorized Desktop – (192.168.0.18)

VMware Products

• vSphere
• vCenter
• NSX and Application Rule Manager

Application in question –

Open Source Healthcare Application:

• OpenMRS – Open Source EMR system
  • Apache/PHP Web Server
  • MySQL Database Server

Infrastructure Services:

• NTP

The first thing we need to do to utilize ARM is establish the VM’s that we’re going to look at the flows from.  These systems will represent the session that we will build and run for a period of time to examine what flows are coming in and going out, and between these systems.

Since Infrastructure Services are generally global services that affect more than one system within a data center, we’re going to build the rules that accommodate our NTP virtual machine first.  We’re going to see more flows to other VM’s in the lab, but we’ll be narrowing it down to just the EMR systems.  Then when we’ve got those working, we’ll do the EMR-based rules.

First, we need to establish our naming scheme and methodology to writing our rule sets.  Below is a chart I use when creating my rules for the applications.  It helps me lay things out logically and then apply it quickly in the DFW.

You will notice that I like to nest my Security, Services, and Groups.  I do this because at scale, this makes changing things much simpler and more efficient.  If I have to swap in a service or change a port, I want the changes to be reflected to all affected groupings down the table without having to seek out where else I might need to make a change.  This has consequences as well, you must pay attention to what you’re doing when you make a change as it can have effect on other applications.  This is why I name things the way I do, to ensure that any change you make that you understand the implications.

Now that we have all our custom services and naming convention in place, we can go back to the ARM and start resolving Services, Service Groups, and building out rules.  We’re going to take these flows and squeeze them down even further to create very succinct rules with as few entries as possible.

We’re going to create a session called INFRA MONITOR and add the NTP-01a server into the monitor so we can see all the flows coming in and out of it.

The amount of time that you collect flows for is up to you.  My advice is the following:

• Understand the application you’re planning for. If this application is a billing cycle application that does 30 day monthly cycles, ARM my not be the tool to use.  ARM was built to monitor flows in real-time over the maximum of a 7-day period.  If the application you’re looking at has periodic flows that occur over a longer period of time, I suggest using vRealize Network Insight for those applications.  It has a longer retention period for looking up flows.
• Keep the applications small. ARM has a 30 VM upper limit on monitoring of flows per session, with up to 5 sessions.  While most applications may fall into this range, some applications may have more than 30 machines.  In that case, my suggestion is breaking up the application into smaller chunks and running the chunks in different sessions to keep the numbers down.
• ARM uses NSX Manager resources. Just like Flow Monitoring, ARM uses resources from the NSX Manager to collect flow data.  Be careful of NSX Manager utilization during this process.  Unlike Flow Monitoring, if you exit the interface, ARM will continue to run until you stop the session collection.

Once we hit OK, the flow collections will begin.

When we’re comfortable that we’ve captured all the flows from the VM, we can stop the collection and ARM will take the number of flows, de-duplicate the flows down to unique flows for us.

Now we’re going to analyze our flows.  This is going to resolve any VM names it can as well as resolve the ports to protocols that they could be.

I’ve hidden other flows to other systems within the lab for simplicity sake for now, and only focused on finding the flows to the EMR system.  You’ll also notice a column that I added into the View Flows tab.  The column is Rule ID.  The Rule ID column will show the applied rule that affects that flow in the View Flows table.

As you can see from the output, all the flows appear to be hitting the Rule ID of 1001.  If you click on the Rule ID 1001 link, it will pop open the details of the Rule from the DFW.  While this is showing an allowed flow, we will see later, that we can also see flows being blocked by a Rule ID.

The View Flows tab only resolves to VM names.  We’ll need to take this information and create appropriate Security Group, Service, and Service Group names for our rules.  We’ll click on the gear icon next to the EMR-DB-01a VM and select ‘Create Security Group and Replace’.

We’ll name the Security Group after the naming scheme you’ll find below of EMR-SG-DB and we’ll statically add in the VM, EMR-DB-01a.

What we’ll find is that the Security Group now replaces the virtual machine as the Source.  But since we’re both of the VM’s that make up the EMR talking to the same destination, we’ll create an EMR-SG-ALL Security Group and nest the individual VM Security Groups within it.  To do this, we can click on the gear for the Source we just changed, and select ‘Create Security Group and Replace’ same as before but this time we’re going to add the EMR-SG-DB Security Group to the EMR-SG-ALL Security Group we’re going to create.

We should now see that the Source has changed to the EMR-SG-ALL GROUP.

For the EMR-WEB-01a VM, we’ll do the same thing.  Create a Security Group called EMR-SG-WEB and add that VM to it.

We’ll then click on the gear and select ‘Add to existing Security Group and Replace’.  Then select the EMR-SG-ALL group, and this will nest the EMR-SG-WEB Security Group into the main group.

We repeat this same process for the NTP-01a VM according to the layout above.  Create an INFRA-SG-NTP and add the NTP-01a system.  Create a INFRA-SG-ALL and add the INFRA-SG-NTP Security Group to it.  This will allow us to add more SG’s to the main group later as needed without having to write another rule to do it.  When done, it should look like this.

You only need to do this to one rule as you’ll see later, we’ll grab both of these flows to write one rules.  We focus in on the Services that we found now.  If you click on the ‘2 Services’ link, you’ll see what ARM resolved the ports to, protocol-wise.

I like creating my own Services and Service Groups.  This way we know if we make changes, we know what could possibly be impacted.  So with this, we’re going to perform a similar workflow as we did above.  Create a Service and Replace.  Create a Service Group and Replace.  This will add the custom Service we create to the custom Service Group we create.

Leaving us with this final result.

We’re now ready to create our rule from the two flows.  Selecting both rules, we right click and select ‘Create Firewall Rule’.

We’ll see some unresolved items in the list, but we’re going to make modifications so that this one rule covers both flows.  We’re going to:

• Remove the NTP-01a as the Destination leaving the INFRA-SG-ALL
• Remove the UDP 123 Service leaving the INFRA-SVG-ALL
• Remove the vNIC for NTP-01a and add EMR-SG-ALL and INFRA-SG-ALL

Once we click on OK, we’ll have a new rule created in the ‘Firewall Rules’ tab.

Double check our rule and make sure it looks correct and then hit ‘Publish’.  We’re then prompted to create a section name and select where to insert it.

We should get confirmation that the publish was successful and we can go to the ‘Firewall’ interface and verify our work.

Everything looks good!

Now we can focus on the EMR system.  Talking with our applications team, we have determined that the VM’s in question are:

• EMR-WEB-01a
• EMR-DB-01a

Now that we know the names of the VM’s, we can go into the Flow Monitoring section of the NSX Management console and select Application Rule Manager.  From here we can Start New Session.

We’ll select the VM’s we discussed above, EMR-WEB-01a and EMR-DB-01a, and we’ll add them to the session.  This will start collection of flow data from the vNIC’s of these VM’s and post them in the View Flows table.

From here we will create a new session, calling it EMR MONITOR, and add in our VM’s to the session.  Once we hit OK, the collection will start and we can stop the collection when we’re comfortable with the period of time we collected for.  In this instance, I’m going to collect data for 15 minutes.  I have automated tasks running to provide traffic to the EMR so we can see flows that run every 5 minutes, so this should be long enough for this demonstration.

Once we have stopped the collection, we should see the View Flows table has several flows showing.  ARM will attempt to de-duplicate repeated flows as much as possible.

Analyze the output

Now that ARM has analyzed our flow data and matched where it can we can see a few things:

• Direction – This tells us in what direction the flow came to the source
  • IN – Inbound
  • OUT – Outbound
  • INTRA – Between
• Source – Resolved to a VM name if the VM falls under the scope of the vCenter/NSX Manager relationship. If an IP address, represents an external IP system that is not resolvable in the vCenter/NSX Manager relationship.
• Destination – Resolved to a VM name if the VM falls under the scope of the vCenter/NSX Manager relationship. If an IP address, represents an external IP system that is not resolvable in the vCenter/NSX Manager relationship.
• Service – Resolved to all services that exist within NSX. If more than one service is shown, this means that the user will need to manually pick the service it correlates to because NSX has more than one service definition with that corresponding port number. If you want to create a custom Service or Service Group, which we will

With the information shown we have the following:

• Inbound 192.168.0.99 > EMR-WEB-01a TCP 8080
  • 168.0.99 is identified as the Management Jumpbox for the infrastructure.
• Inbound 192.168.0.36 > EMR-WEB-01a TCP 8080
  • 168.0.36 is identified as the Clinician Desktop
• *Inbound HL7-01a > EMR-DB-01a TCP 3306 – Skipping this for now
• *Outbound EMR-DB-01a > NTP-01a UDP 123 – Rule ID 1028
• *Outbound EMR-WEB-01a > NTP-01a UDP 123 – Rule ID 1028
• Intra EMR-WEB-01a > EMR-DB-01a TCP 3306

What you may notice when you have the Rule ID column unhidden, is that the two flows we already built for above in the Infrastructure section are showing as the Rule ID 1028 which is the Rule ID for that Infrastructure rule.  As I said before, ARM can show you any flow and the corresponding rule in the DFW that it’s hitting.  All of the rest of the flows are hitting the Default Allow Rule ID of 1001 just like Infrastructure before we started.

What this means for us, is that we’re already covered on Rule ID 1028 for NTP Services and we can hide these two flows.  We’re also going to hide the HL7-01a flow so we can see another interesting feature that ARM can show us.

This leaves us with three flows we need to write rules for.  Using the same methodology as we did above, we’re going to create our table for naming and apply it to the VM’s, Services, Security Groups, Service Groups for the EMR.

Good news is that we’ve already created Security Groups for the EMR system when we created the Infrastructure Rules.  We just need to ‘Replace with Membership’ each VM with the appropriate Security Group.

When you select ‘Replace with Membership’, ARM will show you all the Security Groups that the VM belongs to.  You’ll notice the differences when you change EMR-WEB-01a and EMR-DB-01a.

We do however, need to create IP Sets for the Desktop systems that we’re going to allow access to the EMR through.  We do this by clicking on the gear and selecting ‘Create IP Set and Replace’.

We then create the EMR-SG-ACCESS Security Group and add these two IP Sets to it and replace.

We need to resolve our Services now as well.  We have two unresolved ports of 3306 and 8080.  Again, we’re going to refer to the table and create new Services and Service Groups for these.

There’s no reason to resolve the service for the second IP-based flow, as the services are the same and we’re going to create one rule like we did above for these two flows since they’re fundamentally the same.  Time to swap in Service Groups.

We’re now left with this output.

Let’s create our rules.  We’re going to select both of the IP-based rules and combine them.

Cleaned up it looks like this

Now to create our INTRA communication rule between the EMR-WEB-01a and EMR-DB-01a VMs.

Cleaned up it looks like this

We now have our two rules that we can publish to the DFW for the EMR application.

We’ll create a new section called ‘EMR’ and place it below our Infrastructure Section.

Double check our work.

Things look correct.  Let’s add in our Block Rules so we can do our testing to make sure we did this correctly and it meets the requirements.

Here’s the list of requirements the customer gave us:

• Allow EMR Client Application to communicate with EMR Web/App Server
• Allow EMR Web/App Server to communicate with EMR Database Server
• Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.
• Allow bi-directional communication between the Infrastructure Services and the entire EMR Application

With ARM, we can check that these requirements are hitting the correct rules very quickly.  We’re going to create another Monitor session to monitor EMR-WEB-01a, EMR-DB-01a, and NTP-01a.  When we see the flows that come out, we should see the rules that they hit.  This will quickly tell us if things are working correctly.  I’m also going to generate traffic that should hit the block rules because they don’t meet the requirements.  Let’s start!

We can see flows hitting the Rule ID’s to the right which is good.  We’re going to stop the collection and do an analysis on the flows captured.

We can see the flows captured and the Rule ID’s associated.  Nothing appears to be hitting the Default Allow Rule ID 1001 which is good.  The Application is functional and NTP is accepting connections from the VMs.  If we take a look at the first requirement:

• Allow EMR Client Application to communicate with EMR Web/App Server

You can see that both of the IP-based flows are captured under Rule ID 1030 which when you click on it, shows the correct DFW rule.  Let’s check the next requirement.

• Allow EMR Web/App Server to communicate with EMR Database Server

You can see that by clicking on the Rule ID of 1029 for the EMR-WEB-01a to EMR-DB-01a flow, that that traffic is hitting the correct rule.  Let’s check the next requirement.

• Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.

When we look for block based rules, we’re looking for anything hitting a block rule in the DFW.  In this case, Rule ID 1032 is a Block rule.  We seem to have two flows that are hitting that rule.

When we drill into Rule ID 1032, we see that it is indeed, our Block rule.  Let’s look at the last requirement.

• Allow bi-directional communication between the Infrastructure Services and the entire EMR Application

From this picture, we can see that there are two flows from the EMR VMs to the NTP server.  Both are hitting Rule ID 1028.  When we open up that rule we see that it is indeed hitting the correct rule to allow NTP traffic to flow to the NTP server from the EMR.

This completes our requirements set forth by the customer to meet to secure the EMR application.

As you can see, ARM is very adept at helping simply micro-segmentation with VMware NSX.  Taking the concepts we learned and leveraged with Log Insight, we can remove quite a bit of manual processes involved with resolving services and VM’s IP addresses to their names.  I was able to perform this process in about 15-20 minutes outside of flow capture time from start to finish and that’s what makes ARM another very useful toolset to use for reducing the complexity of micro-segmentation.