Business Continuity - Where to Start?

I sat down with Chris Harms, one of our Solution Architects here at Sentinel Technologies in Grand Rapids.  Chris holds a variety of certifications from Cisco, HPE and VMware and has familiarity with many data center related technologies including Microsoft Active Directory, Exchange/Office 365, VEEAM and NetApp.  We talked about Business Continuity and Disaster Recovery.

---

Josh:

Chris, let's talk about business continuity, disaster recovery and how companies might be able to get into that quickly and relatively inexpensively. Is that even a possibility today?

Chris:

So the first decision that any company needs to make when considering business continuity is determining what is required for their solution. Is there a subset of their current infrastructure that is required? If everything running in the primary data center is required then the larger the footprint of VMs and applications going to DR will increase the cost. So you want to make sure that your are covering what the organization absolutely needs.

Domain controllers are a must. Database servers most likely are critical any cloud connection servers that you have out there. Of course, whatever primary applications that they need, file shares things along those lines. Those are all part of, you know, the the footprint that I would recommend organizations have.

Josh:

It's easy to sit back and just say, put everything we have running into our business continuity plan, right? Because if it's running, we need it. But in reality, you could easily trim a lot of that back, like you said.

Chris:

That's right.  Then once you have an idea of what's required, then you need to start considering your RPO and your RTO, the Recovery Point Objective and your Recovery Time Objective. The Recovery Point Objective is just that it is the point in time in the past at which you are OK with bringing your infrastructure back to.
For example, if you're OK with having your entire infrastructure set up from the previous night's configuration than you have a 24 hour recovery point objective. If you need a four hour RPO, then your solution needs to make sure that it's shipping data to your DR site every four hours to make sure that you can meet your four hour recovery point

Then, once you determine your recovery point, then you need to consider your Recovery Time Objective (RTO). How quickly do you need to have everything back up and running in the event of a dire situation? RPO is a big influencer in cost, but the RTO is the number one factor for cost. The faster that an environment has to be up, the exponential increase in cost for the DR solution. If you're OK having part of your infrastructure up within 24 hours, that could be greatly less reduced cost than if you need to make sure that your entire infrastructure is up within an hour. Because you know there's different solutions for different aspects of this. But the faster that you need DR back up and running, it increases the cost exponentially.

 

Josh:

Regarding backup and DR, I feel like we're seeing a lot of these these companies that maybe only did backup in the past start to take on some more disaster recovery features and vice versa. What are your thoughts on that and how does that change the decision process for choosing solutions?

Chris:

There are a number of backup vendors out there on the market that will back up your data. They also now provide replication of that data to other locations. Problem with those solutions is that even though they're backing up and replicating the data, you're not really poised to have that restored easily into whatever infrastructure you backed that up to, whether it be, you know, a DRT facility, public cloud, private cloud. Those are just the backup files from your backup, your backup infrastructure. They're ready, waiting to go and be restored out of a backup infrastructure. So if you are using a backup product to replicate your data to a DRT location but you don't have the backup infrastructure in place there, you'll be waiting to set all that up. Then you have to do the restore out of the backups which typically takes a long time (effecting your RTO), then you'll have to start bringing everything up yourself. It's a time consuming laborious process, and you may even have to put hardware at a DR facility in order to do some of that.

On the other side, there are dedicated DR products. They can get your data from your primary data center sent to your DR data center very quickly. They can spin up your infrastructure at the DR data center very, very fast using orchestration and automation features. However, they're not really designed for recovery of individual files or individual VMs. You can get data out of one of those solutions, but it's a very time consuming and laborious process.

We are seeing the two feature sets (Backup and BC/DR) coming together in many of the market leaders and that is a good thing.

Josh:

What about leveraging public cloud for the DR location? Are you seeing tools that integrate into the public cloud and let you leverage IaaS (Infrastructure as a Service) only when you're in that dire situation?

Chris:

Yeah, that is very common today, and it is a by far my number one recommendation for a DR solution because you're not relying on having to provide the hardware yourself.  I'm not really a fan of having a co-location or another data center with a bunch of hardware that's just sitting there waiting. It's not a good use of funds for an organization, depending on the scale of the organization, of course, but most organizations just can't afford to have that investment in a DR facility. So going to a public or private cloud is amazing because you just have to pay for the storage and maybe a VM or two that are running in real time for your solution.

In the event you declare a DR event and you start bringing it up, of course, you're going to start incurring  more costs because you're now consuming more cloud resources. But at that point, if you're declaring a DR event, you are in need of having those resources at that point so that that is totally to be expected, but you're not on the hook for any of that infrastructure cost while not having a DR event.  So I'm a huge fan of going to either a public or private cloud for your DR scenarios. And again, every organization is different. There are organizations that require their standalone data centers with hardware ready to go. But for the majority of organizations, having DR into the cloud in one fashion or another is going to be the right solution for them.

Josh:

There's also a lot of As a Service solutions as well, right, whether it's going to public cloud, or keeping DR internal, the DR application itself, the implementation and management of DR including testing can be offered as a managed service?

Chris:

Yep. As with any platform, it's only as good as the management that's supporting it. So once you deploy a DR solution and get it up and running, you can't just let it sit there. You need people who are managing that platform for you, keeping up with changes in the primary data center.

DR as a service is becoming very popular because an organization can partner with a technology company that specializes in BC/DR. They will provide the software, they will provide the storage and compute resources, then they manage everything for you. They work with your team to get everything set up, get everything configured, and they then they make sure everything stays running. They also will most likely conduct a annual test of your DR to make sure that it's meeting your needs to make sure it's working.

Josh:

Connectivity between the DR site and all other business locations is often overlooked isn't it?

Chris:

It is, and not only that but connectivity to the VMs and resources that are now running at the DR site.  This often requires IP address changes and possibly routing changes in the network and that can be complicated to implement which can increase RTO.  I think using tools that orchestrate these changes provides the best results and having it all wrapped with a managed service can make it fool proof.

---

If you are facing a challenge similar to this in your IT organization, we are prepared to help get you moving in the right direction.  Feel free to reach out to me and we can get the conversation started.

Endpoint Protection - Renew or Review?

Brian Dudek and I had a short conversation about the Endpoint Protection space. Brian has been in the IT industry for 25 years with a slant toward data security.  He is an Adjunct Professor at Davenport University creating data security curriculum.  He is currently a Strategic Business Advisor providing guidance to IT leadership throughout West Michigan.

---

Josh:

All right, so, Brian, let's talk about endpoint protection and maybe walk us through how that's changed over the years. I mean, I remember 20 years ago, it was basically the antivirus scanning your PC looking for known file names and whatnot. It's much more sophisticated today. Maybe help me understand that. 

Brian:

Yeah. So back about 20 years ago, antivirus was all signature based, which means that when a threat was found in the wild in terms of a virus or worm, the antivirus products would create a signature that would be released. It would take anywhere between 30 to 90 days for that signature to populate around the Internet to all the various antivirus products. The change has been moving away from the signature based antivirus to more behavioral based or A.I. based, which means that the software is taking in a lot more data and making comparisons. One key differentiator is that they are constantly scanning the files that you're opening and looking for anything anomalous that may happen - that is one data point. They also continue to use signatures which is another data point. 

Josh: 

Those 30 to 90 days before that signature made its way out to all the clients became a problem as some of the malware became more destructive. So they had to figure out some new ways of detection, that's where the live scanning comes in and things like that. Is that right? 

Brian: 

Yeah, that's correct. And also the fact that the people who are making these viruses and malware attacks had taken the original virus and they make some slight changes to it. So what that means is that the signature based stuff would not find the new version of the virus for another 30 to 90 days - the antivirus just couldn't keep up and that's why the behavioral based capability is so important. 

Josh: 

Some of the new software is also leveraging network controls - is that another data point? 

Brian: 

Leveraging network control is important as a data point for detection of malware activity as well as preventing further actions. Modern tools provide inbound firewall features to stop access to vulnerable services, application whitelisting and blacklisting to prevent certain executables access to the network, as well as complete isolation in response to a serious event. Those are things that have been adopted in the antivirus products these days. 

Josh: 

Ok, what about integration into for instance, Sentinel has a managed SIEM and SOC. I imagine the Endpoint Protection plugs into the SIEM - is there any automation there? Automatically detecting malware and taking action? 

Brian: 

Yep, that is true. And when you talk about tools like a SIEM, they have always been able to ingest these logs. There are safeguards put in place for the A.I. to shut the network connectivity down for a client system, but the best practice is always to have somebody (a person) looking at the events to determine if there is an issue or not. This is why a quality Security Operations Center (SOC) is so important - we aren't at a point where automation can completely run your data security systems.

Josh: 

How has increased remote work changed the situation? 

Brian: 

What's a bigger problem is a lot of our clients, when people moved off site, started working remote they click on something and they click on a link or they're browsing Facebook at home and they don't have the web filtering at home like they used to when in the office. They don't have the firewall and intrusion protection infrastructure. So now all of a sudden, that antivirus product is the first, last and only defense that the laptop has. So that's the biggest challenge that we have right now is that organizations, when they sent users off to work, they didn't have multiple defense in depth for those client systems. So, yes, if they use their personal PC at home, that's another challenge in itself, too. And a lot of our clients were trying to figure out, OK, do we make them install another antivirus product? So there I mean, because if you're buying something for your company to use from a virus perspective, you can't quite put it on an employee workstation at home that you don't own. Right. So but but yeah. I mean, that's that's there has been a challenge. Sure. 

Josh: 

Is there anything else worth discussing, anything new coming? 

Brian: 

Really, the one thing that I want to reiterate is that everybody should be taking a look at what they currently have and evaluating if it's signature based or behavioral based. If it's a signature based product, which there are a lot of companies out there, then you may want to look into modern products that include more A.I. and behavioral capability. A lot of legacy antivirus products make it really easy to renew licensing so a lot of companies just go that direction and don't realize there is much more comprehensive security coverage available to them.

---

If you are facing a challenge similar to this in your IT organization, we are prepared to help get you moving in the right direction.  Feel free to reach out to me and we can get the conversation started.

Merging Two Companies: Part I - Active Directory

I sat down with Chris Harms, one of our Solution Architects here at Sentinel Technologies in Grand Rapids.  Chris holds a variety of certifications from Cisco, HPE and VMware and has familiarity with many data center related technologies including Microsoft Active Directory, Exchange/Office 365, VEEAM and NetApp.  We talked about the process of merging companies together after acquisition from a technical perspective.

---

Josh:

All right, so, Chris, let's talk about a situation where one company is merging with another, purchasing another, perhaps are of similar size, and there's all of the traditional I.T. resources, data centers, networking, you know, network operating systems with user accounts and applications and all that stuff.

The process of bringing those two organizations together: Where do you start with that? What's kind of a high level approach to begin thinking through all the different technologies there are, where does it begin in your eyes?

Chris:

So the first step you have to take is you have to decide on how you want to merge the organizations. Is one organization going to be the parent organization? You have to decide, you know, is there one environment that's going to be the one that gets migrated into the other? Once you decide that, then  assuming you're using Active Directory for your user access controls and managing your environment, which most organizations are, then the first thing you do is set up a domain trust between the two organizations.

As part of the trust, you have to set up a site to site tunnel between both organizations from firewall to firewall. And then once that's established, you create an Active Directory Federation Trust between the two organizations. This will allow permissions and access to be passed back and forth between the organizations from there. Then the organization needs to sit down and determine what all of the information that exists in an account in the environment that is getting migrated into the into the parent organization. What needs to come across? What do they want to do? What do they need to keep from each side? Do they want to, you know, just create new accounts for everyone, give them new accounts and then just delete everything? Or is there inherent data or knowledge that needs to be transmitted or migrated from the environment that's going away into the new combined environment? 

 

Click here for more information on AD Federation Services.

Josh:

Regarding that tunnel between the two organizations that allows that Microsoft Active Directory domain trust to be established, does that assume that both of these networks internally are unique? And if you have a situation where you have maybe some network overlap, does that have to be addressed first or are there ways to kind of get around some of that?

Chris:

As long as the domain controllers for both organizations are unique, that does not need to be addressed at this time. To establish the trust if there is an overlap of an IP address on the domain controllers then other steps need to be taken prior to the trust.

Josh:

Ok, so once the trust is in place, that allows you to leverage user accounts on both sides, essentially to make it a little more easy to transition applications, is that right?

Chris:

It makes it easier to transition access, not necessarily applications, but access. So people from organization one will be able to access resources on organization two and vice versa. There are a couple of different ways to set up the trust to limit that access, but essentially it is to allow access to flow between the two organizations.

Josh:

Ok, so after that is established then, are we looking at just general connectivity so that you can then look into how applications will be leveraged or what is the next step?

Chris:

So, yeah, once you've established the trust then the next major step is you need to start looking at how you're going to merge or migrate the environment into into one larger environment. So you have to go through the process of identifying what needs to come from the environment that's going away and what needs to be moved into the future environment, because that will that dictate a lot of other application security and a lot of other resource constraints that need to be identified through the project.

For example, if you know an organization that has been acquired has no on premise servers or applications, database servers, no financial applications, no H.R. systems that they have to migrate, do you need to migrate any access permissions crossed for those user accounts? Do you need to migrate user accounts? Can you just create new accounts in the environments that is remaining?

So what you want to do is identify what needs to be migrated across. If there's nothing else from an applications standpoint, then that means you most likely don't need to migrate any objects either. So it's all about looking at this from a holistic approach and see what's there today and what you want to keep and what you don't, depending on what you want to keep, will dictate the next steps

Josh:

So in some ways it might be, even though you've chosen a primary organization versus the secondary organization, you may still be picking and choosing applications from both of them to ultimately become the new environment?

Chris:

Exactly. It's important to note that these are not technological decisions that we as a VAR (Value Added Reseller) can make for an organization. These are business decisions that the organizations need to make. We have Advisory services that can help guide them through that process, but it's a business decision that really drives the direction here.

Josh:

That process in and of itself may also steer the future number of data centers and locations? If both organizations have a couple of data centers, maybe some hybrid cloud - when they come together - where will those applications live, which ones need to live through the merger, are going to dictate which data centers stay, what cloud applications continue to stay, right?

Chris:

Yeah, absolutely. From a Sentinel perspective, we have Advisory Services to consult with C-level individuals to help make those business decisions and then we have Solutions Architects, PMO and Engineering talent to implement the movement of resources between data centers and into and across hybrid cloud instances.

Josh:

It wouldn't be uncommon for some manner of consolidation to follow, bringing data centers together, choosing one cloud platform, maybe over another?

Chris:

Absolutely. Consolidation is the ultimate goal here. You don't want to have to manage four data centers when you only need two. You don't want to have to manage two or three cloud providers when you only maybe need one or two. So a consolidation will come about because of this whole process and that ultimately that will help, you know, reduce overall costs for the organization. Sure.

---

There is a lot more to consider when merging whole network infrastructures - Chris and I continue the conversation in Part II.

If you are facing a challenge similar to this in your IT organization, we are prepared to help get you moving in the right direction.  Feel free to reach out to me and we can get the conversation started.