The Elephant in the Room

The big tech story over the past week, perhaps ever has been the CrowdStrike incident. It would be impossible to put out a newsletter article this week and not draw some attention to it. Luckily, none of our managed customers have been directly impacted, although we have had discussions with CrowdStrike in the past, their particular security offering is not well aligned to small business requirements. 

But, could this have happened to another provider? The very simple answer is yes, this is the worst-case nightmare of all IT professionals, a bad update is a reality for all software, and we all dread the day it happens to us on any scale. The irony is that the very software designed to help prevent these situations has created the biggest and most costly IT outage in history. 

The reason for the scale of this outage is that the update got pushed to approximately 8.5 million Windows computers and servers, although servers have redundant systems with backups, laptops and desktops do not. The fix for these devices by in large required multiple passwords, often not easily obtainable, and physical access to the device and procedures not readily available to the common end user.

The net effect of this bad update was vast, Hospitals and Doctors across the US & UK were unable to operate, Upwards of 5,000 flights have been cancelled with a backlog of around 40,000 delays across the US and abroad that is expected to take months to clear. Major public transport providers were affected halting Train and bus services. Locally, mostly Banks and Payment systems were the only major systems affected. In all of this, we should still count ourselves lucky. This particular incident affected less than 1% of all Windows devices globally.

So the question arises, what happens in a larger outage? What happens when a bigger brand pushes a bad update? What happens when a major Internet or power provider goes offline? What happens when my cloud services go offline? Not only are these possible scenarios, they are all real-world scenarios that have happened in the past. 

Many questions will be asked over the coming months, what happens if a bigger brand pushes a bad update? Who pays? How can it be prevented? Are we too reliant on a few major Mega corps? The reality is that any solution to risk carries trade-offs that will erode security and require careful consideration. Delaying security updates and allowing individual company admins to test each update is an unreasonable overhead and would create windows for cyber miscreants. 

The question then gets asked, is it time for the humble desktop to be retired and all users moved to a centralised remote desktop architecture with backups and more redundancy? The de-centralised nature of the desktop has been one of its enduring strengths, and moving all users to a centralised model would, at first glance, appear to increase the risk of major meltdowns, not decrease it. How will your business operate during a large-scale IT outage regardless of the architecture employed?

So, what do we all do now? This will be a question thousands of businesses are asking and there will be plenty of new company's popping up saying they have the answer. My thoughts,

• Talk to your technology team, ensure they've got good documentation, would the appropriate passwords be available quickly should a similar issue arise? 
• Look at your Business Continuity (BC) plans, too often these plans are left up to the tech department. The whole business needs to be involved in your BC plans, the old adage, to a man with a hammer, everything looks like a nail is apt here. If you leave your tech department to work on your BC plans alone, they will come up with a technology solution. Technology is not the solution to every business problem. 
• You still need security, don't look to weaken your security to prevent this. The software update in question could have equally been released by any number of software vendors, on Windows, Apple, or Linux devices. 
• There is no quick fix, every option has trade-offs and these need to be considered carefully before any wholesale changes are made, don't buy-in to the marketing of any business that claims they have a solution. 

As always, our team is happy to discuss any concerns you may have, we have people versed in business continuity planning, security and appropriate digital architecture. We are happy to work through these discussions with your business, coming up with solutions that meet your requirements.