Managing the Risks of Incomplete Vendor Updates

Editor’s Note: we are thrilled to welcome Asimily’s head of product marketing, Rajesh F. Krishnan, as a guest author for today’s blog. As a frequent contributor to Asimily’s content, his extensive experience in cybersecurity lends a unique perspective to the tricky nature of patching, incomplete patches, and what to do when your risk mitigation playbook falls short.

Applying firmware upgrades often falls under the best practice for mitigating risk in cybersecurity. But what happens when a patch is released incompletely or incorrectly, then applied only to cause further error? Tech companies have recently received a spate of advisories and publications regarding incomplete or failed patches from manufacturers. Examples from NVIDIA, Avanti, and Fortinet each weighed in on this peculiar area of cybersecurity. What lessons can be drawn from this, and what can operators of IT and IoT do to protect themselves?

First, it’s important to note that this type of occurrence can happen to any vendor; these recent examples do not inherently reflect the products associated with them. Cybersecurity is complicated, and resources are very limited for any organization. These vendors have thousands of customers, and even the most subtle updates can have severe implications, often introducing operational downtime, which the business usually associates with a financial impact. Security patches are developed to implement the smallest change possible to proprietary code to eliminate a newly discovered vulnerability. It’s an imperfect science because there’s no way to prove that a vulnerability was completely eliminated by a patch, excluding open source.

Security vulnerabilities don’t exist in a vacuum. They’re part of a dynamic system of signals and responses from manufacturers, users, researchers, and attackers. I used to work in the field of PTaaS (Pen Testing as a Service), where multiple cybersecurity researchers analyzed exploits for various vulnerabilities in an orchestrated approach. It’s the same part of ethical hacking that is often referred to as a ‘Bug Bounty’ or crowdsourced penetration testing.  Whenever a security vulnerability was found, it was made available to other researchers so they wouldn’t waste their time investigating an existing fix. Regardless, attention on that part of the attack surface would skyrocket after disclosure (to the researchers) of a new vulnerability. The way attackers think – logically – is that where there’s one security vulnerability, there are usually more hiding in plain sight.

Similarly, NVIDIA (CVE-2024-0132) found and patched a vulnerability in their code. However, Trend Micro kept analyzing each release, and their research led to the discovery of an additional exploit. Given the influence of NVIDIA in the AI space, they disclosed this information. 

In the Fortinet story, exploits let attackers create symbolic links to internal resources that could be accessed through a public portal.  Fortinet released a patch that remediated the exploit. However, the pre-existing symbolic links were not capable of being addressed as part of the update. As a result, the unauthorized access may still exist, even though the patch was successfully deployed. The clear takeaway is the patch isn’t the end of the remediation work – incident response and anomaly detection is still required, even more so after a patch is applied.

The Ivanti case teaches us a different lesson. There was a security flaw that was discovered and judged to be a product bug by Ivanti. They did not go through the entire CVE process, which has well known and expected levels of disclosure in detail processes (e.g., Common Vulnerabilities and Exposures). They did fix it to some degree. However, they later separated their research from Mandiant and Rapid7 indicating that there were active attacks happening exploiting the weakness. The telling line in the story about this from SecurityWeek is from Rapid7: “This is a salient reminder that state-sponsored threat actors are actively reverse engineering vendor patches for high-profile software targets, and are able to identify silently patched (or otherwise not publicly disclosed) vulnerabilities.” 

The very act of Ivanti releasing a patch focused attackers’ attention on that part of the attack surface, just like the Pen Test as a Service example above. This led to more creative exploits being developed and used.  The lesson here for the industry is wherever possible when it’s a choice between assessing something as a security bug or a product bug, lean towards security disclosure. This leads the whole industry to more rigorous standards for not only patch development, but incident response and recovery where necessary.

 So what can time and resource-scarce organizations do to protect themselves when not even every patch is going to be completely effective. First, don’t give up on patches.  They still are the best way to thwart attackers. Users are still going to be dependent on manufacturers’ cybersecurity skills.  In other words, nothing changed from when you acquired the product. IMO, the current value of software is the net present value of all future security updates, plus the product functionality.  

The only thing that’s under your control in all of these stories is the ability to take a patch offered by a vendor in good faith, and get it into your assets as quickly as possible. That means that your organization must have an effective way to continually look for these patches, deploy them, monitor for issues with the patch process in the real world, and do all of this at scale without being disruptive. You can also pick vendors with good reputations for cybersecurity. For example, Asimily’s ProSecure helps organizations determine the best vendors to work with in terms of security risk.

 Asimily developed its entire patch capability with fast, effective patching in mind.  When we thought of what the minimum viable product was, it wouldn’t be as simple as being able to click on a device and then execute a patch. It had to have that full detect-patch-deploy-patch-workflow because that is the true problem, especially for IoT. 

Instead of cynicism inspired by the fact that three patches from high-quality vendors have hit the news lately because they were incomplete, I take a more hopeful lesson out of this. In a way, this is a good sign, and shows that the cybersecurity industry as a whole is capable of finding weaknesses and making the changes necessary to get them out of your attack surface. I’m glad that our cybersecurity industry is constantly working to make IoT, OT and IoMT just a little bit safer.