On November 24, 2021, Chen Zhaojun, part of the security team of Alibaba Cloud, discovered a previously unknown zero-day vulnerability in the Log4j Java logging library, now known as Log4Shell (CVE-2021-44228). The discovery was made public on December 9, 2021, sending the community into a frenzy to remediate it. 

Since its initial report, the Log4Shell vulnerability has been highly exploited, and some of the world’s most widely used software and services have been under attack. According to Wired, “attackers have exploited the flaw to install cryptominers on vulnerable systems, steal system credentials, burrow deeper within compromised networks, and steal data.”

Unlike many of the blog posts already published on this topic, we will not discuss the technical aspect of this vulnerability and why it happened. Instead, we will focus on the organizational and strategic measures we should consider in the future to be better prepared for the next time this happens, which will be sooner rather than later.

The Open Source Tradeoff

As developers, we have had access to high-quality programming languages and compilers for several decades now. The ability to create clean and reusable code has been available for many years. But the most important advancement in the last few years has been the acceptance of open source in our organizations.

In the past when an organization built a system, it needed to start from scratch. Apart from some modules or libraries previously built within the organization or even a collection of code carried over by the individual developer, the new system had to be made from the ground up. Every single part of the system — every need — had to be made all over again. 

It is no wonder why organizations cared deeply about their code, as it was a valuable intellectual property that gave them an edge against the competition. But that required developers to reinvent the wheel time after time, and the loss of that knowledge with the attrition of the developers or the evolution of the tools and languages we used made it fragile and easy to be lost.

That is where the open-source movement comes in. The idea is to pool all of our knowledge — all of our intellectual capacity — into a common repository that is easily and freely shared. Those system needs which we find in every new one can be met with a previously developed and tested solution.

The effect on our productivity is exponential.

We have access to thousands of readily available frameworks and libraries, from the simplest of things to complete scaffolding for our systems. We can do more — and faster than ever before. And we can do it in a more secure way, as those libraries, used by thousands of different organizations, are tested both in a rigorous and practical way by all of us, making the detection and remediation of errors and vulnerabilities a shared task, 

In short, open-source enables the community to pull more resources into development and testing than a single organization could ever muster.

But it has a tradeoff, as everything does. We are trading that improvement in productivity for a loss of control, potentially making our systems more vulnerable. The libraries we use are not completely controlled by us, and we should adapt and plan for that.

Let us give an example.

On March 23, 2016, a developer, completely in his right as the author, unpublished many packages he had chosen to make available on NPM, the most popular Javascript package manager. One of those packages, left-pad, consisting of 11 lines of code doing a simple task, was used by thousands of different other packages and frameworks, including some of the most widely used ones in the Javascript community. Unable to find the left-pad package dependency, these packages and frameworks became unusable, stopping the development efforts of many organizations for hours or days.

What happened recently with the Log4j vulnerability is another instance of the same risk. The ability to log the behavior of our systems is a very important part of a well-made solution. It is a need we all have, and that is very similar for all of us. To use a very well-understood and tested library as Apache Log4j is much more productive than trying to build a new solution from scratch every single time. So we use it in almost every Java system that there is. And when a big vulnerability is found in it, it affects every single one of them at the same time. 

But let’s be very clear here. To stop using open source libraries like left-pad or Log4j is not the solution. The improvements in productivity that they allow are vital for the software-centric times we live in right now. It is because of the open-source community that those libraries were developed, used, and tested. It was because of their open nature that the vulnerability was found, and was remediated quickly.

Lessons Learned: How to Move Forward into a More Secure Open Source Future

So what are the lessons we should take from this? As we move forward into an open-source future where software is developed and managed not by organizations, but by global communities, we must keep the following principles in mind: 

• We should, as developers, be careful of the libraries we use. Yes, there are thousands, readily available for us. But that doesn’t mean we should use all of them. We should use as few of them as possible. Each one of them is a dependency we should take care of, and minimizing the number of them will make the management of our systems, over time, easier and cheaper.
• We need to manage the dependencies we have. If we don’t know which dependencies our systems have (and even the version of each of them), it is very challenging to find out if we are affected by a specific vulnerability, even if the vulnerability is well identified and already remediated by the community. If we don’t know all of our code dependencies (and the dependencies of those dependencies of third-party solutions we have) we can hardly remediate their vulnerabilities.
• We need to have support in our third-party solutions. All of the third-party software solutions we use will have vulnerabilities, and we need to make sure those vulnerabilities are remediated as quickly as possible. To keep track of those solutions, and patch them as recommended by our providers, is very important. Having third-party solutions without support is a risk we shouldn’t accept.
• We must have the bandwidth to make quick remediations in our existing systems. The developers in our organizations must have the capacity to patch our systems to remediate vulnerabilities or issues in our dependencies quickly and frictionlessly at any moment.
• We need to track everything! The need for new tools to track and conform to security and organizational requirements as the number of dependencies our systems have has exploded. Some examples are tools that make sure we are not using dependencies we don’t have approved, that the open source license conforms to our needs, and that they are patched (remember the Equifax breach?). These tools already exist, and we should make good use of them to keep our systems safe.

Furthermore, having a good Continuous Integration/Continuous Delivery (CI/CD) pipeline can make the patching of the system a quick, fast, and painless process. Having automated testing of our applications and a well-understood mechanism to deploy changes when needed is vital. If our process to deploy code needs months of lead time, the repercussions for our security posture are unacceptable. 

If the only other option is to bypass the testing-QA-deploy process to allow the change to happen quickly enough, the risk for the availability and even security of the applications can be worse than the original vulnerability. That is why having an automated DevOps pipeline is vital not just for the normal lifecycle of our applications, but also for these special situations.

And let’s remember, open source is not the problem. Open source is a great tool to boost our productivity and security to higher levels than before. As with everything, we need to know how to use it properly, but its advantages make it worthwhile.

Working with a software development and security consulting partner like Wizeline with excellent capabilities and knowledge, ample experience, and a pool of developers available at a moment’s notice can resolve many of these needs. Every day, we help our customers face their security and development challenges, ensuring that they survive, thrive, and grow in the software economy of the future. To learn more, contact consulting@wizeline.com today.