First publicized on December 9, 2021, this vulnerability caused grave concern because, Log4J, a commonly used Java library that processes log messages, is omnipresent. As we’ve said before, Log4J can be compared to sugar. It’s sweet, it’s everywhere, and, when abused, it’s not good for you. Like sugar, it also often shows up inside products you didn’t expect contained it. Log4j can be used in third-party software, customized software, and/or the third-party components or plugins they use. It can be present on a server or in a container. It can be found as a .class file or a .class file contained in .jar, .war, or .ear files, all of which can be bundled in a zip file. It can be used by the SaaS services you consume and thus cannot inspect.
Identify, patch, scan, repeat
First, some good news: Schuberg Philis encountered no Log4J-related security incidents over the Christmas and New Year’s period, which many observers worldwide anticipated could be a time for cybercriminals to launch large-scale ransomware attacks. That means our customers didn’t either.
To ensure our customers would not be impacted by any potential incidents, our customer teams worked to identify where Log4J might’ve been hiding in their IT landscapes and what versions were used. We undertook upgrading, emergency patching, and ensuring that SaaS providers took any necessary mitigation steps. On top of our regular scanning, we repeatedly scanned all known IP addresses for which we’re responsible. As a precaution, we restarting all servers and containers that contained Log4J, just to make sure no malicious actors were hiding in memory. For customers using our Security Information and Event Management (SIEM) service, we implemented additional checks to hunt for Log4J compromises and catch them early. For all customers, we double-checked our regular monitoring.
Over the holidays, each team kept a 24-hour standby schedule. We made sure the right people were always available to provide security support and crisis management, also preparing for the eventuality that multiple teams would face crises at the same time.
Back to normal?
Our efforts throughout December paid off. The fact that we’ve so far had no Log4J-related incidents attests to that. We’re proud of the work we all did.
By no means were we alone in handling this crisis. Guided by the Dutch National Cyber Security Centre (NCSC_NL), the Dutch cyber security community pulled together. Along with many others, we helped create an overview of vulnerable and non-vulnerable software, and will continue sharing our experiences and best practices with this community and generally support each other.
So nearly a month after Log4Shell was first announced, we’re returning to a more normal mode of operation. We can do this because we’ve sufficiently patched and prepared,and have the situation firmly under control. Chances of multiple customers being hit by this vulnerability at the same time have dropped significantly.
A more hostile internet
But now some bad news: Log4Shell is not going away. It is likely that in the months to come, we will see threat actors abusing systems they got access to before they were patched as well as finding new systems that have been overlooked and thus left unpatched. Indeed, attacks only get better, they never get worse – Log4Shell has given security researchers, both white-hat and black-hat, new avenues to explore. This allready spilled over into other projects that use JNDI (Java Naming and Directory Interface), the root cause of the vulnerability, and more will likely follow. In short: the internet has become more hostile.
Log4Shell has taught us that we need to find new ways to quickly identify all components of the software we use, including hidden ones, such as Log4J. That way we can defend ourselves against similar threats in the future.
5% inspiration, 95% perspiration
The quick actions Schuberg Philis took reflect our commitment to security in mission-critical landscapes. Yet, once again, this incident underlines that security is not a single system, a specific department, or a particular tool. It is an attitude. It is our teams’ constant attention to details and tedious work to continuously build on a strong defensible infrastructure that keeps our customers secure.
