I understand; auditors like me are often seen as the spoilsports who interrupt the flow and the fun of DevOps . After all, when teams of engineers can work cozily together on their product, they master their sprints and get into the zen of automation. Why would they want to break that up to do paperwork and also automate security checks? Why would they want some external auditor to come waltzing in and asking for receipts, literally and figuratively? So yes, it’s been hard for the DevOps community to see an auditor as someone who can actually help the entire team fulfill its value-delivering goals.
Yet, mission-critical auditing needn’t be dichotomous to DevOps. A good IT auditor should in fact support, not slow down, engineering projects. Ultimately, the audit goal should be to help improve the quality of their processes and outcomes.
Life made better?
One of the main reasons auditing and DevOps have become so dichotomized can be traced to that classic three lines of defense setup. In this arrangement, there is a cut-and-dry separation of powers between departments that handle business; security and risk; and auditing . The whole system is not in the room , as Schuberg Philis always stresses is crucial to meet the multidisciplinary, multi-departmental demands of mission-critical IT. Instead, experts each work in their own silos and, as I’ve described before, none speaks each other’s language.
DevOps requires a different control approach than previously used. This has been the conclusion of multiple research papers and white papers , one notable example being the latest ISACA-published COBIT framework (short for Control Objectives for Information and Related Technologies). The DevOps community itself attempted to reinvent its ways in 2018, creating the DevOps Risk and Controls Matrix. Amusingly, the matrix is prefaced by a self-described “love letter to auditors from DevOps”, where they promise to make life better illustrated by a giant heart. “Dear Auditor” is written in white cursive, giving the impression of a Valentine’s Day card, which is cute but no doubt an indication of the need to improve feelings of affection between these two groups.
The matrix marks a starting point and gives some good clues. But unfortunately, four years later, we see no big improvements in our lives with DevOps engineers across the board. Both of us are still unfamiliar with engaging each other and gaining trust.
Bringing worlds together
But having audited software development projects for a decade now, I’ve observed ways we can depolarize these groups and bridge gaps in their relationship. I took a step in this direction with the report “DevOps and Agile in control ,” which is written in response to the lack of control frameworks specifically tailored toward DevOps . The report has been authored with NOREA, the IT auditors’ professional association in the Netherlands, where I’m chair of the DevOps and software development work group.
In the report, a control framework is presented to conceptualize the mitigation of the key IT risks associated with Agile and DevOps principles. The framework is based on my experiences at Schuberg Philis and interviews I conducted with engineers, IT managers, and project managers encountered through my work. And while the NOREA report serves as a kind of handbook, with concrete recommendations for auditors like me, I imagine equivalent guidelines would be helpful to the DevOps community too.
The control framework presented currently consists of 15 suggested controls. We advise teams to start anywhere in the framework according to the needs and risk areas. Wherever they start, the goal is to improve security and predictability by implementing the controls . Below we present a preview of the focus areas of the controls, along with common points for improvements I encountered.
- To expedite the work of auditors and therefore accelerate the entire DevOps project, several controls are defined. They focus on improving the way of working in the DevOps teams. We still often observe teams changing their way of working multiple times per year without any documentation and having engineers still following old agreements or a combination of old and current agreements. This negatively impacts a team’s efficiency as well as quality.
- In an effort to unite the worlds of auditors and DevOps, auditors should dare to leave behind a reality solely determined by policies, procedures, and documentation as evidence. A good way to merge worlds could be through accentuating the everything-as-code principle and a centralized version control system. The automated deployment pipeline also has a lot to offer because it is not just highly automated, but also sequential – and sequentiality really speaks to auditors.
- Still, to bridge the two worlds effectively, teams do have to document their environment better. We seldom encounter a properly documented DevOps team, often leaving us with the same questions. What applications or software are in scope? Through which deployment pipeline are they managed? Which solutions, third-party or otherwise, are used in these pipelines? Are the pipelines managed through the version control systems as well? How have these pipelines been maintained and changed throughout the year? Are the product teams or separate platform teams responsible for the pipelines?
- Once teams have defined and documented their way of working and deployment pipelines, we look to see if implementation or configuration has been done properly. We ask: Do all pipelines fulfill requirements set by the team ? How is access and authorization to these critical configuration settings managed? And crucially, do we feel a team is in control?
- DevOps teams are renowned for high automation. Still, we need to know: has a team adapted a test strategy? Does it also include security tests? Are there differences between the different solutions or services they develop? Per type of test, have goals been defined and deemed relevant? For example, are test coverage goals measured? Without these answers, we would struggle to give a proper opinion on proper testing by the teams.
- Peer review is a common and accepted practice in most DevOps teams. However, which checks are executed by the peer reviewer is often not visible to auditors. It would be awesome, therefore, if a reviewer could document which tests they encountered, what the achieved test coverage was, if it meets the quality and security requirements of the solution or service, and if no critical findings were open.
- Another salient characteristic of DevOps teams is their independence. This is great, but proper involvement of the end user and an organization’s business is crucial not only for the requirements phase, but also for change acceptance or deployment. And so we want to know: has the team made agreements meriting these parties’ involvement for testing and deployment approval? Does this fit with the associated risk profile of the solution or service? Are the activities executed by these parties documented as well?