In 2016, Dan Ashby wrote a blog post, ‘Continuous Testing in DevOps’. The post included a hand drawn sketch of the DevOps loop and a modification on how to include testing. Dan explained that this meant testing in all parts of the DevOps model. Since then, that image has been quoted in every discussion of “testing DevOps”, “continuous testing” and more.
In the book, Exploring Requirements, Weinberg and Gause, describe the ‘Mary had a little lamb’ heuristic. They advocate using the heuristic for ‘identifying problem-statement ambiguity’. The statement ‘Mary had a little lamb’ can be interpreted in many ways. This includes, ‘She had only one lamb, not several’ and ‘It really was surprisingly small’. (Here is a public article on the concept by Harry Robinson.) I’ve applied the same heuristic to testing the phrase ‘Testing in DevOps’. This also applies to the related concept, ‘Continuous Testing’.
There are three aspects of the ‘Continuous Testing’ image which need to be parsed:
1. Testing
2. DevOps
3. The DevOps loop
Testing
‘Testing in DevOps’ depends on how a person understands ‘testing’. Even among career testers, very few truly understand testing. Many testers who were focused on test automation before DevOps, now add to those skills, by focusing on CI/CD and related technology, rather than the abstract idea of testing.
Many DevOps engineers have a strong background in operations. The operations world did not have a formal tradition of unit testing or code reviews (related to infrastructure) similar to developers (the likes of Grady Booch, Kent Beck, Ron Jeffries). There is no tradition of discussing process or development ideology. Ideas related to testing are far removed from operations. (Important: Operations does have heavy process like ITIL. Concepts like policies, procedures, risk management are very common in operations work. However, I found that very different from individuals explaining what it means to provision a server or thinking about what can go wrong.)
Whether you can test or are motivated to test, depends on your understanding of ‘testing’.
DevOps
When I cheered Dan’s blog post, I had never worked in a team using DevOps. We mostly weren’t the worse for it. ‘Testing in DevOps’ depends on whether you understand DevOps. DevOps includes engineers who have spent decades managing infrastructure. On top of that, DevOps commonly includes cloud based infrastructure. Testers, like other engineers, would need to work very closely with an ops/devops team and get popular certifications (AWS Solutions Architect) or the related knowledge, in order to match a strong operations engineer.
To understand what is involved in DevOps, you can take a quick glance at the services provided by cloud based services like AWS or Azure. You also need an understanding of general infrastructure including networking and related equipment. You need to be very proficient in working with an operating system. You may have to install and maintain large enterprise software or infrastructure software. You may not have an option to ignore security, which is another vast topic. There are other concepts such as monitoring, source code control and CI/CD.
Can you test something so complex if you start with little or no knowledge of the subject?
The DevOps loop — Agile++
The DevOps loop seems like an extension of the image commonly seen for Scrum. DevOps seems like a faster and continuous agile. The big difference is that DevOps isn’t only a process or culture change. As explained in the previous paragraphs, DevOps includes a large amount of technology. The DevOps loop is a desirable state. It doesn’t show all the technology that goes into making the loop work. This is different from agile, which didn’t have any additional technology. Do we need to test all that technology? Is anyone testing it? How does test automation impact all that infrastructure? What part of the loop accounts for test automation? Does testing matter?
Instead of the loop
Instead of another image of DevOps, I have listed some of the important technology and process. This can determine answers to the questions asked in the last paragraph.
Infrastructure as code
Infrastructure — You may use public cloud or private cloud. Working with virtual resources on a public cloud involves a vast number of services. These services are continuously changing and improved. There are many aspects of the service beyond technology, such as costs, different geographies, redundancy, performance. This is probably the most complex part of DevOps technology.
Provisioning — There are different technologies for provisioning such as CloudFormation, Chef or Vagrant.
Source code control
Source code control is one of the more robust technologies in DevOps. However, you still might have weak links. You may have to think about security of the repositories. You have to think about managing access. You may need to control onboarding and offboarding. What if you need to give access to the repositories to a customer? What if a third party needs to collaborate with you? Are the repositories audit ready? What if the organization name changes?
CI/CD, Automation server
An automation server like Jenkins is a critical part of continuous integration and continuous delivery. Jenkins (and alternatives) has more moving parts than Git, but much less than a cloud platform. Jenkins plugins might get messy. Plugins may have security vulnerabilities. Jenkins jobs allow for flexibility which may not be maintainable. When creating jobs, engineers may not plan for failure.
Monitoring
The naïve view of monitoring is that it is deploying monitoring tools. Monitoring depends on what you want to monitor, the application, how a team will respond and much more. Monitoring involves a lot of knowledge of diverse technology, including both backend web technology, front end as well as application knowledge. There are many moving parts, including the people involved. You may not know what you are doing wrong, until there is a problem. Problems are few and far between. That allows a false sense of indifference.
ITIL
You could cursorily wave off ITIL. However, ITIL is very much a part of many enterprises. If you are interested in process and want to comment on the DevOps loop, you need to take ITIL into account.
ITIL service management is an important part of responding to operations problems.
Enterprise software — infrastructure and applications
An important part of DevOps which is often missed, is the deployment and management of enterprise software. Enterprise software may be related to infrastructure or applications. Infrastructure software could include the use of a firewall or other security software. It might include the use of Splunk. Applications may be a CMS or ERP — applications like Adobe AEM, Pega, SAP, Jira. The challenge in all these cases is deploying a large application without having detailed knowledge of how the application works.
Enterprise software vs. software products
Any introductory discussion of DevOps starts with a description of Netflix deploying their software multiple times a day (hour?). Software product companies like Netflix have a very focused mission. Engineers can stay in the same role for years. Netflix has deep pockets and can attract and retain top quality talent. Operations engineers in Netflix have strong development skills and can develop their own tools.
The dynamics are very different in enterprises like banks, telecom, government or service providers. In enterprises you may be working with multiple vendors, including some with remote teams. Vendors may work on short term projects. Engineers may have to work with many different software, depending on the organization’s needs. Organizations may not be able to attract top talent. They also may not be able to retain their engineers like Netflix.
Most of the discussion on DevOps is from the point of view of businesses like Netflix. As an overall theory/approach of software development, the “Netflix approach”, may not be applicable to a large number (majority?) of software projects, in teams which are smaller or which are part of enterprises.
Ground Reality of Testing in DevOps
Since Snowbird, agile developers have made few inroads into testing. Testing for developers is still a fuzzy mix of pull requests, linting, unit tests, test automation, TDD (rarely), BDD (done by someone else). DevOps and operations/infrastructure engineers are yet another step removed from software testing.
There are interesting, powerful ideas related to testing, like monitoring, observability, testing in production, chaosmonkey. All of these are very technical and can keep teams busy for a long time. None of these ideas are a substitute for critical thinking about risk, whether applications or infrastructure.
Testing in DevOps, or continuous testing, really means test automation. Test automation, API tests and unit tests can be run on every code check-in. They are also run periodically, e.g., nightly and before releases. A good understanding of testing is mostly optional. Everyone will agree, even insist that testing is nice to have or essential. On the other hand, testers will be hard pressed to make an impact on the operations (Ops) part of DevOps, given their limited understanding of technology related to DevOps or their motivation to understand testing.
I’d still like to think that an Ernest Mueller or a Dan Ashby or a Lisa Crispin as part of a whole team can ask questions to test DevOps. Here’s hoping you can too. You can also apply the ‘Mary had a little lamb’ heuristic to “testing in DevOps”, with your unique outlook.
Notes
- My previous blog post on how to understand testing, if you are in Ops/DevOps.
- Testing is anticipating problems — how do you do that in DevOps?
- What kind of defects do you find in DevOps?
- I took one aspect of DevOps, monitoring, and explained what it means to test it.
- This is a simple example of testing a calculator. There is no terminology or jargon or explicit process.
