In 2016, Dan Ashby wrote a blog post, ‘Continuous Testing in DevOps’. The post included a hand drawn sketch of the DevOps loop and a modification on how to include testing. Dan explained that this meant testing in all parts of the DevOps model. Since then, that image has been quoted in every discussion of “testing DevOps”, “continuous testing” and more.
In the book, Exploring Requirements, Weinberg and Gause, describe the ‘Mary had a little lamb’ heuristic. They advocate using the heuristic for ‘identifying problem-statement ambiguity’. The statement ‘Mary had a little lamb’ can be interpreted in many ways. This…
Monitoring has been a very important concept in network or infrastructure operations for a long time. It has got much more attention in DevOps as part of the feedback loop on operations.
I happened to sit in on a few DevOps/Ops engineers, talking about monitoring, to a junior engineer, among others. My focus was on testing.
How do you test monitoring? What does that even mean?!!
If you are new to monitoring, you may realise there are tools available for monitoring, like Newrelic. Can we just deploy Newrelic for monitoring and be done with it? When deploying any software, it’s…
Agile software development radically changed the nature of software creation. One of the hallmarks of agile is the creation of small increments of end user functionality as opposed to creating software in one or few large portions. Agile also included many other innovations such as starting with confirmatory tests as opposed to creating tests after construction and creating self managed teams as opposed to traditional command and control. There was an earlier, parallel movement in understanding the design and creation of mechanical objects, including building architecture, product design and graphic communication. One of the major contributors to that movement was…
Good ideas in software testing endure, despite changes in technology and development approaches. In 1988, Cem Kaner first described a ‘defect catalog’ for software in his landmark book, Testing Computer Software. In this blog post I describe how to create a defect catalog for Infrastructure as Code as part of DevOps.
Kaner’s defect catalog was a list of common errors in software. He listed the different ways you can use such a list:
- Evaluate test materials developed for you by someone else.
- Developing your own tests.
- Generate hypotheses for bugs which are difficult to reproduce.
- Generate related ideas for unexpected…
There are different interpretations of the term, ‘DevOps’. However, in the DevOps values — CAMS (Culture, Automation, Measurement, Sharing), culture is considered the most important value.
The methodology begins by looking for the right people or person to fulfill a needed role. Secondly, ensure that role’s process is correct and optimal. Lastly, improve the process’ efficiency by reducing steps through some tool based automation. Notice, tools and automation are last!
People Over Process over Tools, Alex Honor.
Despite the focus on culture, collaboration and organization change, a critical part of DevOps is the enabling technology. There are different aspects of…
If you are an operations or infrastructure type engineer, testing may mean traditional “QA”, UAT or the newer test automation. An alternate way to think about tests is that you are asking a question about the software or infrastructure. The question is open ended — there is no definitive answer. What will happen when the instance runs out of disk space? It will trigger an alert. Will it? (Why did the bedroom alarm clock not ring on the day of your important meeting?) If the alert is triggered, will the people who are paged know how to respond? If we…
When testing API you use tools like Postman or write tests in Java, Python or another language. Reviewing defects reported by users, reveals critical techniques that you can use to test API. In this blog post, I review defects in a Python package to access Twitter, to get insights on how to test the API. There are useful lessons for testing API in general.
Twitter has it’s own Python package for accessing the API. The package I review here is not the official Twitter Python package.
When reviewing defects, I want to explicitly avoid ‘fixing’ the problem. When someone or…
What does it mean to test AI? If you look at the spam detection feature in your email, it sometimes works, less often doesn’t. Even if you flag items as not spam, some again show up in spam after some time. You could categorize that as a machine learning FAIL?
Amazon recommends books related to your purchase. Some of them make sense. Others don’t. Hopefully the algorithm learns and gets better over time. Are the incorrect recommendations a failure of the algorithm? Isn’t it expected that recommendations will learn and improve over time? So, failures are in fact not failures?
…
Initial state of the system
Software Test Automation
When software testers first encounter agile, they can be intimidated by all the terminology and technology. A better way to understand testing is to answer a few fundamental questions. It’s a good idea to not accept any answers, even when they are widely adopted, at face value. You should try to find the answers for yourself and keep checking if there are alternative explanations.
Any discussion on testing in agile teams is dominated by test automation. The rationale for test automation is that it allows you to release software often, often many times a day. If you parse that sentence…
Nilanjan
Software testing, project management, managing testers
_(2).jpg,){kind=link}