Untested code is defect code
You might have heard of the saying, “if you’re not backing your data up, it needs to be considered deleted data”. Similarly, if your code isn’t covered by automatic tests , you might as well get ready to consider it as defect code.
The emphasis here is on automatic tests.
I am always shocked when I see organizations that still think manually testing software from a fixed test specification is a good idea.
If you have the time to specify your tests on a piece of paper (or in a PDF) including the manual steps to prepare and clean up the test environment then you definitely have the time to write code that tests your implementation automatically.
Manual testing doesn’t scale
At the time I am writing this article the Exasol SQL Statement Builder for Java — a project that is still in its infancy — already has 172 unit tests, which are executing in less half a second.
I run those tests after each small change I make. And why shouldn’t I? Half a second doesn’t slow me down in any way. This way I know that not only the new features I write are working but also that I didn’t break any of the existing implementation. Of course, you have probably already heard of this concept called regression testing.
Now imagine what would happen if you tried to run all those tests by hand. Let’s for the sake of the argument assume that you’re an insanely fast and focused tester with superhuman bladder control and are able to execute an average of one test every two minutes. Executing all tests would still take you almost 6 hours.
Of course, no one is that fast. Realistically, we’re talking about at least two full, two person workdays. This is 100,000 times slower than the automatic tests. And in addition to your developers and their machines, you now have to spend your money on a large number of testers and their equipment. Plus, rent for the extra space and electricity.
And if the tests aren’t executed regularly and in large frequency, you’ll be blocking the implementation team. So to keep even a team of ten implementors busy, you would need a giant pool of testers to work in parallel. But that increases the communication overhead to a point where you spend all your time on communication. Frederick Brooks mathematically proved this in his book The mythical man month.
Manual testing simply doesn’t scale. You will inevitably reach the point where people stop regression testing and only test new features. A while later, it will end up as a subset of the new features only. By this time your code will be so hopelessly broken, that you might as well start over.
Don’t force people to do a machine’s job
And we didn’t even touch on the fact that people make mistakes, especially when working on repetitive and deadly boring tasks like executing manual tests according to a playbook. Machines on the other hand don’t mind. They can repeat the same test a gazillion of times without ever getting tired or distracted.
Bottom line: don’t force a person to do a machine’s job. It is inefficient and cruel.
No excuses
With some tests, you need to access the unit you’re testing through physical interfaces. For example, at some point when a car is built, you need to test whether moving the indicator lever causes, well, the indicator lights to blink.
If you’re now thinking, “that’s obviously a manual test”- think again. In most cases you can still automate the tests. You just need additional hardware. Sure, a robot arm operating the lever might sound like overkill in the beginning. But if your production line produces a thousand vehicles each day, programming a robot to press buttons and move levers doesn’t sound so far fetched anymore, does it?
And it does not always have to be that expensive or complicated. For tests during development a simple remote controllable relay box can take over the job of the mechanical switches just fine.
Then what do I do with my test department?
So is that crazy Exasol person really suggesting, I should fire all my testers?
No.
I am making the case for tasking them with something that is worth the attention and ingenuity of a human being.
In my eyes the very best way to benefit from having dedicated testers is to let them engage in exploratory testing. In a nutshell exploratory testing is like taking a flashlight and digging through the deepest corners of your attic to find that wasps nest you’re sure must be hiding somewhere in there.
It’s intentionally giving the testers the freedom to tear apart your product and checking those corner cases that even the original developers didn’t think about.
Let’s break things
Let me tell you an anecdote. A friend of mine worked as a software quality engineer for a while. His job involved certification testing the ATMs produced by private companies. Those ATMs had to pass a rigorous testing regime that came in a well documented form. The tests were expensive and extensive – and the manufacturers came well prepared with a detailed spec for him.
But his first test was always to simply press all buttons randomly for half a minute at high speed. And the ATMs crashed. All of them. Invariably.
That was before automatic fuzzing became a thing. I am going to cover fuzzing in a later article. The lesson learned here is that he was a valuable tester not because he adhered to the test spec but because he ignored it for the most part. He chose to torture the unit under test to the best of his abilities. And with a devious creativity that only a human with a sense of humor can develop.
If you repeat the test by hand, you are doing it wrong.
Does this mean that you should pay people to always and forever randomly press buttons in your software?
Most certainly not. At this point we would be back to mindless, repetitive tasks and as we discussed before, this is a machines job. So after your clever tester managed to break your precious product by doings something previously unsuspected, automate what she did. Then run the test to make sure it fails, fix the implementation, run the test again — this time successfully — and move on.
The test will now be part of your ever growing regression test suit. If this particular bug ever rises its ugly head again, you’ll have a guardian in place that immediately warns you.
And once the machine takes over the repetitive testing, your tester will be free to figure out the next way to (ab)use the software in ways you never intended.
Crack your own software
If that testing concept sounds somehow familiar to you, you will realize the similarities between what your testers do and crackers. Crackers by the way is a more precise name for what the popular press means when they say “hackers”.
Someone will execute those tests outside of your test spec, and it better be people in your organization before black hats get their hands on your product. And if you’re creating safety or security relevant software you’ll probably also want to spend the extra buck and get an independent third party to review your testing. After all, we can all do with a fresh pair of eyes after a hard day coding.