Testing is an integral part of any successful software project. In software testing, there are two main areas to complete the verification of a product — automated testing and manual testing.
Manual software testing, as the name implies, is executed by a person without the help of any tool or script. Automated software testing, on the other hand, is executed with the assistance of tools, scripts, and software.
Nowadays, there's a question that is often heard around software companies; a question that's in every manager's head around the globe; a question that — if answered incorrectly — may end up hurting the quality of our products. This question is as follows: “Should we manually test applications or should we implement automated tests?”
Almost all development companies get to a point where they have to decide on a test strategy. Do they go with pure BDD/TDD (behavior-driven development / test-driven development) which leads to one or no tester? Or do they hire/keep manual testers to perform the product verification? There's a reason this discussion keeps being brought up — the need for “cutting costs.” From experience, when making decisions about budgeting, the area that takes the heavy hit is usually testing. Let's face it; it's not cheap to maintain a lot of resources whose sole function is to test applications.
Companies can get into a real dilemma trying to answer this question, but is there really a rivalry between these two testing techniques?
Clarifying the difference
When comparing these two areas, we already know that automation testing has several benefits such as cost reduction, reusability, speed, and reliability. For those projects that do not have complex business rules, this technique helps reduce unnecessary manual effort and allows us to complete testing more quickly in the SDLC (software development life cycle) thanks to BDD/TDD.
The need to get applications to the market in a shorter timeframe has caused many companies to pull resources away from QA (quality assurance) staff on the assumption that automation is suitable for all testing needs, but is this really a good decision?
Even though BDD-TDD techniques have made significant advances, there's no reason to think that manual testing cannot live alongside automated testing. In fact, both testing techniques are 100% necessary and can complement each other in multiple ways. Since no testing method is perfect, by using them together, one approach can identify errors that the other technique is prone to miss.
We have to remember that automated testing was originally designed to reduce costs on repetitive test cases that needed to be run many times — for regression testing and for building up long term product quality. As we keep adding automated tests, the automation suite grows more robust and it takes on the responsibility of running test cases that can cause boredom in human testers and errors in execution.
There's another way of thinking about automated tests and about how we execute these test cases. We can consider them to be “confirmation tests,” because if the first execution doesn't find a bug and we keep executing through the cycles, we probably won't find any new bugs. The next execution would only be to confirm that no new bugs are found. These automated test cases confirm that an application is working as expected when it was correctly built in a continuous integration environment. The problem here is that the tests may start losing quality value because they stop providing meaningful data. This is particularly true if the reporting tool is not implemented along with the test, in which case it will only confirm code for how that particular feature (or an old feature) works. To avoid losing quality value, we need a resource to look after the whole automated suite — a resource with sufficient quality knowledge and project experience to track application changes with the support of a tool.
This is one of the main reasons we need people to collaborate on testing our products — to attempt to “destroy” the application, to seek out errors, and to think of all the possible ways to wreck digital security, databases, and services. Test cases to complete these actions could definitely be performed by automation, but — speaking from personal experience — the cost of writing those scripts isn't worth it for one fundamental reason: It might only be run once!
When we're planning our project testing, we need to answer some basic questions in order to decide the correct mix of both areas/techniques:
- Project time frame — how much time do we have?
- Core business — how critical is our software to the business running it?
- Who is the targeted audience?
- Costs — are you running on non-freeware technologies and need to consider the cost of licensing?
- How subjective is the application. Are the business rules the application serves too complex?
Considering all the variables, the choice between manual and automated tests usually comes down to a few key issues as follows:
- Number of Test Cases: If the test case execution occurs a small number of times, manual testing would be better. For example, projects with a static page, with no database connection, and few elements interacting with the page.
- Number of Sub-Projects: If the project is comprised of small projects of a similar type, automation can take care of the features they have in common and you can run a manual test after or before the automated test scenarios to complete the verification.
- Time Constraints: Exploratory testing (manually-executed) + automated test are always a good combination for projects where we don't have much time or where requirement specification is poorly written. This type of testing technique requires experienced testers, with creativity and intuition. Automation comes in when we find the spare time to create regression tests for pre-existing features we need to keep an eye on.
In order to answer these three questions and to help find the right mix of tests, we need to have a good vantage on upcoming projects and to consider the pros and cons of both systems. It's impossible to create automated tests for everything and — as always — a minimum amount of tests should be executed for any product.
Pros and cons
Finally, we have to keep in mind that — since professional testers and automation tools aren't perfect — bugs can still end up in the final product. It's almost inevitable that no matter how big or small your project, these techniques still need to be matched, considered, and sensibly studied when making testing decisions. The right choice can save you lots of time and offer improved results, minimizing the bugs found in production and giving your application more chances to succeed.