Testing

When testing, we execute a set of test cases. A test case specifies how to perform a test. At a minimum, it specifies the input to the software under test (SUT) and the expected behavior.

Test cases can be determined based on the specification, reviewing similar existing systems, or comparing to the past behavior of the SUT.

For each test case we do the following:

1. Feed the input to the SUT
2. Observe the actual output
3. Compare actual output with the expected output

A test case failure is a mismatch between the expected behavior and the actual behavior. A failure indicates a potential defect (or a bug), unless the error is in the test case itself.

In OOP code, it is common to write one or more unit tests for each public method of a class.

class Foo{
    String read(){
        //...
    }
    
    void write(String input){
        //...
    }
    
}

class FooTest{
    
    @Test
    void read(){
        //a unit test for Foo#read() method
    }
    
    @Test
    void write_emptyInput_exceptionThrown(){
        //a unit tests for Foo#write(String) method
    }  
    
    @Test
    void write_normalInput_writtenCorrectly(){
        //another unit tests for Foo#write(String) method
    }
}

import unittest

class Foo:
  def read(self):
      # ...
  
  def write(self, input):
      # ...


class FooTest(unittest.TestCase):
  
  def test_read(sefl):
      # a unit test for read() method
  
  def test_write_emptyIntput_ignored(self):
      # a unit tests for write(string) method
  
  def test_write_normalInput_writtenCorrectly(self):
      # another unit tests for write(string) method

Integration testing : testing whether different parts of the software work together (i.e. integrates) as expected. Integration tests aim to discover bugs in the 'glue code' related to how components interact with each other. These bugs are often the result of misunderstanding of what the parts are supposed to do vs what the parts are actually doing.

System testing is typically done by a testing team (also called a QA team).

System test cases are based on the specified external behavior of the system. Sometimes, system tests go beyond the bounds defined in the specification. This is useful when testing that the system fails 'gracefully' having pushed beyond its limits.

Test case: load a web page that is too big
* Input: load a web page containing more than 5000 characters. 
* Expected behavior: abort the loading of the page and show a meaningful error message. 

System testing includes testing against non-functional requirements too. Here are some examples.

• Performance testing – to ensure the system responds quickly.
• Load testing (also called stress testing or scalability testing) – to ensure the system can work under heavy load.
• Security testing – to test how secure the system is.
• Compatibility testing, interoperability testing – to check whether the system can work with other systems.
• Usability testing – to test how easy it is to use the system.
• Portability testing – to test whether the system works on different platforms.

Alpha testing is performed by the users, under controlled conditions set by the software development team.

Beta testing is performed by a selected subset of target users of the system in their natural work setting.

An open beta release is the release of not-yet-production-quality-but-almost-there software to the general population. For example, Google’s Gmail was in 'beta' for many years before the label was finally removed.

Developer testing is the testing done by the developers themselves as opposed to professional testers or end-users.

Delaying testing until the full product is complete has a number of disadvantages:

• Locating the cause of such a test case failure is difficult due to a large search space; in a large system, the search space could be millions of lines of code, written by hundreds of developers! The failure may also be due to multiple inter-related bugs.
• Fixing a bug found during such testing could result in major rework, especially if the bug originated during the design or during requirements specification i.e. a faulty design or faulty requirements.
• One bug might 'hide' other bugs, which could emerge only after the first bug is fixed.
• The delivery may have to be delayed if too many bugs were found during testing.

Therefore, it is better to do early testing, as hinted by the popular rule of thumb given below, also illustrated by the graph below it.

The earlier a bug is found, the easier and cheaper to have it fixed.

Such early testing of partially developed software is usually, and by necessity, done by the developers themselves i.e. developer testing.

Here are two alternative approaches to testing a software: Scripted testing and Exploratory testing

1. Scripted testing: First write a set of test cases based on the expected behavior of the SUT, and then perform testing based on that set of test cases.

2. Exploratory testing: Devise test cases on-the-fly, creating new test cases based on the results of the past test cases.

Exploratory testing is ‘the simultaneous learning, test design, and test execution’ [source: bach-et-explained] whereby the nature of the follow-up test case is decided based on the behavior of the previous test cases. In other words, running the system and trying out various operations. It is called exploratory testing because testing is driven by observations during testing. Exploratory testing usually starts with areas identified as error-prone, based on the tester’s past experience with similar systems. One tends to conduct more tests for those operations where more faults are found.

Which approach is better – scripted or exploratory? A mix is better.

The success of exploratory testing depends on the tester’s prior experience and intuition. Exploratory testing should be done by experienced testers, using a clear strategy/plan/framework. Ad-hoc exploratory testing by unskilled or inexperienced testers without a clear strategy is not recommended for real-world non-trivial systems. While exploratory testing may allow us to detect some problems in a relatively short time, it is not prudent to use exploratory testing as the sole means of testing a critical system.

Scripted testing is more systematic, and hence, likely to discover more bugs given sufficient time, while exploratory testing would aid in quick error discovery, especially if the tester has a lot of experience in testing similar systems.

In some contexts, you will achieve your testing mission better through a more scripted approach; in other contexts, your mission will benefit more from the ability to create and improve tests as you execute them. I find that most situations benefit from a mix of scripted and exploratory approaches. --[source: bach-et-explained]

Acceptance tests give an assurance to the customer that the system does what it is intended to do. Acceptance test cases are often defined at the beginning of the project, usually based on the use case specification. Successful completion of UAT is often a prerequisite to the project sign-off.

Acceptance testing comes after system testing. Similar to system testing, acceptance testing involves testing the whole system.

Some differences between system testing and acceptance testing:

Note: negative test cases: cases where the SUT is not expected to work normally e.g. incorrect inputs; positive test cases: cases where the SUT is expected to work normally

Passing system tests does not necessarily mean passing acceptance testing. Some examples:

• The system might work on the testbed environments but might not work the same way in the deployment environment, due to subtle differences between the two environments.
• The system might conform to the system specification but could fail to solve the problem it was supposed to solve for the user, due to flaws in the system design.

When we modify a system, the modification may result in some unintended and undesirable effects on the system. Such an effect is called a regression.

Regression testing is the re-testing of the software to detect regressions. Note that to detect regressions, we need to retest all related components, even if they had been tested before.

Regression testing is more effective when it is done frequently, after each small change. However, doing so can be prohibitively expensive if testing is done manually. Hence, regression testing is more practical when it is automated.

A simple way to semi-automate testing of a CLI(Command Line Interface) app is by using input/output re-direction.

• First, we feed the app with a sequence of test inputs that is stored in a file while redirecting the output to another file.
• Next, we compare the actual output file with another file containing the expected output.

Let us assume we are testing a CLI app called AddressBook. Here are the detailed steps:

1. Store the test input in the text file input.txt.

   add Valid Name p/12345 valid@email.butNoPrefix
   add Valid Name 12345 e/valid@email.butPhonePrefixMissing
   

2. Store the output we expect from the SUT in another text file expected.txt.

   Command: || [add Valid Name p/12345 valid@email.butNoPrefix]
   Invalid command format: add 
   
   Command: || [add Valid Name 12345 e/valid@email.butPhonePrefixMissing]
   Invalid command format: add 
   

3. Run the program as given below, which will redirect the text in input.txt as the input to AddressBook and similarly, will redirect the output of AddressBook to a text file output.txt. Note that this does not require any code changes to AddressBook.

   java AddressBook < input.txt > output.txt
   

   • The way to run a CLI program differs based on the language.
     e.g., In Python, assuming the code is in AddressBook.py file, use the command
     python AddressBook.py < input.txt > output.txt

   • If you are using Windows, use a normal command window to run the app, not a Power Shell window.

   More on the > operator and the < operator extra

   A CLI program takes input from the keyboard and outputs to the console. That is because those two are default input and output streams, respectively. But you can change that behavior using < and > operators. For example, if you run AddressBook in a command window, the output will be shown in the console, but if you run it like this,

   java AddressBook > output.txt 
   

   the Operating System then creates a file output.txt and stores the output in that file instead of displaying it in the console. No file I/O coding is required. Similarly, adding < input.txt (or any other filename) makes the OS redirect the contents of the file as input to the program, as if the user typed the content of the file one line at a time.

   Resources:

   • Using command redirection operators in Windows

4. Next, we compare output.txt with the expected.txt. This can be done using a utility such as Windows FC (i.e. File Compare) command, Unix diff command, or a GUI tool such as WinMerge.

   FC output.txt expected.txt
   

Note that the above technique is only suitable when testing CLI apps, and only if the exact output can be predetermined. If the output varies from one run to the other (e.g. it contains a time stamp), this technique will not work. In those cases we need more sophisticated ways of automating tests.

A test driver is the code that ‘drives’ the SUT for the purpose of testing i.e. invoking the SUT with test inputs and verifying the behavior is as expected.

public class PayrollTestDriver {
    public static void main(String[] args) throws Exception {

        //test setup
        Payroll p = new Payroll();

        //test case 1
        p.setEmployees(new String[]{"E001", "E002"});
        // automatically verify the response
        if (p.totalSalary() != 6400) {
            throw new Error("case 1 failed ");
        }

        //test case 2
        p.setEmployees(new String[]{"E001"});
        if (p.totalSalary() != 2300) {
            throw new Error("case 2 failed ");
        }

        //more tests...

        System.out.println("All tests passed");
    }
}

JUnit is a tool for automated testing of Java programs. Similar tools are available for other languages and for automating different types of testing.

@Test
public void testTotalSalary(){
    Payroll p = new Payroll();

    //test case 1
    p.setEmployees(new String[]{"E001", "E002"});
    assertEquals(6400, p.totalSalary());

    //test case 2
    p.setEmployees(new String[]{"E001"});
    assertEquals(2300, p.totalSalary());

    //more tests...
}

Most modern IDEs has integrated support for testing tools. The figure below shows the JUnit output when running some JUnit tests using the Eclipse IDE.