In the age of StackOverflow, ChatGPT etc. the casual software developer can be overcome by the feeling that in a few years time, software development will be mainly done by computers and the only people needed in the world of programming will be the experts developing the tools that to software development themselves. Regardless of whether the industry will be taken over by AI software developers, there’s one important step that will always require human supervision: testing the software.

Let’s suppose you found a perfect AI tool that spits out working code for every task you ask it to solve (yes, that’s not really possible, but let’s suppose it is).

Now, would you ask your AI companion to write an autopilot software for a passanger plane, then upload it to the flight computer and board the plane to fly? If you would, don’t continue reading and *please don’t become an aerospace engineer.*

Software testing has always been an important task in the software development lifecycle. As more and more code is AI generated, testing will probably become even more important. For this task, a number of tools have been designed in the last decades, e.g. for

  • Python:
    • unittest: Python’s built-in testing framework in xUnit style and mimicks Java’s JUnit
    • pytest: Third-party testing framework. Simpler and more feature-rich (test fixtures, parameterized testing etc.) than unittest
  • C++:
    • Google Test: Google’s open-source testing framework for C++. It offers support for testing with mock objects using Google Mock
    • cppunit: Java’s JUnit ported to C++, open-source
    • Cantata++: Commercial testing framework offering a wide range of features including test automation and compliance with safety standards

These tools, deisgned to run tests are called test runners and allow for convenient design, execution and evaluation of tests. In this post, we’ll be focusing on Google Test (GTest) for C++ and give an example with pytest for Python for comparison.

Google Test

Tests are organized in test suites that are collections of related individual test cases gathered in groups. Running these test suites is simple and convenient.

A few guidelines for good test cases are gathered below.

  • test cases should be organized (in test suites)
  • test cases should be repeatable (their outcome shouldn’t change when re-run with the same input)
  • test cases should be independent from each other (the output of one should be independent of the outcomes of the others)
  • preferably, tests should be fast
  • failed tests should provide as much information as possible about the failure

Google Test provides two distinct basic ways for tests to indicate something’s wrong:

  • assertion: test execution stops if the outcome isn’t as expected (for fatal failures)
  • expectation: test execution continues if the outcome isn’t as expected (for non-fatal failures)

Let’s look at some examples. Let’s suppose we’ve written a (arguably really silly) function that calculates exponents of e:

File: exponential.h

#ifndef EXPONENTIAL_H
#define EXPONENTIAL_H

double exponential(double x);

#endif // EXPONENTIAL_H

File: exponential.cpp

#include "exponential.h"
#include <cmath>

double exponential(double x) {
    return std::exp(x);
}

For this simple program, we can write the following tests:

File: exponential_test.cpp

#include <gtest/gtest.h>
#include "exponential.h"

// Fixture class for Exponential function
class ExponentialTest : public ::testing::Test {
protected:
    void SetUp() override {
        // Code here will be called immediately after the constructor (right before each test)
    }

    void TearDown() override {
        // Code here will be called immediately after each test (right before the destructor)
    }
};

// Test functions

int main(int argc, char **argv) {
    ::testing::InitGoogleTest(&argc, argv);
    return RUN_ALL_TESTS();
}

Now, we can replace the placeholder comment with actual test functions. Let’s start by checking if positive exponents are handled correctly:

TEST_F(ExponentialTest, HandlesPositiveInput) {
    EXPECT_DOUBLE_EQ(exponential(1.0), std::exp(1.0));
    EXPECT_DOUBLE_EQ(exponential(2.0), std::exp(2.0));
    EXPECT_DOUBLE_EQ(exponential(10.0), std::exp(10.0));
}

Then we check the 0 case:

TEST_F(ExponentialTest, HandlesZeroInput) {
    EXPECT_DOUBLE_EQ(exponential(0.0), 1.0);
}

Next, we check negative exponents:

TEST_F(ExponentialTest, HandlesNegativeInput) {
    EXPECT_DOUBLE_EQ(exponential(-1.0), std::exp(-1.0));
    EXPECT_DOUBLE_EQ(exponential(-2.0), std::exp(-2.0));
    EXPECT_DOUBLE_EQ(exponential(-10.0), std::exp(-10.0));
}

Lastly, we check fractional exponents:

TEST_F(ExponentialTest, HandlesFractionalInput) {
    EXPECT_DOUBLE_EQ(exponential(0.5), std::exp(0.5));
    EXPECT_DOUBLE_EQ(exponential(-0.5), std::exp(-0.5));
    EXPECT_DOUBLE_EQ(exponential(1.5), std::exp(1.5));
}

We can now compile and run the above function and test files with

g++ -std=c++11 -isystem /usr/local/include -pthread exponential.cpp test_exponential.cpp /usr/local/lib/libgtest.a /usr/local/lib/libgtest_main.a -o test
./test

if using the standard installation paths. If all goes well, we now see the output

[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from ExponentialTest
[ RUN      ] ExponentialTest.HandlesZeroInput
[       OK ] ExponentialTest.HandlesZeroInput (0 ms)
[ RUN      ] ExponentialTest.HandlesPositiveInput
[       OK ] ExponentialTest.HandlesPositiveInput (0 ms)
[ RUN      ] ExponentialTest.HandlesNegativeInput
[       OK ] ExponentialTest.HandlesNegativeInput (0 ms)
[ RUN      ] ExponentialTest.HandlesFractionalInput
[       OK ] ExponentialTest.HandlesFractionalInput (0 ms)
[----------] 4 tests from ExponentialTest (0 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (0 ms total)
[  PASSED  ] 4 tests.

which indicates that all our test ran and passed. Great, our oneliner function is now tested by multiple functions!

pytest

The process is fairly similar in Python. Take a look:

File: exponential.py

import math

def exponential(x):
    return math.exp(x)

Pretty simple, huh? Now, let’s define our tests:

# test_exponential.py
import pytest
from exponential import exponential

def test_handles_zero_input():
    assert exponential(0.0) == 1.0

def test_handles_positive_input():
    assert exponential(1.0) == math.exp(1.0)
    assert exponential(2.0) == math.exp(2.0)
    assert exponential(10.0) == math.exp(10.0)

def test_handles_negative_input():
    assert exponential(-1.0) == math.exp(-1.0)
    assert exponential(-2.0) == math.exp(-2.0)
    assert exponential(-10.0) == math.exp(-10.0)

def test_handles_fractional_input():
    assert exponential(0.5) == math.exp(0.5)
    assert exponential(-0.5) == math.exp(-0.5)
    assert exponential(1.5) == math.exp(1.5)

We can now run the tests by simply executing

pytest

If all went well, we see the results of our tests:

============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-6.2.5, pluggy-1.0.0
rootdir: /path/to/your/tests
collected 4 items

test_exponential.py ....                                                [100%]

============================== 4 passed in 0.01s ===============================

That’s it - now no more excuses for not testing your code thoroughly!