QC I: Test Environment

Historically, test framework or rather in simpler terms, test code is often used to check if an implemented software component works. However, there is not much considerable effort put into designing a test framework, when one is designing a module. From a Unit-Testing (UT) perspective, a module developers feels that a no-frills simple-code would suffice. Though being a big fan of simple coding practices, I do feel that some amount of thought should be put into designing a test framework irrespective of the level of the component under consideration.

In a typical embedded systems stack, multiple hierarchies or levels of software abstraction would be present. Everyone of these layers abstracts an underlying layer and provides an extended functionality to the higher layer. Consider a typical stack which has 5 different layers from the lowest implementation layer to the final API (Application Programmer Interface). One of the primary objectives is to develop a good test-framework based off the API documentation of every logical layer.

For the overall product verification, it is imperative that each of the logical separations in software is independently verified. Incremental software quality assurance from layered perspective helps to isolate multitude of issues arising due to integration of new layers on top of existing stack. Hence, a high quality test framework at each of these layers is order of the day which ensures the integrity and quality of the underlying software.

Hence, from these perspectives, I have captured some lessons which can serve as a reference point for developing high quality software.

Lesson 1: Maintain all test frameworks throughout the lifecycle of the software product

In typical software development, module level test frameworks often get neglected with passage of time. Once integration tests start working fine, more often than not, module level test code is not maintained (including build issues). One of the most common reasons quoted for this would be priority i.e. maintaining test code is not of high importance to the project.

However, the value of these test frameworks is often appreciated whenever there is an error reported during maintenance phase of the project. Debugging an issue at the overall system level would often be very tricky, cumbersome and plain frustrating. To isolate the issue, one would work their way up from lower levels of software abstraction to higher levels. During this process, availability of a good test code is very essential and if a project has maintained the same, it will definitely benefit from faster debugging and resolution of customer issue and definitely gain end-customer satisfaction.

If such a test environment is not available, the debugging process will tail-spin into multiple experiments which will enhance the existing pressure and can be felt on the individuals under the scanner. This is definitely not a good situation to be in. Hence, as a project, one should definitely consider some effort in maintaining the test frameworks as part of their software development strategy.

Tip: A well documented test framework is like an icing on the cake and is always worth its weight in gold.

Lesson 2: Automation is order of the day

Build once, Run Many times. This should be the mantra of a test-framework developer. If a test-framework can’t be automated, it is of very little use. Automation is definitely order of the modern-day software development, given the fast pace of development and customer delivery.

The need of automation can be well understood from the amount of time, it takes to build a system for testing. Consider that one is developing a component X which has the following time requirements for different phases of execution:

Compile and Build:                                            10 minutes
Flash onto the device:                                       5 minutes
Run the test and document the result:             2 minutes

which comes to a grand-total of 17 minutes. If the test-framework or test-code is not automated i.e. a fresh compile-build-flash-test is required for every test, the average test time for every test-case is 17 minutes per test. If the overall number of test-cases to be run is 100, then we are looking at 1700 minutes or roughly 3.5 man days (1 man day = 8 hours) of testing, which is a waste of precious developer time or even the tester’s time.

If the test framework is automated i.e. compile-build  and flash is required once, then to run to 100 testcases, one would require 15+200 minutes which is roughly 3.5 hours.  Amazingly, the test time has been reduced to 1/7 with a little bit of foresight, design and implementation. This precious time is very valuable under high pressure scenarios and faster test-beds aid in improving the quality of the product to a large extent.

Tip: Judicious choice of extent of automation is extremely important. Automation should enable faster test-cycles without necessarily introducing additional complexities.

Lesson 3: Command Line Interface

To automate a test-framework, one would implement command-line or built-in automation methodologies. Consider the following test framework implementations for a component X.

Implementation 1:

void testcode()
       //do_test_case_4(); /* Note the commented out statement */
       //do_test_case_5(); /* Note the commented out statement */


  • Does implement the build-one, run always strategy
  • Requires 1 build + 1 flash + 1 run – Easier to automate without the need of specific scripting

Potential Drawbacks:

  • Enabling/Disabling one set of test-suites  or Adding a new test-case requires the entire cycle to be rerun – Large turn-around time
  • Parameters, paths etc are hard-coded into the code – Hampers flexibility and doesn’t augur very well for location-independent processing
  • With larger number of parameters, the number of tests will increase exponentially and code will be bulky, cumbersome, difficult to read and maintain
  • As parameters are hard-coded, very little scope for negative testing and one needs to have knowledge of internals of code to introduce a new test-case.

Implementation 2:

void testcode()
        fptr = fopen("param.txt", "rb");
        get_params_from_param_file(fptr, &params);


  • Truly implements build-one, run forever principle
  • Test framework can easily be developed from the API document following the syntax of the parameter file
  • Introduction of newer test-cases requires only an update of the parameter file and a fresh run – Extremely Time Efficient
  • Handling higher number of parameters will be easy as param file can be viewed on a text editor

Potential Drawbacks:

  • Path of the parameter file is hard-coded – Dependency should be captured in documentation and this scenario should be handled in the test code

Implementation 3:

void testcode(int argc, char* argv[])
        get_params_from_command_line(argc, argv, &params);


  • Yet another implementation of build-one, run forever methodology – Parameters are passed through Command line interface
  • Multiple test-cases can be scripted and adding a new test-case is another addition to the existing script

Potential Drawbacks:

  • One time effort to generate the script is required
  • When higher number of parameters are present, extreme care has to be exercised to ensure that right strings are passed – This can be automated using a script and can easily be overcome.

From the different implementations captured above, one can observe that there is NO one unified way of achieving the goal. From an ease-of-implementation, extensibility and reuse perspective, Methods 2 and 3 are better suited. A blend of Implementations 2 and 3 can be utilized to develop a pretty good test framework.

To conclude, the value of a test code/framework is immense and hence, should be given adequate focus during the course of software development. A good solid test framework automatically translates into a good piece of code (component) and hence, high quality software which one can be very proud of.

Postscript: I have been very impressed by the amount of test-code introduced into Android. Though going through only one specific dimension of this revolutionary software, one can truly appreciate the need and subsequent value-add of different test-frameworks. Way to go!!!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s