Code Coverage Is Not Enough

Code Coverage has been an important quality metric to measure the robustness of the code for several years now. It is part of the release criteria for many organizations from development to quality assurance and, at times, even for the senior management team. In fact, it is not unusual that several teams strive for a higher code coverage metric anywhere from 60-80% as the bare minimum threshold for a release. With all that said it is important to understand that code coverage by itself is not as useful. For example, even if you had a very high code coverage of 80% and the majority of your core logic is in that other 20% of the code, chances are high the code could have some serious defects that may slip into the release. What is worse that even if the code coverage were 100% that does not mean all the functionality has been covered! There could be important functional scenarios that might have been missed in the production code and the unit tests, which may again lead to an incomplete release. Don’t get me wrong. Code Coverage is important, but not when it is looked in isolation. When combined with good functional unit tests, it can be extremely useful. And, that’s what we will cover in this article.

Let’s See Some Code

What would be a better way of showing other than using an actual code? So, let’s take a barebone Inventory Management example that offers the following basic functionality.
Note: The functionality and logic are intentionally kept simple so that we can focus on the main topic of this article.

Get product details.
Get the price of a product.
Get the price change of a product.

In order to support this functionality, we have 3 classes.

InventoryManagerDelegate: This contains the core business logic and will typically be used by another tier or layer, such as a REST API layer to offer the inventory management APIs.
InventoryDAO: The Data Access Object to handle all database interactions, such as adding and retrieving a product.
Product: The model or data object to store product information (a.k.a. POJO (Plain Old Java Object) in the Java lingo).

Here is the source code for these classes.

package com.cloudnineapps.samples;

/**
 * The inventory manager delegate.
 */
public class InventoryManagerDelegate {

  /** The DAO. */
  private InventoryDAO dao;
  
  
  /** Initializes the delegate. */
  public void init() throws Exception {
    dao = new InventoryDAO();
    dao.createSchema();
  }

  /**
   * Returns the details for the product with the supplied title.
   * 
   * @param title The product title.
   */
  public Product getProductDetails(String title) throws Exception {
    Product product = null;
    try {
      product = dao.getProduct(title);
    }
    catch(IllegalArgumentException ex) {
      // Product not found, handle gracefully
      product = new Product();
    }
    
    return product;
  }
  
  /**
   * Returns the price for the specified product.
   * 
   * @param title The product title.
   */
  public double getProductPrice(String title) throws Exception {
    // NOTE: This code does not gracefully handle the product not found scenario
    Product product = dao.getProduct(title);
    return product.getPrice();
  }
  
  /**
   * Returns the price change for the specified product from the current price. A negative value
   * indicates decrease in price and a positive value indicates an increase.
   * 
   * @param title The product title.
   * @param currentPrice The current price of the product.
   */
  public double getProductPriceChange(String title, double currentPrice) throws Exception {
    // NOTE: This code does not gracefully handle the product not found scenario
    double price = 0;
    try {
      Product product = dao.getProduct(title);
      price = product.getPrice();
    }
    catch(Exception ex) {
    }
    return (currentPrice - price);
  }
  
  /**
   * Adds a product with the supplied details.
   * 
   * @param id The product ID.
   * @param title The product title.
   * @param description The product description.
   * @param price The product price.
   */
  public void addProduct(int id, String title, String description, double price) throws Exception {
    dao.addProduct(id, title, description, price);
  }
}

package com.cloudnineapps.samples;

/**

* The inventory manager delegate.

public class InventoryManagerDelegate {

/** The DAO. */

private InventoryDAO dao;

/** Initializes the delegate. */

public void init() throws Exception {

dao = new InventoryDAO();

dao.createSchema();

}

/**

* Returns the details for the product with the supplied title.

* @param title The product title.

public Product getProductDetails(String title) throws Exception {

Product product = null;

try {

product = dao.getProduct(title);

}

catch(IllegalArgumentException ex) {

// Product not found, handle gracefully

product = new Product();

}

return product;

}

/**

* Returns the price for the specified product.

* @param title The product title.

public double getProductPrice(String title) throws Exception {

// NOTE: This code does not gracefully handle the product not found scenario

Product product = dao.getProduct(title);

return product.getPrice();

}

/**

* Returns the price change for the specified product from the current price. A negative value

* indicates decrease in price and a positive value indicates an increase.

* @param title The product title.

* @param currentPrice The current price of the product.

public double getProductPriceChange(String title, double currentPrice) throws Exception {

// NOTE: This code does not gracefully handle the product not found scenario

double price = 0;

try {

Product product = dao.getProduct(title);

price = product.getPrice();

}

catch(Exception ex) {

}

return (currentPrice - price);

}

/**

* Adds a product with the supplied details.

* @param id The product ID.

* @param title The product title.

* @param description The product description.

* @param price The product price.

public void addProduct(int id, String title, String description, double price) throws Exception {

dao.addProduct(id, title, description, price);

}

package com.cloudnineapps.samples;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.Statement;

/**
 * The inventory DAO.
 */
public class InventoryDAO {
  
  /** The database name. */
  private static final String DB_NAME = "inventory.db";

  /** Returns the current database connection. */
  public Connection getConnection() throws Exception {
    return DriverManager.getConnection(String.format("jdbc:hsqldb:file:data/%s", DB_NAME), "SA", "");
  }
  
  /** Closes the supplied connection. */
  public void closeConnection(Connection con) throws Exception {
    if (con != null) {
      con.close();
    }
  }
  
  /** Creates the database schema. */
  public void createSchema() throws Exception {
    String[] sqls = {
      "CREATE TABLE IF NOT EXISTS product("
        + "id INT NOT NULL,"
        + "title VARCHAR(255) NOT NULL,"
        + " description VARCHAR(255) NOT NULL,"
        + " price REAL NOT NULL,"
        + " PRIMARY KEY(id));",
      "CREATE UNIQUE INDEX IF NOT EXISTS idx_product_title ON product(title);"
    };
    Connection con = getConnection();
    try {
      for(int i = 0; i < sqls.length; i++) {
        Statement stmt = con.createStatement();
        stmt.execute(sqls[i]);
        stmt.close();
      }
    }
    finally {
      closeConnection(con);
    }
  }
  
  /**
   * Returns the product with the specified title, if any. Otherwise, null.
   * 
   * @param title The product title.
   * 
   * @exception IllegalArgumentException when the title was not found.
   */
  public Product getProduct(String title) throws Exception {
    Product product = null;
    String sql = "SELECT id, description, price FROM product WHERE title=?";
    Connection con = getConnection();
    PreparedStatement stmt = null;
    ResultSet rs = null;
    try {
      stmt = con.prepareStatement(sql);
      stmt.setString(1, title);
      rs = stmt.executeQuery();
      if (rs.next()) {
        int i = 1;
        int id = rs.getInt(i++);
        String description = rs.getString(i++);
        double price = rs.getDouble(i++);
        product = new Product();
        product.setId(id);
        product.setTitle(title);
        product.setDescription(description);
        product.setPrice(price);
      }
      else {
        // Product not found
        throw new IllegalArgumentException(String.format("A product with title '%s' not found.", title));
      }
    }
    finally {
      rs.close();
      stmt.close();
      closeConnection(con);
    }
        
    return product;
  }
  
  /**
   * Adds a product with the supplied details.
   * 
   * @param id The product ID.
   * @param title The product title.
   * @param description The product description.
   * @param price The product price.
   */
  public void addProduct(int id, String title, String description, double price) throws Exception {
    String sql = "INSERT INTO product(id, title, description, price) VALUES(?, ?, ?, ?)";
    try {
      if (getProduct(title) != null) {
        // Product already exists, for now skip processing
        return;
      }
    }
    catch(IllegalArgumentException ex) {}
    Connection con = getConnection();
    PreparedStatement stmt = null;
    try {
      stmt = con.prepareStatement(sql);
      int i = 1;
      stmt.setInt(i++, id);
      stmt.setString(i++, title);
      stmt.setString(i++, description);
      stmt.setDouble(i++, price);
      stmt.executeUpdate();
    }
    finally {
      stmt.close();
      closeConnection(con);
    }
  }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

package com.cloudnineapps.samples;

import java.sql.Connection;

import java.sql.DriverManager;

import java.sql.PreparedStatement;

import java.sql.ResultSet;

import java.sql.Statement;

/**

* The inventory DAO.

public class InventoryDAO {

/** The database name. */

private static final String DB_NAME = "inventory.db";

/** Returns the current database connection. */

public Connection getConnection() throws Exception {

return DriverManager.getConnection(String.format("jdbc:hsqldb:file:data/%s", DB_NAME), "SA", "");

}

/** Closes the supplied connection. */

public void closeConnection(Connection con) throws Exception {

if (con != null) {

con.close();

}

/** Creates the database schema. */

public void createSchema() throws Exception {

String[] sqls = {

"CREATE TABLE IF NOT EXISTS product("

+ "id INT NOT NULL,"

+ "title VARCHAR(255) NOT NULL,"

+ " description VARCHAR(255) NOT NULL,"

+ " price REAL NOT NULL,"

+ " PRIMARY KEY(id));",

"CREATE UNIQUE INDEX IF NOT EXISTS idx_product_title ON product(title);"

};

Connection con = getConnection();

try {

for(int i = 0; i < sqls.length; i++) {

Statement stmt = con.createStatement();

stmt.execute(sqls[i]);

stmt.close();

}

finally {

closeConnection(con);

}

/**

* Returns the product with the specified title, if any. Otherwise, null.

* @param title The product title.

* @exception IllegalArgumentException when the title was not found.

public Product getProduct(String title) throws Exception {

Product product = null;

String sql = "SELECT id, description, price FROM product WHERE title=?";

Connection con = getConnection();

PreparedStatement stmt = null;

ResultSet rs = null;

try {

stmt = con.prepareStatement(sql);

stmt.setString(1, title);

rs = stmt.executeQuery();

if (rs.next()) {

int i = 1;

int id = rs.getInt(i++);

String description = rs.getString(i++);

double price = rs.getDouble(i++);

product = new Product();

product.setId(id);

product.setTitle(title);

product.setDescription(description);

product.setPrice(price);

}

else {

// Product not found

throw new IllegalArgumentException(String.format("A product with title '%s' not found.", title));

}

finally {

rs.close();

stmt.close();

closeConnection(con);

}

return product;

}

/**

* Adds a product with the supplied details.

* @param id The product ID.

* @param title The product title.

* @param description The product description.

* @param price The product price.

public void addProduct(int id, String title, String description, double price) throws Exception {

String sql = "INSERT INTO product(id, title, description, price) VALUES(?, ?, ?, ?)";

try {

if (getProduct(title) != null) {

// Product already exists, for now skip processing

return;

}

catch(IllegalArgumentException ex) {}

Connection con = getConnection();

PreparedStatement stmt = null;

try {

stmt = con.prepareStatement(sql);

int i = 1;

stmt.setInt(i++, id);

stmt.setString(i++, title);

stmt.setString(i++, description);

stmt.setDouble(i++, price);

stmt.executeUpdate();

}

finally {

stmt.close();

closeConnection(con);

}

package com.cloudnineapps.samples;

/**
 * The product.
 */
public class Product {
  
  private int id;
  private String title;
  private String description;
  private double price;

  public int getId() {
    return id;
  }
  
  public void setId(int id) {
    this.id = id;
  }
  
  public String getTitle() {
    return title;
  }
  
  public void setTitle(String title) {
    this.title = title;
  }
  
  public String getDescription() {
    return description;
  }
  
  public void setDescription(String description) {
    this.description = description;
  }
  
  public double getPrice() {
    return price;
  }
  
  public void setPrice(double price) {
    this.price = price;
  }
}

package com.cloudnineapps.samples;

/**

* The product.

public class Product {

private int id;

private String title;

private String description;

private double price;

public int getId() {

return id;

}

public void setId(int id) {

this.id = id;

}

public String getTitle() {

return title;

}

public void setTitle(String title) {

this.title = title;

}

public String getDescription() {

return description;

}

public void setDescription(String description) {

this.description = description;

}

public double getPrice() {

return price;

}

public void setPrice(double price) {

this.price = price;

}

Most of the code is self-explanatory. But, here are some highlights.

The delegate initializes the DAO, which initializes the schema.
The delegate offers 3 methods that correspond to the 3 core use cases – getProductDetails(), getProductPrice() and getProductPriceChange().
You may have already noticed in the comments in the getProductPrice() and getProductPriceChange() method bodies that these do not handle the scenario when the product with the specified title is not found.

Let’s Unit Test This Code

Note: There are many types of unit testing methodologies used from unit testing each layer to sub-integration style tests. For the purpose of this article, this distinction is not relevant as the points covered can be applied to multiple types of unit tests.

package com.cloudnineapps.samples;

import junit.framework.TestCase;

/**
 * The unit tests for Inventory Manager delegate.
 */
public class TestInventoryManagerDelegate extends TestCase {

  /** The delegate. */
  private InventoryManagerDelegate delegate;
  
  
  public void setUp() throws Exception {
    delegate = new InventoryManagerDelegate();
    delegate.init();
    delegate.addProduct(1, "TV", "4K TV", 299.99);
    delegate.addProduct(2, "Receiver", "4K Receiver", 149.99);
    delegate.addProduct(3, "Audio System", "Surround Sound", 199.99);
  }
  
  /** Tests retrieval of product details successfully. */
  public void testGetProductDetailsSuccess() throws Exception {
    // Initialize
    String title = "TV";
    
    // Execute
    Product product = delegate.getProductDetails(title);
    
    // Validate
    assertEquals("Incorrect title.", title, product.getTitle());
  }
  
  /** Tests retrieval of product details when title does not exist. */
  public void testGetProductDetailsWithNonExistingTitle() throws Exception {
    // Initialize
    String title = "Remote";
    
    // Execute
    Product product = delegate.getProductDetails(title);
    
    // Validate
    assertNull("The title must be null.", product.getTitle());
  }
  
  /** Tests retrieval of product price successfully. */
  public void testGetProductPriceSuccess() throws Exception {
    // Initialize
    String title = "TV";
    double expectedPrice = 299.99;
    
    // Execute
    double price = delegate.getProductPrice(title);
    
    // Validate
    assertEquals("Incorrect price.", expectedPrice, price);
  }
  
  /** Tests retrieval of product price change successfully. */
  public void testGetProductPriceChangeSuccess() throws Exception {
    // Initialize
    String title = "TV";
    double currentPrice = 249.99;
    double expectedPriceChange = -50;
    
    // Execute
    double price = delegate.getProductPriceChange(title, currentPrice);
    
    // Validate
    assertEquals("Incorrect price change.", expectedPriceChange, price);
  }
}

package com.cloudnineapps.samples;

import junit.framework.TestCase;

/**

* The unit tests for Inventory Manager delegate.

public class TestInventoryManagerDelegate extends TestCase {

/** The delegate. */

private InventoryManagerDelegate delegate;

public void setUp() throws Exception {

delegate = new InventoryManagerDelegate();

delegate.init();

delegate.addProduct(1, "TV", "4K TV", 299.99);

delegate.addProduct(2, "Receiver", "4K Receiver", 149.99);

delegate.addProduct(3, "Audio System", "Surround Sound", 199.99);

}

/** Tests retrieval of product details successfully. */

public void testGetProductDetailsSuccess() throws Exception {

// Initialize

String title = "TV";

// Execute

Product product = delegate.getProductDetails(title);

// Validate

assertEquals("Incorrect title.", title, product.getTitle());

}

/** Tests retrieval of product details when title does not exist. */

public void testGetProductDetailsWithNonExistingTitle() throws Exception {

// Initialize

String title = "Remote";

// Execute

Product product = delegate.getProductDetails(title);

// Validate

assertNull("The title must be null.", product.getTitle());

}

/** Tests retrieval of product price successfully. */

public void testGetProductPriceSuccess() throws Exception {

// Initialize

String title = "TV";

double expectedPrice = 299.99;

// Execute

double price = delegate.getProductPrice(title);

// Validate

assertEquals("Incorrect price.", expectedPrice, price);

}

/** Tests retrieval of product price change successfully. */

public void testGetProductPriceChangeSuccess() throws Exception {

// Initialize

String title = "TV";

double currentPrice = 249.99;

double expectedPriceChange = -50;

// Execute

double price = delegate.getProductPriceChange(title, currentPrice);

// Validate

assertEquals("Incorrect price change.", expectedPriceChange, price);

}

Fairly straight-forward, right?

The first thing to notice is these unit tests are written with functionality in mind. For example, successful retrieval of product details and when product title was not found.
A couple of functional tests for getting product price and price change have been intentionally left out to show why simply relying on code coverage is not a good idea. For example, the test to check the behavior when a product is not found while getting price.

Let’s Generate The Code Coverage

Before we can talk about why code coverage is not enough, let’s generate it for this code. Here are some screenshots.

Overall Code Coverage

Delegate Code Coverage

Code Coverage of DAO getProduct() method

As we can see, the code coverage seems to be fairly good.

Why Code Coverage Is Not Enough?

Now, we can talk about why code coverage is not enough. The code exhibits 2 commonly observed issues in coding.

Missing Code For Functionality: The getProductPrice() method does not have code to gracefully handle the scenario when the product is not found. Instead, it simply propagates the exception.
Defective Code For Functionality: The getProductPriceChange() method does not correctly handle the scenario when the product is not found. In this case, the price change will actually return the current price of the product thus leading to incorrect behavior. This is a defect.

So, even though the code coverage is indicating a very high number (almost 100%), it by no means is a guarantee that the code is going to meet the business requirements. The first issue is very obvious as that code does not even exist in the production code. So, there is no way a code coverage tool will be able to measure its impact. However, the second issue is more subtle. It is an indication of how critical defects can easily slip through even when the code coverage was so high. Hence, it is extremely important that code coverage is not used solely to gauge the quality of code, especially by the developers.

Another important observation is the code coverage tool will report a line to be covered as long as it was hit at least once in the entire execution. However, as we can see, 2 of our methods have not been tested for missing product and hence the line where the DAO is generating the IllegalArgumentException is not even hit in the unit testing for these methods. But, it was hit for the get product details unit tests and hence the code coverage tool will report it as covered. While some code coverage tools may be better than others, it is perhaps easy to understand for a developer that these type of details are often buried in details and may not be easy to find. The point being, just because a line is covered in the code coverage report, it is not an indication that it is covered for each relevant functional scenario.

So, What’s The Solution?

In my experience, a combination of good functional unit tests along with code coverage is a reasonably good way of tackling this. A functional unit test is written with the business requirement in mind (like the one we saw in our sample code). There are many advantages to using this approach.

Firstly, you stop thinking in terms of making permutations and combinations, which are easy to miss. Many developers (including some of the smartest ones) are confused when it comes to unit testing. “What should I write? If a method takes 3 parameters, should I just pass null and combinations of these in my unit tests.” Not that this thought process is not important. But, functional unit tests make it easy to think in terms of how the product will be used and thus what positive and negative scenarios could arise.
The functional unit tests focus on real-world scenarios and make sure your code can handle these correctly. Hence, by writing these you are not only making the code more robust but also ensuring that you are not missing any critical functionality.
As a developer, your focus changes from attaining a code coverage number to ensuring as much of relevant functional scenarios are covered as possible.
I firmly believe that writing functional unit tests is a lot more fun. Sometimes, you have to be bit creative, both from coding the tests as well as coming up with scenarios.

I am sure at this point someone may be thinking “What about QA folks? Won’t they cover it in their tests?” The answer is “It is quite likely they would.” But, as a developer, you do not want to rely on giving a drop to QA to measure its quality. If you could unit test your code more effectively, won’t the code be much better and make you a stronger professional? Besides, the later the defects are found, the more expensive are these to fix. So, the sooner, you can take care of issues in the development lifecycle, the better. And, when you combine these functional unit tests with a continuously improving code coverage, you have a lot more reliable and robust code that meets the business needs.

Happy coding!
– Nitin

Enhance your AWS skills with these hands-on courses for real-world deployments.

Learn AWS basics for FREE.

Learn practical application development on AWS.

Also published on Medium.

Code Coverage Is Not Enough

Code Coverage Is Not Enough

Let’s See Some Code

Let’s Unit Test This Code

Let’s Generate The Code Coverage

Why Code Coverage Is Not Enough?

So, What’s The Solution?

Enhance your AWS skills with these hands-on courses for real-world deployments.

Leave a Reply Cancel reply