Code Coverage Is Not Enough

Code Coverage has been an important quality metric to measure the robustness of the code for several years now. It is part of the release criteria for many organizations from development to quality assurance and, at times, even for the senior management team. In fact, it is not unusual that several teams strive for a higher code coverage metric anywhere from 60-80% as the bare minimum threshold for a release. With all that said it is important to understand that code coverage by itself is not as useful. For example, even if you had a very high code coverage of 80% and the majority of your core logic is in that other 20% of the code, chances are high the code could have some serious defects that may slip into the release. What is worse that even if the code coverage were 100% that does not mean all the functionality has been covered! There could be important functional scenarios that might have been missed in the production code and the unit tests, which may again lead to an incomplete release. Don’t get me wrong. Code Coverage is important, but not when it is looked in isolation. When combined with good functional unit tests, it can be extremely useful. And, that’s what we will cover in this article.

Let’s See Some Code

What would be a better way of showing other than using an actual code? So, let’s take a barebone Inventory Management example that offers the following basic functionality.
Note: The functionality and logic are intentionally kept simple so that we can focus on the main topic of this article.

  • Get product details.
  • Get the price of a product.
  • Get the price change of a product.

In order to support this functionality, we have 3 classes.

  • InventoryManagerDelegate: This contains the core business logic and will typically be used by another tier or layer, such as a REST API layer to offer the inventory management APIs.
  • InventoryDAO: The Data Access Object to handle all database interactions, such as adding and retrieving a product.
  • Product: The model or data object to store product information (a.k.a. POJO (Plain Old Java Object) in the Java lingo).

Here is the source code for these classes.

Most of the code is self-explanatory. But, here are some highlights.

  • The delegate initializes the DAO, which initializes the schema.
  • The delegate offers 3 methods that correspond to the 3 core use cases – getProductDetails(), getProductPrice() and getProductPriceChange().
  • You may have already noticed in the comments in the getProductPrice() and getProductPriceChange() method bodies that these do not handle the scenario when the product with the specified title is not found.

Let’s Unit Test This Code

Note: There are many types of unit testing methodologies used from unit testing each layer to sub-integration style tests. For the purpose of this article, this distinction is not relevant as the points covered can be applied to multiple types of unit tests.

Fairly straight-forward, right?

  • The first thing to notice is these unit tests are written with functionality in mind. For example, successful retrieval of product details and when product title was not found.
  • A couple of functional tests for getting product price and price change have been intentionally left out to show why simply relying on code coverage is not a good idea. For example, the test to check the behavior when a product is not found while getting price.

Let’s Generate The Code Coverage

Before we can talk about why code coverage is not enough, let’s generate it for this code. Here are some screenshots.

Overall Code Coverage

Delegate Code Coverage

Code Coverage of DAO getProduct() method

As we can see, the code coverage seems to be fairly good.

Why Code Coverage Is Not Enough?

Now, we can talk about why code coverage is not enough. The code exhibits 2 commonly observed issues in coding.

  1. Missing Code For Functionality: The getProductPrice() method does not have code to gracefully handle the scenario when the product is not found. Instead, it simply propagates the exception.
  2. Defective Code For Functionality: The getProductPriceChange() method does not correctly handle the scenario when the product is not found. In this case, the price change will actually return the current price of the product thus leading to incorrect behavior. This is a defect.

So, even though the code coverage is indicating a very high number (almost 100%), it by no means is a guarantee that the code is going to meet the business requirements. The first issue is very obvious as that code does not even exist in the production code. So, there is no way a code coverage tool will be able to measure its impact. However, the second issue is more subtle. It is an indication of how critical defects can easily slip through even when the code coverage was so high. Hence, it is extremely important that code coverage is not used solely to gauge the quality of code, especially by the developers.

Another important observation is the code coverage tool will report a line to be covered as long as it was hit at least once in the entire execution. However, as we can see, 2 of our methods have not been tested for missing product and hence the line where the DAO is generating the IllegalArgumentException is not even hit in the unit testing for these methods. But, it was hit for the get product details unit tests and hence the code coverage tool will report it as covered. While some code coverage tools may be better than others, it is perhaps easy to understand for a developer that these type of details are often buried in details and may not be easy to find. The point being, just because a line is covered in the code coverage report, it is not an indication that it is covered for each relevant functional scenario.

So, What’s The Solution?

In my experience, a combination of good functional unit tests along with code coverage is a reasonably good way of tackling this. A functional unit test is written with the business requirement in mind (like the one we saw in our sample code). There are many advantages to using this approach.

  • Firstly, you stop thinking in terms of making permutations and combinations, which are easy to miss. Many developers (including some of the smartest ones) are confused when it comes to unit testing. “What should I write? If a method takes 3 parameters, should I just pass null and combinations of these in my unit tests.” Not that this thought process is not important. But, functional unit tests make it easy to think in terms of how the product will be used and thus what positive and negative scenarios could arise.
  • The functional unit tests focus on real-world scenarios and make sure your code can handle these correctly. Hence, by writing these you are not only making the code more robust but also ensuring that you are not missing any critical functionality.
  • As a developer, your focus changes from attaining a code coverage number to ensuring as much of relevant functional scenarios are covered as possible.
  • I firmly believe that writing functional unit tests is a lot more fun. Sometimes, you have to be bit creative, both from coding the tests as well as coming up with scenarios.

I am sure at this point someone may be thinking “What about QA folks? Won’t they cover it in their tests?” The answer is “It is quite likely they would.” But, as a developer, you do not want to rely on giving a drop to QA to measure its quality. If you could unit test your code more effectively, won’t the code be much better and make you a stronger professional? Besides, the later the defects are found, the more expensive are these to fix. So, the sooner, you can take care of issues in the development lifecycle, the better. And, when you combine these functional unit tests with a continuously improving code coverage, you have a lot more reliable and robust code that meets the business needs.

Happy coding!
– Nitin


Enhance your AWS skills with these hands-on courses for real-world deployments.

Learn AWS basics for FREE.

Learn practical application development on AWS.

Also published on Medium.

Leave a Reply

Your email address will not be published. Required fields are marked *