Friday, June 19, 2015

Using OpenCover and ReportGenerator to get Unit Testing Code Coverage Metrics in .NET

If you are fortunate enough to use the highest offered VS.NET edition inclusive of additional testing abilities, or purchased a license to a product such as dotCover then you already have access to unit testing code coverage tools. However there is still an easy and powerful way to get the same type metrics using a combination of msbuild.exe and (2) open source tools: OpenCover and ReportGenerator.



OpenCover will leverage msbuild.exe and analyze code to determine the amount of code coverage your application has in reference to the unit tests written against it. ReportGenerator will then leverage those results and display them in a .html report output that is generated. The really cool part of it all is that since it is all scriptable, you could make the output report an artifact of a Continuous Integration (CI) build definition. In this manner you can see how a team or project is doing in reference to the code base and unit testing after each check-in and build.


A quick word on what this post will not get into - What percentage is good to have that indicates the code has decent coverage? 100%? 75%? Does it matter? The answer is, it depends and there is no single benchmark to use. The answer lies in the fact that one should strive to create unit tests that are are meaningful. Unit testing getters and setters to achieve 100% code coverage might not be a good use of time. Testing to make sure a critical workflow's state and behavior are as intended are examples of unit tests to be written. The output of these tools will just help highlight any holes in unit testing that might exist. I could go into detail on percentages of code coverage and what types of tests to write in another post. The bottom line - just make sure you are at least doing some unit testing. 0% is not acceptable by any means.

Prior to starting, you also can get this script from my 'BowlingSPAService' solution in GitHub within the BowlingSPA repository. You can clone the repository to your machine and inspect or run the script. Run the script as an Administrator and view the output.

BowlingSPA GitHub


1. Download NuGet Packages:


Download and import the following (2) open source packages from NuGet into your test project. If you have multiple test projects, no worries. The package will be referenced in the script via its 'packages' folder location and not via any specific project. The test projects output is the target of these packages.

OpenCover - Used for calculating the metrics

Report Generator - Used for displaying the metrics

The documentation you'll need to refer to most is for OpenCover. It's Wiki is on GitHub and can be found at the location below. ReportGenerator doesn't need to be tweaked so much as it really just displays the output metrics report in HTML generated by OpenCover. This was my guide for creating the batch file commands used in this article.

OpenCover Wiki


2. Create a .bat file script in your solution 


I prefer to place these types of artifacts in a 'Solution Folder' (virtual folder) at the root to be easily accessible.

3. Use the following to commands to generate the metrics and report

   a. Run OpenCover using mstest.exe as the target:

Note: Make sure the file versions in this script code are updated to match whatever NuGet package version you have downloaded.

"%~dp0..\packages\OpenCover.4.5.3723\OpenCover.Console.exe" ^
-register:user ^
-target:"%VS120COMNTOOLS%\..\IDE\mstest.exe" ^
-targetargs:"/testcontainer:\"%~dp0..\BowlingSPAService.Tests\bin\Debug\BowlingSPAService.Tests.dll\" /resultsfile:\"%~dp0BowlingSPAService.trx\"" ^
-filter:"+[BowlingSPAService*]* -[BowlingSPAService.Tests]* -[*]BowlingSPAService.RouteConfig" ^
-mergebyhash ^
-skipautoprops ^
-output:"%~dp0\GeneratedReports\BowlingSPAServiceReport.xml"



The main pieces to point out here are the following:
  • Leverages mstest.exe to target the 'BowlingSPAService.Tests.dll' and send the test results to an output .trx file. Note: you can chain together as many test .dlls as you have in your solution; you might certainly have more than 1 test project
  • I've added some filters that will add anything in the 'BowlingSPAService' namespace, but also removing code in the 'BowlingSPAService.Tests' namespace as I don't want metrics on the test code itself or for it to show up on the output report. Note: these filters can have as many or few conditions you need for your application. You will after getting familiar with the report probably want to remove auto-generated classes (i.e. Entity Framework, WCF, etc.) from the test results via their namespace.
  • Use 'mergebyhash' to merge results loaded from multiple assemblies
  • Use 'skipautoprops' to skip .NET 'AutoProperties' from being analyzed (basic getters and setters don't require unit tests and thus shouldn't be reported on the output)
  • Output the information for the report (used by ReportGenerator) to 'BowlingSPAServiceReport.xml'

   b. Run Report Generator to create a human readable HTML report

"%~dp0..\packages\ReportGenerator.2.1.5.0\ReportGenerator.exe" ^
-reports:"%~dp0\GeneratedReports\BowlingSPAServiceReport.xml" ^
-targetdir:"%~dp0\GeneratedReports\ReportGenerator Output"

The main pieces to point out here are the following:
  • Calls ReportGenerator.exe from the packages directory (NuGet), providing the output .xml report file genrated from #3(a) above, and specifying the output target directory folder to generate the index.htm page. 
  • The report creation directory can be anywhere you wish, but I created a folder named 'ReportGenerator Output'

   c. Automatically open the report in the browser

start "report" "%~dp0\GeneratedReports\ReportGenerator Output\index.htm"

The main pieces to point out here are the following:
  • This will open the generated report in the machine's default browser
  • Note: if IE is used, you will be prompted to allow 'Blocked Content.' I usually allow as it provides links on the page with options to collapse and expand report sections.

4. Stitch together all the sections into a single script to run


REM Create a 'GeneratedReports' folder if it does not exist
if not exist "%~dp0GeneratedReports" mkdir "%~dp0GeneratedReports"

REM Remove any previous test execution files to prevent issues overwriting
IF EXIST "%~dp0BowlingSPAService.trx" del "%~dp0BowlingSPAService.trx%"

REM Remove any previously created test output directories
CD %~dp0
FOR /D /R %%X IN (%USERNAME%*) DO RD /S /Q "%%X"

REM Run the tests against the targeted output
call :RunOpenCoverUnitTestMetrics

REM Generate the report output based on the test results
if %errorlevel% equ 0 ( 
 call :RunReportGeneratorOutput 
)

REM Launch the report
if %errorlevel% equ 0 ( 
 call :RunLaunchReport 
)
exit /b %errorlevel%

:RunOpenCoverUnitTestMetrics
"%~dp0..\packages\OpenCover.4.5.3723\OpenCover.Console.exe" ^
-register:user ^
-target:"%VS120COMNTOOLS%\..\IDE\mstest.exe" ^
-targetargs:"/testcontainer:\"%~dp0..\BowlingSPAService.Tests\bin\Debug\BowlingSPAService.Tests.dll\" /resultsfile:\"%~dp0BowlingSPAService.trx\"" ^
-filter:"+[BowlingSPAService*]* -[BowlingSPAService.Tests]* -[*]BowlingSPAService.RouteConfig" ^
-mergebyhash ^
-skipautoprops ^
-output:"%~dp0\GeneratedReports\BowlingSPAServiceReport.xml"
exit /b %errorlevel%

:RunReportGeneratorOutput
"%~dp0..\packages\ReportGenerator.2.1.5.0\ReportGenerator.exe" ^
-reports:"%~dp0\GeneratedReports\BowlingSPAServiceReport.xml" ^
-targetdir:"%~dp0\GeneratedReports\ReportGenerator Output"
exit /b %errorlevel%

:RunLaunchReport
start "report" "%~dp0\GeneratedReports\ReportGenerator Output\index.htm"
exit /b %errorlevel%


This is what the complete script could look like. It adds the following pieces:

    a. Create an output directory for the report if it doesn't already exist
    a. Remove previous test execution files to prevent overwrite issues
    b. Remove previously created test output directories
    c. Run all sections together synchronously ensuring each step finishes successfully before proceeding


5. Analyze the output



Upon loading the report you can see immediately in percentages how well the projects are covered. As one can see below I have a decent start to the coverage in the pertinent areas of BowlingSPAService, but the report shows I need some additional testing. However, this is exactly the tool that makes me aware of this void. I need to write some more unit tests! (I was eager to get this post completed and published before finishing unit testing) :)

You can expand an individual project to get a visual of each of the class's unit test coverage:


By selecting an individual class, you can see a line level coverage visually with red and green highlighting. Red means there isn't coverage, and green means there is coverage. This report and visual metric highlights well when unit tests have probably only been written for 'happy path' scenarios, and there are unit test gaps for required negative or branching scenarios. By inspecting the classes that are not at 100% coverage, you can easily identify gaps and write additional unit tests to increase the coverage where needed.




Upon review you might find individual 'template' classes or namespaces that add 'noise' and are not realistically valid targets for the unit testing metrics. Add the following style filters to the filter switch targeting OpenCover to remove a namespace or a single class respectively:
  • -[*]BowlingSPAService.WebAPI.Areas.HelpPage.*
  • -[*]BowlingSPAService.WebApiConfig
After adding or modifying unit tests, you can run the report again and see the updated results. Questions about the particulars of how the metrics are evaluated are better directed toward the source for OpenCover on GitHub.

That's all you should need to get up and running to generate unit test metrics for your .NET solutions! As mentioned previously, you might want to add this as an output artifact to your CI build server and provide the report link to a wider audience for general viewing.  There are also additional fine tuning options and customizations you can make so be sure to check out the OpenCover Wiki I posted previously. 

18 comments:

  1. In the line-level coverage report, what do the numbers in the margin signifiy? In your example they are either 0 or 1. Is this a boolean indicating that the line has coverage? Is it the number of times the line was reached in all unit tests? Is something else?

    ReplyDelete
  2. Hey Allen,

    Your blog is very helpful for generating Unit Test Code coverage report. I am able to generate report for single dll but my project has multiple dlls , so I am facing the issue to generate combined report for multiple dlls.
    Could you please help me on this.

    ReplyDelete
  3. This is great. Thank you. But you skipped over the part about how to run it. I get how I can integrate it into a CI server. But how do I run it locally when I want to see if i need to create more unit tests?

    Do I open a standard command window? The script seems to be using some VS env vars so I assume it needs to be launched from VS somehow.

    Thanks
    Greg

    ReplyDelete
    Replies
    1. Yes, this process is creating a .bat (batch) file. You can just double click the batch file from explorer, and if it is set up correctly the commands within will execute and the very last step is to auto open the report.

      Now if you run the .bat file and nothing happens that means there is an error. To debug, make sure the command window doesn't close itself.

      Delete
  4. Ok, I figured out a couple things that I had wrong.
    I am using VS2015. So the VS120... env var is now VS140...
    And all the nuget packages put their binaries in a directory called tools so the paths had to be altered for that.

    I then added it as an "External tool" under the Tools menu. I can now launch it any time from within VS by selecting the new menu item.

    Thank you very much for taking the time to write this blog. It was very helpful!

    ReplyDelete
  5. Hello Allen,
    I want to use open cover to validate the coverage file by file. For e.g I have math.cs and mathtest.cs(NUnit). I tried to provide the cmd line args as -register:user -target:"C:\Program Files (x86)\NUnit 2.6.2\bin\nunit-console.exe" -targetdir:"\NUnit_Application\Test" -output:"\NUnit_Application\Test\Report.xml". The test cases are generating 0 for coverage. Can you please suggest if I have gone wrong in providing cmd line args or is there any other way to achieve this

    ReplyDelete
  6. Hello Everyone,
    i want to find out code coverage of application which is test by automation code. e.g. my automation script code (written in c#) has a login method which perform login to XYZ application using selenium c#. now i want to know how much my login(automation code)method effects to XYZ application like, how many line of code are executed and other features which normally code coverage tools providing.

    ReplyDelete
  7. I have created the batch file and made changes to the current version of OpenCover and ReportGenerator. I have also created a script inside a new batch file
    (@cmd /c %1
    @pause ) so that I can run Allen Conway's batch file with my batch file. The command prompt opens but nothing happens after that. Could someone help me with this? Also I guess it has something to do with what Greg Veres has posted above.

    ReplyDelete
  8. I did have to change my VS version like Greg mentions above. Also, in newer versions of OpenCover and ReportGenerator the .exe's moved to the /tools folder. So..

    "%~dp0packages\OpenCover.4.6.519\OpenCover.Console.exe" ^

    Should be...
    "%~dp0packages\OpenCover.4.6.519\tools\OpenCover.Console.exe" ^

    You may also need to alter the paths a bit to match your solution.

    ReplyDelete
  9. Hello Allen,
    Excellent tutorial. I could make it work thanks to you. It was very helpful!
    Regards,
    Matias

    ReplyDelete
  10. Hi Allen,

    I tried to execute the mentioned batch script, but it is giving the error "The input line is too long" , on "RunOpenCoverUnitTestMetrics" this method.
    Can you please help on this?

    ReplyDelete
  11. Great blog thanks.

    For anyone using xUnit you need to add -noshadow to the targetargs otherwise you don't get any results.

    ReplyDelete
  12. excelent, should you add
    @setlocal enableextensions
    @cd /d "%~dp0"
    and then remove the others %~dp0 ocurrences :)

    ReplyDelete
  13. Allen, thanks bunches!!! this is perfect and helped me get to the end report quickly.

    ReplyDelete
  14. Thanks man, good job!

    ReplyDelete
  15. I'm trying to generate a code coverage report for a project with xUnit, OpenCover and ReportGenerator, but I don't know why the code coverage is reporting 0 (zero) coverage for all my code.

    I have one solution for my project's code and another solution for my unit tests. Do you think this may be the cause?

    ReplyDelete
  16. The integration with OpenCover and ReportGenerator is working fine. It generates the report. The only problem is that for some reason my tested code isn't reporting any code coverage value. Keeps at zero even though I'm sure I've tested 100% of one the classes.

    ReplyDelete