Yesterday I wanted to setup the code coverage in our sonar instance for integration tests launched using the failsafe plugin of Apache Maven. The awaited result in sonar is something like this :
I found several constraints while implementing it :
- Integration tests may require to have some tasks called in the pre-integration-test phase (for example to generate some random ports to launch an application server and avoid to conflicts on your CI server)
- Sonar can itself execute your unit tests managed by maven and its surefire plugin but it has nothing for integration tests by default.
- Sonar can call a maven lifecycle or mojo (property sonar.phase) before being executed but we cannot use this to call integration tests without having it re-executing a second time your unit tests.
- Sonar cannot display integration tests results (only the coverage)
Digging into various posts (see references) I found a satisfying (but not perfect) solution to fit my needs.
In my corporate POM I added these properties
<!-- ************************ -->
<!-- Sonar/Reporting settings -->
<!-- ************************ -->
<!-- Sonar/Jacoco integration. Note that these properties need to be defined outside the "coverage" profile
because we want to be to able to execute mvn sonar:sonar without passing a profile -->
<!-- Tells Sonar to use jacoco for coverage results -->
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<!-- Jacoco version to use -->
<jacoco.version>0.6.3.201306030806</jacoco.version>
<!-- The Sonar Jacoco Listener for JUnit to extract coverage details per test -->
<sonar-jacoco-listeners.version>1.4</sonar-jacoco-listeners.version>
<!-- Don't let Sonar execute tests. We will ask it to Maven -->
<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
<!-- The system property jacoco.outputDir needs to be override on the command line
with an absolute path if you want to merge results from all modules.
Example in a Jenkisn build where ${WORKSPACE} is defined and your project in the root directory of the workspace :
mvn clean install -Prun-its,coverage -Djacoco.outputDir=${WORKSPACE}/target
Note that unfortunately using the following does not work because of
http://jira.codehaus.org/browse/SONAR-3427:
<jacoco.outputDir>${session.executionRootDirectory}/target/</jacoco.outputDir>
-->
<jacoco.outputDir>${project.build.directory}</jacoco.outputDir>
<!-- Jacoco output file for UTs -->
<jacoco.out.ut.file>jacoco-ut.exec</jacoco.out.ut.file>
<!-- Tells Sonar where the Jacoco coverage result file is located for Unit Tests -->
<sonar.jacoco.reportPath>${jacoco.outputDir}/${jacoco.out.ut.file}</sonar.jacoco.reportPath>
<!-- Jacoco output file for ITs -->
<jacoco.out.it.file>jacoco-it.exec</jacoco.out.it.file>
<!-- Tells Sonar where the Jacoco coverage result file is located for Integration Tests -->
<sonar.jacoco.itReportPath>${jacoco.outputDir}/${jacoco.out.it.file}</sonar.jacoco.itReportPath>
And I added this new profile
<profile>
<id>coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>${jacoco.agent.ut.arg}</argLine>
<!-- Specific to generate mapping between tests and covered code -->
<properties>
<property>
<name>listener</name>
<value>org.sonar.java.jacoco.JUnitListener</value>
</property>
</properties>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<argLine>-Xmx1024m -XX:MaxPermSize=256m ${jacoco.agent.it.arg}</argLine>
<!-- Specific to generate mapping between tests and covered code -->
<properties>
<property>
<name>listener</name>
<value>org.sonar.java.jacoco.JUnitListener</value>
</property>
</properties>
<!-- Let's put failsafe reports with surefire to have access to tests failures/success reports in sonar -->
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
</configuration>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<!-- Prepares a variable, jacoco.agent.ut.arg, that contains the info to be passed to the JVM hosting the code
being tested. -->
<execution>
<id>prepare-ut-agent</id>
<phase>process-test-classes</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${sonar.jacoco.reportPath}</destFile>
<propertyName>jacoco.agent.ut.arg</propertyName>
<append>true</append>
</configuration>
</execution>
<!-- Prepares a variable, jacoco.agent.it.arg, that contains the info to be passed to the JVM hosting the code
being tested. -->
<execution>
<id>prepare-it-agent</id>
<phase>pre-integration-test</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${sonar.jacoco.itReportPath}</destFile>
<propertyName>jacoco.agent.it.arg</propertyName>
<append>true</append>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.codehaus.sonar-plugins.java</groupId>
<artifactId>sonar-jacoco-listeners</artifactId>
<version>${sonar-jacoco-listeners.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</profile>
I already had one to launch integration tests with failsafe
<!--
This profile is used to launch integration tests with the failsafe plugin
http://maven.apache.org/plugins/maven-failsafe-plugin/
-->
<profile>
<id>run-its</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>integration-test</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<phase>verify</phase>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
Thus in my CI server (Jenkins) I just have to create a job that calls
mvn verify -Prun-its,coverage -Djacoco.outputDir=${WORKSPACE}/target
And I add Sonar as a post build step which will grab tests and coverage results.
Note : -Djacoco.outputDir=${WORKSPACE}/target is necessary only if you want to merge all coverage results across several modules.
With my test project I obtain the expected result.
References
- Code Coverage by Integration Tests for Java Project
- Code Coverage by Unit Tests for Java Project
- Sonar Analysis Parameters
- Sonar examples
- Code coverage configuration for Xwiki (thanks @vmassol)
- SONAR-3427 : Setting sonar.jacoco.itReportPath property via maven ${session.executionRootDirectory} does not work : I agree with Fabrice I didn’t know myself this property but it is sad to not implement it as it is working in some others plugins (the question is to know if there is a minimum version of Maven to support it)
- SONARJAVA-94 : Provide JUnit Listener to record coverage per test
- Tracking Integration Test Coverage with Maven and SonarQube
- Maven Sonar plugin execute surefire tests twice when setting the property sonar.phase=post-integration-test
- Failsafe tests results in sonar
- Separating Integration and Unit Tests with Maven, Sonar, Failsafe, and JaCoCo
Nice article Arnaud. Just a typo about your CI server name : Jenkins instead of Kenkins
Fixed. Thanks David
Hi,
thanks for your article. Can you explain what does:
listener
org.sonar.java.jacoco.JUnitListener
do? It does not seem to make a difference for me if I incluce it or not. Or maybe I just do not see the difference.
Thanks,
Ronald
Hi Ronald,
It is necessary to see in Sonar your coverage per test : http://docs.codehaus.org/display/SONAR/Code+Coverage+by+Unit+Tests+for+Java+Project
Otherwise you’ll have only a global overview of the coverage for all tests.
Arnaud
Great Article! 10/10 as this was the only place I was able to reference to get the proper reporting working in Sonar. I have one side effect I’m wondering by chance that you have any insight into – I can no longer “Test File” nor “Run Focused Test Method” in Netbeans IDE 7.4. This runs fine from Jenkins as an automated job.
The error observed is:
Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-cli) on project test-project: Execution default-cli of goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test failed: There was an error in the forked process
org.apache.maven.surefire.util.SurefireReflectionException: org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable to access JaCoCo Agent - make sure that you use JaCoCo and version not lower than 0.6.2.; nested exception is org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable to access JaCoCo Agent - make sure that you use JaCoCo and version not lower than 0.6.2.
org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable to access JaCoCo Agent - make sure that you use JaCoCo and version not lower than 0.6.2.
Wonderful work !
It spared me hours!
Your page should be included in the official SonarQube documentation.
Thanks again
Hello
Thanks for this good reference.
Your solutions sounds good to apply on my project, but I’m getting a org.sonar.api.test.exception.CoverageAlreadyExistsException, which points to a test method and class that this test covers. Have you faced some related problem? Seems that it happens because a class is covered more than one test.
The reference for the class that throws the exception is: https://github.com/SonarSource/sonarqube/blob/master/sonar-core/src/main/java/org/sonar/core/test/DefaultTestCase.java
I followed your example and got the following error.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.13:integration-test (default-cli) on project fms-arquilli
an-tests: Execution default-cli of goal org.apache.maven.plugins:maven-failsafe-plugin:2.13:integration-test failed: There was an error i
n the forked process
[ERROR] org.apache.maven.surefire.util.SurefireReflectionException: org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable
to access JaCoCo Agent – make sure that you use JaCoCo and version not lower than 0.6.2.; nested exception is org.sonar.java.jacoco.Jacoc
oController$JacocoControllerError: Unable to access JaCoCo Agent – make sure that you use JaCoCo and version not lower than 0.6.2.
[ERROR] org.sonar.java.jacoco.JacocoController$JacocoControllerError: Unable to access JaCoCo Agent – make sure that you use JaCoCo and v
ersion not lower than 0.6.2.
[ERROR] at org.sonar.java.jacoco.JacocoController.(JacocoController.java:48)
My jacoco version is 0.7.2.201409121644
Thank you.
Awesome, thanks Arnaud! Most succinct article I’ve found on how to get Sonar to report on Integration test results.
Kind regards, Andrew
You need to remove all the listener stuff:
See https://groups.google.com/forum/#!topic/jacoco/NlOKWfXD6sA
sonar
org.apache.maven.plugins
maven-surefire-plugin
${jacoco.agent.ut.arg}
org.jacoco
jacoco-maven-plugin
0.7.2.201409121644
prepare-ut-agent
process-test-classes
prepare-agent
${project.build.directory}/jacoco.exec
jacoco.agent.ut.arg
true
Hi, Can anybody plz give the solution how to fix this problem and what exactly the reason of generating the build. It is a sonar build related error. Build has been failing due to this error below…. ,
[JaCoCo plugin] Loading inclusions files..
03:53:30 [JaCoCo plugin] inclusions: []
03:53:30 [JaCoCo plugin] exclusions: []
03:53:32 ERROR: Publisher hudson.plugins.jacoco.JacocoPublisher aborted due to exception
03:53:32 java.lang.IllegalStateException: Incompatible execution data for class ch/qos/logback/core/util/OptionHelper with id 717f79a9d8de53f0.
03:53:32 at org.jacoco.core.data.ExecutionData.assertCompatibility(ExecutionData.java:181)
03:53:32 at org.jacoco.core.data.ExecutionData.merge(ExecutionData.java:144)
03:53:32 at org.jacoco.core.data.ExecutionData.merge(ExecutionData.java:117)
03:53:32 at org.jacoco.core.data.ExecutionDataStore.put(ExecutionDataStore.java:53)
03:53:32 at org.jacoco.core.data.ExecutionDataStore.visitClassExecution(ExecutionDataStore.java:175)
03:53:32 at org.jacoco.core.data.ExecutionDataReader.readExecutionData(ExecutionDataReader.java:149)
03:53:32 at org.jacoco.core.data.ExecutionDataReader.readBlock(ExecutionDataReader.java:113)
03:53:32 at org.jacoco.core.data.ExecutionDataReader.read(ExecutionDataReader.java:87)
03:53:32 at hudson.plugins.jacoco.ExecutionFileLoader.loadExecutionData(ExecutionFileLoader.java:91)
03:53:32 at hudson.plugins.jacoco.ExecutionFileLoader.loadBundleCoverage(ExecutionFileLoader.java:136)
03:53:32 at hudson.plugins.jacoco.JacocoReportDir.parse(JacocoReportDir.java:102)
03:53:32 at hudson.plugins.jacoco.JacocoBuildAction.loadRatios(JacocoBuildAction.java:291)
03:53:32 at hudson.plugins.jacoco.JacocoBuildAction.load(JacocoBuildAction.java:273)
03:53:32 at hudson.plugins.jacoco.JacocoPublisher.perform(JacocoPublisher.java:371)
03:53:32 at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:51)
03:53:32 at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:736)
03:53:32 at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:714)
03:53:32 at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:690)
03:53:32 at hudson.model.Build$RunnerImpl.post2(Build.java:163)
03:53:32 at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:652)
03:53:32 at hudson.model.Run.run(Run.java:1517)
03:53:32 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
03:53:32 at hudson.model.ResourceController.execute(ResourceController.java:82)
03:53:32 at hudson.model.Executor.run(Executor.java:137)
03:53:32 Skipping sonar analysis due to bad build status FAILURE
03:53:32 Email was triggered for: Failure
03:53:32 Sending email for trigger: Failure
03:53:32 Sending email to: insight-dev-ci@ptc.com insight-qa-ci@ptc.com
03:53:36 [DEBUG] Skipping watched dependency update for build: Trunk_Sonar #766 due to result: FAILURE
03:53:36 Finished: FAILURE
Hi, Can anybody plz give the solution how to fix this problem and what exactly the reason of generating the build. It is a sonar build related error. Build has been failing due to this error below…. ,
[JaCoCo plugin] Loading inclusions files..
03:53:30 [JaCoCo plugin] inclusions: []
03:53:30 [JaCoCo plugin] exclusions: []
03:53:32 ERROR: Publisher hudson.plugins.jacoco.JacocoPublisher aborted due to exception
03:53:32 java.lang.IllegalStateException: Incompatible execution data for class ch/qos/logback/core/util/OptionHelper with id 717f79a9d8de53f0.
03:53:32 at org.jacoco.core.data.ExecutionData.assertCompatibility(ExecutionData.java:181)
03:53:32 at org.jacoco.core.data.ExecutionData.merge(ExecutionData.java:144)
03:53:32 at org.jacoco.core.data.ExecutionData.merge(ExecutionData.java:117)
03:53:32 at org.jacoco.core.data.ExecutionDataStore.put(ExecutionDataStore.java:53)
03:53:32 at org.jacoco.core.data.ExecutionDataStore.visitClassExecution(ExecutionDataStore.java:175)
03:53:32 at org.jacoco.core.data.ExecutionDataReader.readExecutionData(ExecutionDataReader.java:149)
03:53:32 at org.jacoco.core.data.ExecutionDataReader.readBlock(ExecutionDataReader.java:113)
03:53:32 at org.jacoco.core.data.ExecutionDataReader.read(ExecutionDataReader.java:87)
03:53:32 at hudson.plugins.jacoco.ExecutionFileLoader.loadExecutionData(ExecutionFileLoader.java:91)
03:53:32 at hudson.plugins.jacoco.ExecutionFileLoader.loadBundleCoverage(ExecutionFileLoader.java:136)
03:53:32 at hudson.plugins.jacoco.JacocoReportDir.parse(JacocoReportDir.java:102)
03:53:32 at hudson.plugins.jacoco.JacocoBuildAction.loadRatios(JacocoBuildAction.java:291)
03:53:32 at hudson.plugins.jacoco.JacocoBuildAction.load(JacocoBuildAction.java:273)
03:53:32 at hudson.plugins.jacoco.JacocoPublisher.perform(JacocoPublisher.java:371)
03:53:32 at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:51)
03:53:32 at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:736)
03:53:32 at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:714)
03:53:32 at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:690)
03:53:32 at hudson.model.Build$RunnerImpl.post2(Build.java:163)
03:53:32 at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:652)
03:53:32 at hudson.model.Run.run(Run.java:1517)
03:53:32 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
03:53:32 at hudson.model.ResourceController.execute(ResourceController.java:82)
03:53:32 at hudson.model.Executor.run(Executor.java:137)
03:53:32 Skipping sonar analysis due to bad build status FAILURE
03:53:32 Email was triggered for: Failure
03:53:32 Sending email for trigger: Failure
03:53:32 Sending email to: ***********************
03:53:36 [DEBUG] Skipping watched dependency update for build: Trunk_Sonar #766 due to result: FAILURE
03:53:36 Finished: FAILURE
In a multimodule project do I have to add it in all the modules or only in the root module? I have tried to do it in all the modules, the unit test coverage is correct but it seems its not calculating the it coveage properly. The value is too low.