JaCoCo plugin gives different coverage

Question

I'm using the JaCoCo Maven plugin and agent to measure and retrieve the code coverage data of an application which is tested nightly. This is the schema of the architecture:

My Maven project is configured with the JMeter Maven plugin to execute some API tests during the Maven verify phase. The Maven command executed by the Jenkins server is the following

mvn verify org.jacoco:jacoco-maven-plugin:0.7.8:dump sonar:sonar -Djacoco.address=TEST_SERVER -Djacoco.destFile=/proj/coverage-reports/jacoco-it.exec -Dsonar.projectKey=sonar_test -Dsonar.projectName=sonar_test -Dsonar.branch=sonar_test -Dsonar.jacoco.itReportPath=/proj/coverage-reports/jacoco-it.exec -Dsonar.java.coveragePlugin=jacoco -Dsonar.language=java

As you can see first the tests are executed through the verify phase, then the jacoco:dump goal retrieves the coverage data from the test server (I configured the server to run the JaCoCo agent) and at last the data is uploaded to my Sonar server.

The "strange" behaviour I'm having is that if I run this command on my computer and then on Jenkins (configuring the Jenkins project accordingly) in the SonarQube page I get different coverage results. Moreover, if I configure the Jenkins project and then I simply COPY it creating a new (but equivalent) Jenkins project, the results are different.

I tried different configurations and cases, but I cannot understand what the problem can be. Am I not considering some JaCoCo constraints (e.g. someting related to the Jenkins project name)?

few details - how do you pull the code to jenkins for the project and what is the result difference(how and where do you see that)? — Naman
– Naman, Commented Jan 17, 2017 at 16:43
@nullpointer The Jenkins project is configured to pull the code at each execution through the "Source Code Management" step. Example — gvdm
– gvdm, Commented Jan 17, 2017 at 16:49
There is no difference between the local and the pulled copy as no one pushes anything on the project — gvdm
– gvdm, Commented Jan 17, 2017 at 16:50
Couple of notes from me (as JaCoCo developer): class files that are used during report generation should match class files (be the same) that were instrumented on your test server for generation of jacoco-it.exec - check that. Drilling down in two different reports simultaneously you can pinpoint classes that lead to difference of the overall value - can be used for further debugging. — Godin
– Godin, Commented Jan 18, 2017 at 8:43
As was said earlier - remote server should run classes that later will be analyzed to generate report. Such match is required, because otherwise there is no guarantee that report shows exactly what was executed. See jacoco.org/jacoco/trunk/doc/classids.html for more details. And excuse me, but your setup is quite complex, so hard to tell precisely how to achieve this for it. — Godin
– Godin, Commented Jan 18, 2017 at 10:10

Community · Accepted Answer · 2017-05-23 11:52:55Z

As said in the question comments, the artifact deployed on the test server and the one compiled during the verify phase on which the report is generated must be exactly the same, so it is not enough that the code is the same.

To solve my problem I had to implement this workflow with Jenkins:

Do a mvn package on the project
Deploy the generated WAR on the remote server using Ansible (we already use Ansible for nightly deploys and other tasks on remote machines)
Run the remote tests without recompiling the wars. To do this I had to add the Maven flag -Dmaven.compiler.useIncrementalCompilation=false (thanks to this and this for the hints) in order to not re-compile the artifacts during the verify phase
Retrieve (dump) the JaCoCo coverage data

So the Maven command described in the question has been split in two commands: the one which creates the package and the one which performs the tests and retrieves the JaCoCo data without recompiling the artifacts.

Collectives™ on Stack Overflow

JaCoCo plugin gives different coverage

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related