Splitting Unit and Integration Tests using Maven and Surefire plugin

1. OVERVIEW

Generally, when you write Java unit tests, you stub data or mock the target class dependencies. This allows you to verify the business logic in an environment where you don’t depend on third party services.

Integration tests also play an important role in the Continuous Integration / Delivery (CI/CD) process to release software frequent and safe. As I have mentioned before, integration testing goal is to verify that interaction between different parts of the system work well. Examples of such interactions are:

Connecting to a RDBMS to retrieve or persist data
Consuming a message from a broker
Sending an email
Retrieving documents a NoSql data store like from Cosmos DB
Processing an upstream response

In large Enterprise applications development, with hundreds of unit and integration tests, the test suites take a considerable amount of time to complete, normally hours.

Jenkins build - maven-surefire-plugin - unit test fails

Wouldn’t it make sense to split the tests by their purpose and execution speed? It would be beneficial for an organization to get a quicker feedback when tests fail.

It would then be advisable to implement a fail-fast approach and break the build process when unit tests fail. This post covers configuring Maven’s maven-surefire-plugin to split running unit and integration tests.

2. REQUIREMENTS

Java 7+.
Maven 3.2+.

3. THE REST CONTROLLER, UNIT AND INTEGRATION TESTS

Assuming you have generated a Spring Boot 2 application using Spring Initializr, from command line or via your preferred IDE, let’s add a simple REST controller DemoController.java:

@RestController
@RequestMapping(value = "/api/entities")
public class DemoController {
...
  @GetMapping(value = "/{id}")
  public ResponseEntity<String> findEntity(@PathVariable String id) {
    String entity = this.someBusinessService.findEntity(id);
    return new ResponseEntity<>(entity, HttpStatus.OK);
  }
}

Let’s now add the unit and integration test classes meant to run during Maven’s test and integration-test phases.

DemoControllerTest.java:

package com.asimio.demo.rest;
...
@RunWith(MockitoJUnitRunner.class)
public class DemoControllerTest {

  @Mock
  private SomeBusinessService mockSomeBusinessService;
...
  @Test
  public void shouldRetrieveAnEntity() {
    // Given
    Mockito.when(this.mockSomeBusinessService.findEntity("blah")).thenReturn("meh");
    // When
    ResponseEntity<String> actualResponse = this.demoController.findEntity("blah");
    // Then
    Assert.assertThat(actualResponse.getStatusCode(), Matchers.equalTo(HttpStatus.OK));
    Assert.assertThat(actualResponse.getBody(), Matchers.equalTo("meh"));
  }
}

DemoControllerIT.java:

package com.asimio.demo.rest;
...
@RunWith(SpringRunner.class)
@ContextConfiguration(classes = { Application.class })
@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
public class DemoControllerIT {
...
  @Test
  public void shouldRetrieveAnEventById() {
    String body = RestAssured.
      given().
        accept(ContentType.JSON).
      when().
        get("/api/entities/blah").
      then().
        statusCode(HttpStatus.SC_OK).
        contentType(ContentType.JSON).
        extract().asString();

    Assert.assertThat(body, Matchers.equalTo("Retrieved entity with id: blah"));
  }
}

Note: Unit test classes are suffixed with Test and integration test classes with IT. I have kept them in the same package for simplicity, but it would be a good practice to place unit and integration tests in different folders.

4. CONFIGURING maven-surefire-plugin

There are now two unit test classes, DemoControllerTest.java and DefaultSomeBusinessServiceTest and two integration tests, DemoControllerIT.java and ApplicationTests. Let’s configure maven-surefire-plugin in pom.xml to split running them in the test and integration-test phases:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
...
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId> <!-- surefire plugin version managed by Spring Boot -->
  <configuration>
    <skipTests>true</skipTests>
  </configuration>
  <executions>
    <execution>
      <id>unit-tests</id>
      <phase>test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <skipTests>false</skipTests>
        <includes>
          <include>**/*Test.java</include>
        </includes>
      </configuration>
    </execution>
    <execution>
      <id>integration-tests</id>
      <phase>integration-test</phase>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <skipTests>false</skipTests>
        <includes>
          <include>**/*IT.*</include>
          <include>**/*Tests.*</include>
        </includes>
      </configuration>
    </execution>
  </executions>
</plugin>

5. RUNNING THE TESTS AND BUILDING THE ARTIFACTS

5.1. A SUCCESSFUL BUILD

Let’s first run and analyze a successful build:

mvn clean verify
...
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ springboot2-split-unit-integration-tests ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (unit-tests) @ springboot2-split-unit-integration-tests ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.asimio.demo.rest.DemoControllerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.647 s - in com.asimio.demo.rest.DemoControllerTest
[INFO] Running com.asimio.demo.service.DefaultSomeBusinessServiceTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in com.asimio.demo.service.DefaultSomeBusinessServiceTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
...
[INFO] --- maven-jar-plugin:3.1.1:jar (default-jar) @ springboot2-split-unit-integration-tests ---
[INFO] Building jar: /Users/ootero/Projects/bitbucket.org/springboot2-split-unit-integration-tests/target/springboot2-split-unit-integration-tests.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.4.RELEASE:repackage (repackage) @ springboot2-split-unit-integration-tests ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (integration-tests) @ springboot2-split-unit-integration-tests ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.asimio.demo.ApplicationTests
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.151 s - in com.asimio.demo.ApplicationTests
[INFO] Running com.asimio.demo.rest.DemoControllerIT
...
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
...

The verify phase also executes the test, package and integration-test phases among others. Read more at Maven’s Build Default Lifecycle Reference.

During the test phase execution, maven-surefire-plugin’s default-test execution was skipped. That’s the result of setting skipTests to true in pom.xml (lines 5 through 7). The unit-tests execution ran one test in each Test-suffixed class without failure.

Next the package phase ran and built the main artifact. Then spring-boot-maven-plugin replaced the main jar with an uber jar containing all the dependencies this application needs.

The integration-test phase ran right after. This time Maven runs test methods found in DemoControllerIT.java and ApplicationTests classes, ending in a successful build.

Let’s take a look at Maven’s target folder:

ls -1a target/ target/surefire-reports/
target/:
.
..
classes
generated-sources
generated-test-sources
maven-archiver
maven-status
springboot2-split-unit-integration-tests.jar
springboot2-split-unit-integration-tests.jar.original
surefire-reports
test-classes

target/surefire-reports/:
.
..
TEST-com.asimio.demo.ApplicationTests.xml
TEST-com.asimio.demo.rest.DemoControllerIT.xml
TEST-com.asimio.demo.rest.DemoControllerTest.xml
TEST-com.asimio.demo.service.DefaultSomeBusinessServiceTest.xml
com.asimio.demo.ApplicationTests.txt
com.asimio.demo.rest.DemoControllerIT.txt
com.asimio.demo.rest.DemoControllerTest.txt
com.asimio.demo.service.DefaultSomeBusinessServiceTest.txt

Tip: Read on to learn how to upload JaCoCo code coverage reports to SonarQube.

Notice the Maven Surefire reports. Stay tuned, I’ll cover uploading these reports and code coverage using JaCoCo to SonarQube in another post; without running the tests twice as I have seen happening when using Cobertura.

5.2. WHEN UNIT TESTS FAIL

Let’s cause a unit test to fail in DemoControllerTest.java as a second scenario:

    // Then
    Assert.assertThat(actualResponse.getStatusCode(), Matchers.equalTo(HttpStatus.OK));
-   Assert.assertThat(actualResponse.getBody(), Matchers.equalTo("meh"));
+   Assert.assertThat(actualResponse.getBody(), Matchers.equalTo("should-fail"));

And attempting to build again:

mvn clean verify
...
[INFO] --- maven-surefire-plugin:2.22.1:test (unit-tests) @ springboot2-split-unit-integration-tests ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.asimio.demo.rest.DemoControllerTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.857 s <<< FAILURE! - in com.asimio.demo.rest.DemoControllerTest
[ERROR] shouldRetrieveAnEntity(com.asimio.demo.rest.DemoControllerTest)  Time elapsed: 0.113 s  <<< FAILURE!
java.lang.AssertionError:

Expected: "should-fail"
     but: was "meh"
        at com.asimio.demo.rest.DemoControllerTest.shouldRetrieveAnEntity(DemoControllerTest.java:37)

[INFO] Running com.asimio.demo.service.DefaultSomeBusinessServiceTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in com.asimio.demo.service.DefaultSomeBusinessServiceTest
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
...

The test phase failed and prevented the build from running following phases. It’s a fail-fast approach. The artifact wasn’t built and integration tests didn’t run. Read more at Maven’s Build Default Lifecycle Reference.

6. CONCLUSION

It’s a good practice to add unit and integration tests when developing Enterprise applications. While a unit test suite should run in the order of milliseconds or seconds, integration tests take minutes if not hours.

Reseeding data, network traffic, starting Docker containers before executing a test or test suite will slow down your integration tests. It would make sense to split them when your test suites are large enough to have a faster feedback. Why? With a shorter build cycle, you get quicker feedback, allowing you to act faster, increasing your productivity.

Tip: Read on if you are interested in learning how to Collect and Report Code Coverage using Maven and JaCoCo plugin.

Tip: Read on to learn how to upload JaCoCo code coverage reports to SonarQube.

Thanks for reading and as always, feedback is very much appreciated. If you found this post helpful and would like to receive updates when content like this gets published, sign up to the newsletter.