Post-deployment integration tests in Maven

The default Maven lifecycle provides various kinds of test phases. The test phase is probably the most used and is responsible for executing the application unit tests via the Maven Surefire Plugin. Besides, there is also the integration-test phase which uses the Maven Failsafe Plugin. Both of these phases will be executed before the install and deploy stages, so before the application is actually deployed on a (dev/test-)environment.

Besides these default Maven testing phases, we would also like to perform some integration/smoke-tests after deployment. For example to test if changes that are requested through an API, are actually being committed into the database (or other kind of backend system). That’s what we’re having a look at in this blog post.

Not compiling more than required

The main goal was to have the test-sources for these post-deployment integration tests close to the other (unit) test-sources, but that these resources (that are a lot “heavier” than the regular unit tests resources) would not have any impact on the regular Maven test phase.

Assume we have a project setup like the following:

src/
├─ main/
│  └─ ...
└─ test/
   ├─ java/
   │  ├─ integration/
   │  │  └─ IntegrationTest.java
   │  └─ UnitTest.java
   └─ resources/
      ├─ integration/
      │  └─ HeavyIntegrationTestFile.txt
      └─ LightUnitTestFile.txt

When running the regular unit tests, we would want to exclude the “integration” package/folder from both the test-sources and test-resources directories:

src/
├─ main/
│  └─ ...
└─ test/
   ├─ java/
   │  └─ UnitTest.java
   └─ resources/
      └─ LightUnitTestFile.txt

On the other hand, when running the integration tests, we want only these directories to be included. There’s also no need to compile the src/main sources, since we’re going to black-box test the application that was already deployed to an actual environment in an earlier step of the CI/CD pipeline:

src/
└─ test/
   ├─ java/
   │  └─ integration/
   │     └─ IntegrationTest.java
   └─ resources/
      └─ integration/
         └─ HeavyIntegrationTestFile.txt

Excluding the integration test files in regular unit test runs

As described above, the desired behavior for regular unit test runs is fairly simple. First we want to exclude the "integration" package from the src/test/java folder. To exclude this package from being compiled (and thus executed) we can configure a testExcludes filter in the Maven Compiler Plugin.

Next, we also want to exclude the "integration" directory from the src/test/resources folder. To achieve this, we’ll add another exclusion filter, but now for the Maven Resources Plugin.

This will look something like the following:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.10.1</version>
            <configuration>
                <testExcludes>**/integration/*</testExcludes>
            </configuration>
        </plugin>
    </plugins>
    <testResources>
        <testResource>
            <directory>src/test/resources</directory>
            <excludes>
                <exclude>integration/*</exclude>
            </excludes>
        </testResource>
    </testResources>
</build>

When running the tests via the “mvn clean test” command, we’ll see that only the UnitTest class is being executed. Also, when inspecting the “target” folder, we see that none of the integration test resources are compiled/included:

target/
├─ classes/
│  └─ ...
└─ test-classes/
   ├─ UnitTest.class
   └─ LightUnitTestFile.txt

Adding a Maven profile for the integration tests

Now that we have the correct behavior for the regular unit test runs, it’s time to get the integration tests to work as well. To override the behavior that we configured in the previous steps for the integration tests, we start by adding a Maven profile:

<profiles>
    <profile>
        <id>integration-test</id>
        ...
    </profile>
</profiles>

Here we will override the Maven compiler plugin configuration. This time the entire main build will be skipped, since we’re only interested in compiling the test sources. Now, instead of excluding the “integration” directory, we explicitly include this folder and all of its contents.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <skipMain>true</skipMain>
        <testIncludes>**/integration/**</testIncludes>
        <testExcludes combine.self="override"/>
    </configuration>
</plugin>

Please note that we also explicitly tell the plugin that the exclude filters should be overridden when combining the root- and plugin configuration. If we wouldn’t do this, the effective plugin configuration (which can be shown via “mvn help:effective-pom -P integration-test”) would look like:

<configuration>
  <skipMain>true</skipMain>
  <testIncludes>**/integration/**</testIncludes>
  <testExcludes>**/integration/*</testExcludes>
</configuration>

If both testIncludes and testExcludes are set, testExcludes takes precedence. This is why both the testIncludes and overriding of the testExcludes are important in the profile configuration!

Unfortunately, the testResources block does not support anything like overriding behavior. Similar to the above example with the effective pom, the configuration from the profile will just be concatenated, resulting in contradicting inclusions- and exclusions. So here we had to look for a different solution.

Because properties can be properly overridden in profiles, we decided to move the configuration which we need to override, to property values and overriding these in the profile.

This brings the total added configuration to the following:

<properties>
    <test.resources.directory>src/test/resources</test.resources.directory>
    <test.resources.targetPath>${project.build.testOutputDirectory}</test.resources.targetPath>
    <test.resources.exclude>integration/*</test.resources.exclude>
</properties>

<profiles>
    <profile>
        <id>integration-test</id>
        <properties>
            <test.resources.directory>src/test/resources/integration</test.resources.directory>
            <test.resources.targetPath>integration</test.resources.targetPath>
            <test.resources.exclude/>
        </properties>
        <build>
            <directory>target/integration-test</directory>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <configuration>
                        <skipMain>true</skipMain>
                        <testIncludes>**/integration/**</testIncludes>
                        <testExcludes combine.self="override"/>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.10.1</version>
            <configuration>
                <testExcludes>**/integration/*</testExcludes>
            </configuration>
        </plugin>
    </plugins>
    <testResources>
        <testResource>
            <directory>${test.resources.directory}</directory>
            <targetPath>${test.resources.targetPath}</targetPath>
            <excludes>
                <exclude>${test.resources.exclude}</exclude>
            </excludes>
        </testResource>
    </testResources>
</build>

When running the tests via the “mvn clean test” command, we will see exactly the same behavior as before. However, if we run the tests via “mvn clean test -P integration-test”, we see that instead of the UnitTest, now only the IntegrationTest is being executed. When inspecting “target” folder again, we’ll now see that it only contains the integration test resources:

target/
├─ integration-test/
│  └─ classes/
│     └─ <EMPTY>
└─ test-classes/
   ├─ integration/
   │  └─ HeavyIntegrationTestFile.txt
   └─ IntegrationTest.class

Please also note the extra subdirectory “integration-test” within the “target” folder; this is because we also configured the build/directory field within the Maven profile. The positive effect is that if you would compile the entire application via “mvn package” and do a “mvn clean test -P integration-test” afterwards, the “clean” in the latter command will only remove the “integration-test” subdirectory, so not removing the compiled code from the first command.

Conclusion

Using Maven profiles gives us sufficient possibilities to fully separate the unit tests from the integration tests, even though they are in the same Maven project. The simplest part was to configure which classes (not) to include in both cases in the compilation phase.

Figuring out how we could achieve the same for the resource files was a bit more challenging, but by using properties we were also able to get the exact behavior we were aiming for.

A full working example can be found on the Integration & Application Talents Bitbucket.