Speeding up Maven by parallel threads

/ / Blog, Digital Development en
Maven threads
Alfresco 5.0 CE is now available to the community
The DARA-Medieval Aragonese Documents project was presented today

When using Maven with very large projects the mere task of cleaning up work directories can be a long process that can halt work for a considerable time. There are already a whole bunch of tutorials on all of the steps that you can optimize to speed up the process, and so this article only deals with the use of multi threaded execution, an option which is available with Maven version 3.

 

Execution threads in Maven

When a task is launched in Maven it checks project dependencies in order to organize the modules according to the dependencies between them, after which it launches an execution thread that carries out the task in the module buffer prepared. Execution is linear although it can launch several threads to retrieve artifacts from repos. The number of threads can be configured by the property:

-Dmaven.artifact.threads=6

In the case of Maven, to use several execution threads the -T [number of threads] parameter is used, which can be the number of threads resulting from the number of CPU cores with [number of threads per core]C. For example:

mvn -T 4 clean install # 4 hilos de ejecución
mvn -T 1C clean install # 1 hilo de ejecución por cada núcleo
mvn -T 1.5C clean install # 1.5 hilos de ejecución por cada núcleo

 

Testing

To test time differences the project used comprises 76 Maven modules. The operation performed is ‘mvn clean install’ from the root, using the following script:

 

#!/bin/bash
function do_timing {
	echo Command: $1
	echo Command: $1>>timer.report
	(/usr/bin/time -f "Time: %E" $1 >> console.out)2>>timer.report
}
rm -rf console.out
rm -rf timer.report
do_timing "mvn clean install"
do_timing "mvn -T1 clean install"
do_timing "mvn -T2 clean install"
do_timing "mvn -T3 clean install"
do_timing "mvn -T4 clean install"
do_timing "mvn -T5 clean install"
do_timing "mvn -T6 clean install"
do_timing "mvn -T7 clean install"
do_timing "mvn -T8 clean install"
do_timing "mvn -T9 clean install"
do_timing "mvn -T10 clean install"
do_timing "mvn -T1C clean install"
do_timing "mvn -T2C clean install"
do_timing "mvn -T3C clean install"
do_timing "mvn -T4C clean install"
do_timing "mvn -T5C clean install"

 

The exact specifications of the test equipment are not particularly relevant as the start time with just one thread is taken as the reference. What is worth mentioning is that the system’s hard drive is SSD and during executions the CPU works at 100% so that there are not bottlenecks that would distort tests.

The results obtained are:

 

Command: mvn clean install
Time: 0:41.05
Command: mvn -T1 clean install
Time: 0:41.90
Command: mvn -T2 clean install
Time: 0:33.23
Command: mvn -T3 clean install
Time: 0:32.64
Command: mvn -T4 clean install
Time: 0:35.57
Command: mvn -T5 clean install
Time: 0:32.68
Command: mvn -T6 clean install
Time: 0:31.02
Command: mvn -T7 clean install
Time: 0:32.40
Command: mvn -T8 clean install
Time: 0:35.29
Command: mvn -T9 clean install
Time: 0:34.33
Command: mvn -T10 clean install
Time: 0:37.34
Command: mvn -T1C clean install
Time: 0:35.41
Command: mvn -T2C clean install
Time: 0:34.92
Command: mvn -T3C clean install
Time: 0:36.94
Command: mvn -T4C clean install
Time: 0:38.17
Command: mvn -T5C clean install
Time: 0:37.88

Tiempos de ejecución dependiendo del número de hilos

As you can see with 2 or more threads we obtain better times but these do not decrease if we increase the number of threads. Probably with a more complex project structure the results would be better, although a 20% improvement isn’t bad at all.

 

Bear in mind

Although times are considerably improved when we use this parameter, take care when using it because not all of the plugins are ‘thread safe’ and there is no guarantee that they will run correctly. One of the options to control this parameter is to use a Maven command alias with the thread parameter so it is only used it when it can be run safely. For example:

alias mvn-fast='alias mvn="mvn -T2 $1"'

which would determine the use of 2 threads in executions of this console session.

 


 

References:

Configuring Maven – http://maven.apache.org/guides/mini/guide-configuring-maven.html
Parallel builds in Maven 3 – https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3