What was required is to create a flow that looked similar to below.
This is achieved via the Cloudbees Build Flow Plugin, which is free and open-source. The source code is on Github. I am not going to explain more about the basics of the plugin, that can be read for itself on the plugin wiki page. There is enough there to get someone started. However, when you want to achieve the above kind of flow you'll need to know a bit more.
The DSL is Groovy-based, so knowing the basics of the languages and especially Closure syntax is required. The flow itself is controlled via the FlowDSL class. Furthermore calling
buildreturns a JobInvocation instance and dependant on how
parallelis called, it will either return a list or a map containing FlowState instances. The build results are actually stored in a Run instance which, after obtaining a
JobInvocationinstance is available via
.buildproperty-syntax. Knowing this difference is important as it will help you to extract build information later on.
Once you have the
JobInvocationobject you can obtain a number of useful bits of information
Creating Build FlowThe following code snippet illustrates how to create a build flow similar to the graphic depiction.
Define as much as possible as closuresThis will delay execution until such time that is required. This is what
parallelrelies on too.
Capturing build results within the closureNotice how
buildFlowwill store results in
buildResult. (The latter will become a
JobInvocationinstance after execution)
Executing multiple instances of the same jobIt is relatively easy to execute multiple instances in parallel, each running with different parameters. This can be done with a simple Groovy Range and using the
parallelrequires a list of closures to be passed to it,
collectdoes that for you See how
testFlowis defined above. We have even wrapped each
ignoreclosure, so that unstable builds don't break the build pipeline.
Break down complex flows into smaller flowsStringing lots of flow sections together, can be tricky to follow. In such cases break down the flows into smaller manageable sections. As
staticAnalysisFlowtook a very long time to complete, I wanted it to run parallel to everything else. I have therefore created a
mainFlowto run alongside. In order to deal with the complexity of running
performanceFlowin parallel with another flow which already has more parallel flows in it, I have broken it down into smaller flows, which are then strung back together. Strictly speaking the use of arrays and
eachare not necessarily, but when you have even for jobs involved, this style can sometimes be easier to read.
Collecting results from multiple parallel buildsFor all of the aprtitioned test runs, I wanted to collect the build numbers so that it could be passed downstream. As mentioned earlier
FlowStateinstance. It is a simple case of iterating through all of the builds and finding the
lastBuildproperty which will give you a
JobInvocationto work with.
Collecting everythingOnce all of the builds have completed, the necessary information can be passed do a final downstream job, which in turn can aggragate information form all of the other jobs.
For me one of the drawbacks of the current implementation is the lack of Grape support. Sometimes one needs just a bit more decision-making capability than what stock Groovy will offer you. It would be helpful to be able to use @Grab to pull in some extra libraries.
It is also not possible to version the build flow script. It could definitely be useful to store it in source control and then update it before each run. The best workaround for now is to either
- Create the build flow job via Jenkins CLI and version control the config.xml file
- Create the build via Gary Hale's gradle-jenkins-plugin and version control the build.gradle and template.xml files.
- There is no real way of testing the flow outside of Jenkins at present. This makes it hard to syntax check before hand. The best you can do is create a series of fast-completing mock jobs, to experiment with and once you are happy convert the flow to attach the real jobs that you need to link.
Even given the above mentioned limitations, the Build Flow Plugin is a powerful item in the quiver of any Jenkins crew. I hope that this article will help other towards some more complex real world orchestrations.