More Advanced Build Flows with Jenkins

A while ago I needed to orchestrate a much more complex build flow in Jenkins. Besides the normal build, which includes unit testing, there was a need to test performance, performance static analysis on the code and well as execute a full functional test run. The problem was that most of these things took hours to complete. As the start of this, a full functional test run, if done sequentially, took 60 hours. Luckily it was possible to partition the testing so that it could be run in parallel.

What was required is to create a flow that looked similar to below.

This is achieved via the Cloudbees Build Flow Plugin, which is free and open-source. The source code is on Github. I am not going to explain more about the basics of the plugin, that can be read for itself on the plugin wiki page. There is enough there to get someone started. However, when you want to achieve the above kind of flow you'll need to know a bit more.

The DSL is Groovy-based, so knowing the basics of the languages and especially Closure syntax is required.  The flow itself is controlled via the FlowDSL class. Furthermore calling build returns a JobInvocation instance and dependant on how parallel is called, it will either return a list or a map containing FlowState instances. The build results are actually stored in a Run instance which, after obtaining a JobInvocation instance is available via .build property-syntax. Knowing this difference is important as it will help you to extract build information later on.

Once you have the JobInvocation object you can obtain a number of useful bits of information

Creating Build Flow

The following code snippet illustrates how to create a build flow similar to the graphic depiction.

 

Define as much as possible as closures

This will delay execution until such time that is required. This is what parallel relies on too. buildFlow, staticAnalysisFlow and performanceFLow are examples.

Capturing build results within the closure

Notice how buildFlow will store results in buildResult. (The latter will become a JobInvocation instance after execution)

Executing multiple instances of the same job

It is relatively easy to execute multiple instances in parallel, each running with different parameters. This can be done with a simple Groovy Range and using the collect operation. parallel requires a list of closures to be passed to it, collect does that for you See how testFlow is defined above. We have even wrapped each build inside and ignore closure, so that unstable builds don't break the build pipeline.

Break down complex flows into smaller flows

Stringing lots of flow sections together, can be tricky to follow. In such cases break down the flows into smaller manageable sections. As staticAnalysisFlow took a very long time to complete, I wanted it to run parallel to everything else. I have therefore created a mainFlow to run alongside. In order to deal with the complexity of running performanceFlow in parallel with another flow which already has more parallel flows in it, I have broken it down into smaller flows, which are then strung back together. Strictly speaking the use of arrays and each are not necessarily, but when you have even for jobs involved, this style can sometimes be easier to read.

Collecting results from multiple parallel builds

For all of the aprtitioned test runs, I wanted to collect the build numbers so that it could be passed downstream. As mentioned earlier parallel returns a FlowState instance. It is a simple case of iterating through all of the builds and finding the lastBuild property which will give you a JobInvocation to work with.

Collecting everything

Once all of the builds have completed, the necessary information can be passed do a final downstream job, which in turn can aggragate information form all of the other jobs.

Build-flow Limitations

For me one of the drawbacks of the current implementation is the lack of Grape support. Sometimes one needs just a bit more decision-making capability than what stock Groovy will offer you. It would be helpful to be able to use @Grab to pull in some extra libraries.

It is also not possible to version the build flow script. It could definitely be useful to store it in source control and then update it before each run.  The best workaround for now is to either

• Create the build flow job via Jenkins CLI and version control the config.xml file
• Create the build via Gary Hale's gradle-jenkins-plugin and version control the build.gradle and template.xml files.
• There is no real way of testing the flow outside of Jenkins at present. This makes it hard to syntax check before hand. The best you can do is create a series of fast-completing mock jobs, to experiment with and once you are happy convert the flow to attach the real jobs that you need to link.

In conclusion

Even given the above mentioned limitations, the Build Flow Plugin is a powerful item in the quiver of any Jenkins crew. I hope that this article will help other towards some more complex real world orchestrations.

Automated Tests and Deployment are Real Code

Dale Emery has posted a great set of slides. I would recommend this to all software developers, testers, business analysts and everyone in the software management food chain to read.

Slide #9 from Dale's deck

This has reminded ne of a conversation with a tester not too long ago. I was doing some work with him on their test system. We pair-programmed the change - it was his first experience in pairing. It was the also only way we could achieve this tricky modification. The system has grown into a big code base. It was not well documented, apart from some Word documents that were not up to date. There was also a number of methods that mostly did the same thing.

I explained to him that the test system itself should be modular and unit-tested. His jaw dropped: "We should have tests for our tests?". "Indeed", I replied, "you are using this to functionally test the product as if it is installed in a live environment. You need to have the confidence that changes you make will not adversely affect the results."

This was an eye-opener for him.  He has never considered test code to be real code. Surely we sometimes do things in test code that we won't do in production code, but that is intentional as the code is fit for purpose. It still needs to be maintained and structured properly for it will probably live as long as the product itself would. Just as there is a cost to not doing TDD on any sizeable production of code, there is a cost to not doing it for any sizeable test code base. This test code will probably live as long as the product itself, therefore it needs to be maintained with the same fervour as the production code. Good developers understand this about unit tests, but I am seeing that the biggest breakdown still occurs where people need to build test systems to perform functional or integration testing.

The same applies for your deployment code. The DevOps movement tells us that infrastructure is code. Not only do you need to look after your production code and test code, you also need to look after your deployment code with the same zeal. It is hard to do, it requires discipline, but that discipline is what is going to allow you to relax on the weekend, instead of fighting fires in the office. I would call that prosperity:

Agile teams - Prosperity Focused, Values-driven from Schalk Cronjé

I have recently heard of a team that used to work to the mantra of building walking skeletons first, including building the skeleton deployment code first.  Of late, they have been told to manually deploy because THERE_IS_NO_TIME_FOR_WRITING_AUTO_DEPLOYMENT code. Now that is just wrong. It is a mutation of Dale's Automation Last Zombie (see slide #12 of Dale's deck).  We can even call it the Deployment Last Zombie. I am sure that team will pay for this short-term thinking pressed upon them.

Just as test-driven development has done a lot for design and testability we really need Deployment-driven development. Dare I call it DDD? Preferably not, there are too many buzz acronyms around already. Regardless of what we call it, this is what DevOps and the next level of effective software development brings to the table.

Just remember, treat your test automation code and your deployment code with the same respect that you treat your production code with.