About Gradle, Kotlin and Inner Fear.

When the Gradle team made the recent announcement for Kotlin support in Gradle, it set off quite a bit of emotional reaction in Groovy-land. Waiting for the emotions to die down before writing something, Dan Woods beat me to it with a great & balanced response. He already said more than I would have written on the technical side and there is no need repeating it. It is worthwhile reading his response first.

I sense an inner fear in the Apache Groovy community. I have sensed it for a while (long before this announcement). People don't really want to talk too much about it in public media, but the conversations are happening. They are happening at conferences and they are happening in the pub.  People who are seen in public as big stalwarts of the language, are involved in these conversations. There is a fear that Groovy as a language is dying. There is a (misconstrued) fear that Groovy is slow. There is an inner fear that all of the wonderful open-source projects written in Groovy will fall by the wayside. To a great extent this fear is also driven by lack of work - how job ads are out there actively asking for Groovy work. These job ads are also driven by perception of the recruitment industry. Yet there are jobs, definitely not as many as Java, but a lot of these jobs are through personal referral - insider information if you must. All of the jobs I have seen and heard of, all tend to be mostly with innovative, smaller companies.

Groovy-based work tends to be Grails. This framework remains the flagship in the Groovy ecosystem, pushing the limits with what can be done with static typing in Groovy, yet balancing it with the flexibility of dynamic typing. I hardly hear of any complaints regarding performance in deployed Grails apps. (I am sure there are, but I just don't get to hear about them). More work also seems to be coming from Ratpack, and I can tell you using the Ratpack Groovy wrapper is just far more satisfying that using plain Java.

The place for me where Groovy is really excelling in is testing. There are a number of frameworks out there that is based on Groovy or have excellent Groovy wrappers. Spock Framework of course will always be the first mentioned.  This for me remains the best unit-testing framework on the JVM, irrespective of the development lanaguage of the production code. There are a bunch others such as
Geb (for Web UIs) and AccuREST (consumer-driven APIs). Have a look at this set of slides for an idea of how great Groovy-based frameworks are for testing.

Cool Jvm Tools to Help you Test - Aylesbury Testers Version from Schalk Cronjé

Gradle is a next generation build tool. It is exploring next ways of building stuff, of expressing how build-and-deploy pipeline should work. In this evolution we are learning about what we as an industry want from future build tools. If it thus evolves over time that Kotlin shows that it is more than capable of helping shape the future of build tools, then so be it - we have tried and we have learnt.

There is a very good reason why Gradle started with Groovy - expressive & readable DSLs. With all the promise that Kotlin has as an upcoming language,  and as a user of Kotlin too,  it will need some ground-breaking changes to the language to beat Groovy in this respect.Kotlin is far better than Java in both expressiveness and readability, but it still does not match Groovy. (Kotlin has some features that are better than Groovy, but I am leaving that out for now as I am arguing Groovy's case).

The problem remains that Java programmers are a-dime-a-dozen (just like C# ones). It can make it hard to find the right, good technical people - a bit like looking for that needle in the haystack. When looking for polyglot people can be a challenge, as there are less of them. However, the chances are that they are the better ones. They are usually the ones that are willing to explore new ground, willing to think about the problem and willing to apply the correct language to solve the problem. There is too much Java (or C#) is the hammer and everything is a nail attitude with many people who have developer in their job titles. I would rather recruit (or work with) someone who is polyglot.

For me both Kotlin & Groovy are great languages for first-timers on the JVM. I know some people don't agree with me on this, but I think you can get quite a way on the JVM without knowing Java. I would rather start people off on either of these two languages than drown them in the very verbose syntax that is Java.

I can understand that there are inner fears in the Apache Groovy community and I can sympathise with that, but I think for it to grow, we need to keep on creating great DSLs around other libraries or write great DSLs with pure Groovy libraries to keep it alive. Beyond that these wrappers, libraries and frameworks need to be kept evangelised at non-Groovy conferences, otherwise they will remain little niche projects.

I want to applaud Dan Woods for addressing some of these community fears in public, I am looking forward to building more apps in Groovy and I am looking forward to using more Kotlin both inside and outisde of Gradle. I am also looking to talk more openly about these fears people have in public.

Edit: After I published this, Cedric Champeau  a Groovy committer, Gradle Inc employee and someone I have a lot of time for,  published his view on the matter. It's worth a read too.

So what about this halting problem in Gradle?

I have been in the software industry far too long (I think). I have worked and worked with and on various build systems over the years. For me they are like that member of the family no-one wants to talk about, yet it is that one person that somehow carries everyone else through. In the modern age with continuous delivery, this family member, the build tool, is becoming even more important.

Build tools are something I am always very interested in. I have written very complex an extensive build tooling in GNU Make that I never want to repeat in my life. I have hacked on SCons, played with Cons, walked a road with Ant, NAnt, MSBuild, Maven ... the list just carries on. When a new build tool comes out I like to play with it and evaluate it.

I believe build scripts should be very readable and easily maintainable. I also believe that build tools should be flexible for when I need it and the ability to construct software in a polyglot / heterogenous way without me having to fight it in very way.
 
It is for reasons like the above that in this modern age I have come to like Gradle as a build tool. I am also very interested in critisicisms of Gradle. I believe it is important that all opinions should be evaluated, for even if they seem to be wrong at first sight, they might have been formed in a context where they are absolutely correct. I have read a number of Gradle-negative articles the past two years or so. Some of them were just rants, some had good points. but one that really drew my attention of late, is this wiki page on Gradle by Jaroslav Tulach.

This page dicusses what the author considers to be a flawed design in Gradle. It is termed the halting problem and in over-simplified terms can be seen that one could start up a build script, which will never terminate.

I am from an engineering background and in engineering we don't care about perfect, we care about what works.My inititial reaction after reading the Tulach article was "So what?" I am not a computer scientist, I'm an engineer. This halting problem is an interesting dicussion, but how much does this computer scientist meandering really affect my delivery of quality software?

We get taught how to drive on public roads, how to be safe in doing it and to care about he safety of other road users. We don't have to follow those laws and guidelines. Sometimes you have to exceed the speed limit, sometime you have to drive in the wrong lane in order to solve a problem. (As a matter of fact we can drive how we want, but at some stage that behaviour will catch up with us either in the form of an accident, punishment by law or death). Does this mean road transport and road loaws are a bad thing? No, it is about what works for suatainable behaviour and having the flexivility to go outside of the rules when needed.

Since the article did like to compare Maven to Gradle, it should be pointed out that  it is very easy to get a Maven build to lock up via a badly crafted plugin (as is the case for Gradle). The only difference is that one that also cause halting in Gradle via a badly written build script. The author seems to argue that that is a good thing in Maven's XML. I would say So What if Groovy contains a while statement? Should I use it a build script? No, probably not, unless I really need to solve a very tricky problem which at the point in time, cannot be solved otherwise. If I search through all the build.gradle scripts on Github & Bitbucket, how many of them would contain a while statement? Probably none.
 
The author does make a reasonable point about security. It is easier to parse Maven XML to obtain a list of dependencies, that it would be to do from a Gradle script, but once more plugins are addd to Maven those bets are off to as one does not know what can be added through Maven third-party plugins without running the build!

For now though, in my context and the contexts and most other Gradle users, the halting problem does not matter. We know the rules of the road for Gradle. If we stick to the patterns, they will usually do what we need and the Gradle DSL will look like a DSL and not a Groovy script.

I think the one thing that many people are missing is that Maven and Gradle are not really comparable. Gradle falls into a new generation of build tools, a generation beyond that of Maven, Ant or Rake. It is trying to break new ground. There are going to be mistakes made and issues in designs will occur, but without those learning will not happen. Without learning there will be no progress. There will also come a day when there a better build tools that the Leiningen and Gradle generation.

In conclusion, please write more critical (not ranting) articles about Gradle, for then the tool (and other build tools in its generation) can be improved, but please make sure you have your facts right.

More Advanced Build Flows with Jenkins

A while ago I needed to orchestrate a much more complex build flow in Jenkins. Besides the normal build, which includes unit testing, there was a need to test performance, performance static analysis on the code and well as execute a full functional test run. The problem was that most of these things took hours to complete. As the start of this, a full functional test run, if done sequentially, took 60 hours. Luckily it was possible to partition the testing so that it could be run in parallel.

What was required is to create a flow that looked similar to below.

This is achieved via the Cloudbees Build Flow Plugin, which is free and open-source. The source code is on Github. I am not going to explain more about the basics of the plugin, that can be read for itself on the plugin wiki page. There is enough there to get someone started. However, when you want to achieve the above kind of flow you'll need to know a bit more.

The DSL is Groovy-based, so knowing the basics of the languages and especially Closure syntax is required.  The flow itself is controlled via the FlowDSL class. Furthermore calling build returns a JobInvocation instance and dependant on how parallel is called, it will either return a list or a map containing FlowState instances. The build results are actually stored in a Run instance which, after obtaining a JobInvocation instance is available via .build property-syntax. Knowing this difference is important as it will help you to extract build information later on.

Once you have the JobInvocation object you can obtain a number of useful bits of information

Creating Build Flow

The following code snippet illustrates how to create a build flow similar to the graphic depiction.

 

Define as much as possible as closures

This will delay execution until such time that is required. This is what parallel relies on too. buildFlow, staticAnalysisFlow and performanceFLow are examples.

Capturing build results within the closure

Notice how buildFlow will store results in buildResult. (The latter will become a JobInvocation instance after execution)

Executing multiple instances of the same job

It is relatively easy to execute multiple instances in parallel, each running with different parameters. This can be done with a simple Groovy Range and using the collect operation. parallel requires a list of closures to be passed to it, collect does that for you See how testFlow is defined above. We have even wrapped each build inside and ignore closure, so that unstable builds don't break the build pipeline.

Break down complex flows into smaller flows

Stringing lots of flow sections together, can be tricky to follow. In such cases break down the flows into smaller manageable sections. As staticAnalysisFlow took a very long time to complete, I wanted it to run parallel to everything else. I have therefore created a mainFlow to run alongside. In order to deal with the complexity of running performanceFlow in parallel with another flow which already has more parallel flows in it, I have broken it down into smaller flows, which are then strung back together. Strictly speaking the use of arrays and each are not necessarily, but when you have even for jobs involved, this style can sometimes be easier to read.

Collecting results from multiple parallel builds

For all of the aprtitioned test runs, I wanted to collect the build numbers so that it could be passed downstream. As mentioned earlier parallel returns a FlowState instance. It is a simple case of iterating through all of the builds and finding the lastBuild property which will give you a JobInvocation to work with.

Collecting everything

Once all of the builds have completed, the necessary information can be passed do a final downstream job, which in turn can aggragate information form all of the other jobs.

Build-flow Limitations

For me one of the drawbacks of the current implementation is the lack of Grape support. Sometimes one needs just a bit more decision-making capability than what stock Groovy will offer you. It would be helpful to be able to use @Grab to pull in some extra libraries.

It is also not possible to version the build flow script. It could definitely be useful to store it in source control and then update it before each run.  The best workaround for now is to either

• Create the build flow job via Jenkins CLI and version control the config.xml file
• Create the build via Gary Hale's gradle-jenkins-plugin and version control the build.gradle and template.xml files.
• There is no real way of testing the flow outside of Jenkins at present. This makes it hard to syntax check before hand. The best you can do is create a series of fast-completing mock jobs, to experiment with and once you are happy convert the flow to attach the real jobs that you need to link.

In conclusion

Even given the above mentioned limitations, the Build Flow Plugin is a powerful item in the quiver of any Jenkins crew. I hope that this article will help other towards some more complex real world orchestrations.

WS-Security Username Tokens in Groovy

The  simplest of WS-Security tokens:  just send a username and password in the SOAP header. Ignoring the usual arguments about how insecure this might be, there are a number of systems that actually utitlise this and if you are using groovy-wslite, you might need to add it. The following code snippet generates the appropriate XML fix can be placed in the header.

If you place this in a seperate closure, the just call it from your message closure as using mkp.yieldUnescaped.

Kicking off Multiple Processes from Groovy

The following is a code snippet for kicking of a number of similar processes from a Groovy script and waiting for them to finish. The stdout & stderr streams are discarded.

Normally creating interfaces with maps in Groovy when there are overloaded methods present, are non-trivial, but in this case we throw away all content. This leads to a simple nullAppender.

I just used touch as a simple illustration, replace it with whatever you need. If you want to execute your process in anotehr directory than the current worknig directory or want to customise the environment, remember that execute can take two parameters to do exactly that.

Remotely Monitor WSO2 Carbon Instance with Groovy

This is a quick Groovy script that demonstrates how to print the memory usage of any remote JVM.

// Replace HOSTNAME,PORT,USERNAME,PASSWORD as appropriate.

import groovy.jmx.builder.JmxBuilder 
def jmx = new JmxBuilder()
def conn = jmx.connectorClient (
    protocol:"rmi",
    url:"service:jmx:rmi:///jndi/rmi://HOSTNAME:PORT/server"
)

conn.connect([ 'jmx.remote.credentials' :  [ 'USERNAME','PASSWORD' ] as String[] ])

def mem = new GroovyMBean(    conn.getMBeanServerConnection(),'java.lang:type=Memory');

println "Heap:"
println "===="
println 'Committed: ' + mem.HeapMemoryUsage.committed
println '     Used: ' + mem.HeapMemoryUsage.used
println '      Max: ' + mem.HeapMemoryUsage.max

println "\nNon-heap:"
println "========"
println 'Committed: ' + mem.NonHeapMemoryUsage.committed
println '     Used: ' + mem.NonHeapMemoryUsage.used
println '      Max: ' + mem.NonHeapMemoryUsage.max

conn.close()

WSO2 Carbon:
In order to use this to monitor a WSO2 Carbon instance (ESB, Data Services etc.), edit repository/conf/carbon.xml. In <Ports> uncomment the <JMX> section. Restart the WSO2 Carbon process. The username used for access must have the admin role.