About Maven, SDKMAN! and Community Insularity

The email thread on publishing Maven releases on SDKMAN  has just managed to press so many buttons on  the emotive side.


One cannot judge the Maven developer community on one email thread alone. Unfortunately this thread portrays them as an insular group, inward looking and not really understanding a world beyond the one thing that Maven does exceptionally well - building Java apps for enterprise. I have dealt with too many behavioural pathologies in organisations, not to miss seeing this is as being in the same class. But let's just have a look at this in context as to how it started:

I am probably not going to be 100% objective here. I know Marco Vermeulen, the main instigator of SDKMAN! very well. I know his background, how he approaches things and his views in life. He is just an all-round, guitar-playing, good guy, who wants communities to benefit from good stuff.

I also use the SDKMAN! tool nearly everyday. It solves a great number of developer problem for me. I think people misjudge SDKMAN! as simply a package manager.

Call me an anarchist, but I never install Homebrew on any of my Macs. I still use MacPorts for some utilities that I need, but I don't brew stuff. Neither on Linux would I use apt-get, urpmi or yum to install tools like Maven, Gradle or even the Ceylon SDK. Why not? Because all of these tools assume one global version for each package. Even though some tools allow you to manage multiple versions, having them active at the same time are far from a trivial task. Even if the one global version is a good paradigm for system tools, for development tools it is not.

SDKMAN! allows each person on a computer to manage tools and SDK on an individual basis and even have different versions running in different console windows! This is huge for developer productivity.  Let's say I am seeing strange behaviour with Maven 3.5.0 or Gradle 3.4 I simply have to do

sdk use maven 3.5.0

OR

sdk use gradle 3.3

and that version will become available on my path. I can then quickly diagnose whether it is behaviour inducesd by a later version of the tool or my own stupidity. This is but one of the  simple little things where SDKMAN! is so much more powerful for developers than package management too. (Obviously I am not going to install the latest coreutils package on Linux using SDKMAN! - that is what a package manager is for).

Furthermore Maven experts would recognise that using Maven Wrapper is a far superior solution in most cases than having brew install maven. This is similar to the Gradle Wrapper in that it allows for controlling the version of the build tool within the source repository. This is a far better for reproducible builds.

If you are wondering whether you need SDKMAN! if you are using wrappers, then the quick answer is no. The correct answer is yes, you might need it. Consider that you have a project where the Gradle wrapper is at 2.12, but you want to test the build to see what it will do with say Gradle 3.5 all I need to do is

sdk use gradle 3.5

gradle build

as opposed to the usual

./gradlew build

(I've used Gradle and not Maven as an example here, as there are far more versions available through SDKMAN! than  for Maven. The folks at Gradle groks what SDKMAN! is about). A little research, a little bit of asking around might just be better than shooting from the hip like one contributor to the conversation:

This is just arrogance. My immediate perception is we have someone here, who did not even bother to research what SDKMAN! is, and bound by whatever restricts his viewpoint, simply slags off the product. This is way uncool. It reeks of insularity in this community (I can only hope it is an attitude of a minority).

Then there is the release process:

Granted there has to be some manual steps involved in releasing software under the Apache Software Foundation, but does the Maven release process really need 30 steps? Apache Groovy don't need this many and their publication to SDKMAN! is automated.  This tells me two things, which I admit I can be wrong about:

• Firstly the Maven developer community is too inward looking
• Secondly Maven as a build tool is simply not good enough to automate at scale beyond building Java projects. (I know you can try to build other stuff with it, but do you still have a good quality of life after trying to do that at scale?)

I am letting Marco have a final say on that thread

 

He came with good intentions upon an invite and you the Maven tool development community have hated him for it. You need to go back and look at yourselves and look at your process, because from where I stand there is something most definitely wrong somewhere. Maven has been a revolutionary change to software development efficiency when it came about, but where are you as custodians standing now?

You are obviously at right to keep your process and your community the way you want, but if you want to close your borders don't expects tourists and investors to come knocking.  They will go elsewhere and you'll remain a closed an insular community.

P.S. Leiningen, Kobalt, Gradle, Ant, SBT are all on SDKMAN! You can even manage JDK versions with it. If you consider Maven is too precious to mix with the rest, let it be so.

About Gradle, Kotlin and Inner Fear.

When the Gradle team made the recent announcement for Kotlin support in Gradle, it set off quite a bit of emotional reaction in Groovy-land. Waiting for the emotions to die down before writing something, Dan Woods beat me to it with a great & balanced response. He already said more than I would have written on the technical side and there is no need repeating it. It is worthwhile reading his response first.

I sense an inner fear in the Apache Groovy community. I have sensed it for a while (long before this announcement). People don't really want to talk too much about it in public media, but the conversations are happening. They are happening at conferences and they are happening in the pub.  People who are seen in public as big stalwarts of the language, are involved in these conversations. There is a fear that Groovy as a language is dying. There is a (misconstrued) fear that Groovy is slow. There is an inner fear that all of the wonderful open-source projects written in Groovy will fall by the wayside. To a great extent this fear is also driven by lack of work - how job ads are out there actively asking for Groovy work. These job ads are also driven by perception of the recruitment industry. Yet there are jobs, definitely not as many as Java, but a lot of these jobs are through personal referral - insider information if you must. All of the jobs I have seen and heard of, all tend to be mostly with innovative, smaller companies.

Groovy-based work tends to be Grails. This framework remains the flagship in the Groovy ecosystem, pushing the limits with what can be done with static typing in Groovy, yet balancing it with the flexibility of dynamic typing. I hardly hear of any complaints regarding performance in deployed Grails apps. (I am sure there are, but I just don't get to hear about them). More work also seems to be coming from Ratpack, and I can tell you using the Ratpack Groovy wrapper is just far more satisfying that using plain Java.

The place for me where Groovy is really excelling in is testing. There are a number of frameworks out there that is based on Groovy or have excellent Groovy wrappers. Spock Framework of course will always be the first mentioned.  This for me remains the best unit-testing framework on the JVM, irrespective of the development lanaguage of the production code. There are a bunch others such as
Geb (for Web UIs) and AccuREST (consumer-driven APIs). Have a look at this set of slides for an idea of how great Groovy-based frameworks are for testing.

Cool Jvm Tools to Help you Test - Aylesbury Testers Version from Schalk Cronjé

Gradle is a next generation build tool. It is exploring next ways of building stuff, of expressing how build-and-deploy pipeline should work. In this evolution we are learning about what we as an industry want from future build tools. If it thus evolves over time that Kotlin shows that it is more than capable of helping shape the future of build tools, then so be it - we have tried and we have learnt.

There is a very good reason why Gradle started with Groovy - expressive & readable DSLs. With all the promise that Kotlin has as an upcoming language,  and as a user of Kotlin too,  it will need some ground-breaking changes to the language to beat Groovy in this respect.Kotlin is far better than Java in both expressiveness and readability, but it still does not match Groovy. (Kotlin has some features that are better than Groovy, but I am leaving that out for now as I am arguing Groovy's case).

The problem remains that Java programmers are a-dime-a-dozen (just like C# ones). It can make it hard to find the right, good technical people - a bit like looking for that needle in the haystack. When looking for polyglot people can be a challenge, as there are less of them. However, the chances are that they are the better ones. They are usually the ones that are willing to explore new ground, willing to think about the problem and willing to apply the correct language to solve the problem. There is too much Java (or C#) is the hammer and everything is a nail attitude with many people who have developer in their job titles. I would rather recruit (or work with) someone who is polyglot.

For me both Kotlin & Groovy are great languages for first-timers on the JVM. I know some people don't agree with me on this, but I think you can get quite a way on the JVM without knowing Java. I would rather start people off on either of these two languages than drown them in the very verbose syntax that is Java.

I can understand that there are inner fears in the Apache Groovy community and I can sympathise with that, but I think for it to grow, we need to keep on creating great DSLs around other libraries or write great DSLs with pure Groovy libraries to keep it alive. Beyond that these wrappers, libraries and frameworks need to be kept evangelised at non-Groovy conferences, otherwise they will remain little niche projects.

I want to applaud Dan Woods for addressing some of these community fears in public, I am looking forward to building more apps in Groovy and I am looking forward to using more Kotlin both inside and outisde of Gradle. I am also looking to talk more openly about these fears people have in public.

Edit: After I published this, Cedric Champeau  a Groovy committer, Gradle Inc employee and someone I have a lot of time for,  published his view on the matter. It's worth a read too.

So what about this halting problem in Gradle?

I have been in the software industry far too long (I think). I have worked and worked with and on various build systems over the years. For me they are like that member of the family no-one wants to talk about, yet it is that one person that somehow carries everyone else through. In the modern age with continuous delivery, this family member, the build tool, is becoming even more important.

Build tools are something I am always very interested in. I have written very complex an extensive build tooling in GNU Make that I never want to repeat in my life. I have hacked on SCons, played with Cons, walked a road with Ant, NAnt, MSBuild, Maven ... the list just carries on. When a new build tool comes out I like to play with it and evaluate it.

I believe build scripts should be very readable and easily maintainable. I also believe that build tools should be flexible for when I need it and the ability to construct software in a polyglot / heterogenous way without me having to fight it in very way.
 
It is for reasons like the above that in this modern age I have come to like Gradle as a build tool. I am also very interested in critisicisms of Gradle. I believe it is important that all opinions should be evaluated, for even if they seem to be wrong at first sight, they might have been formed in a context where they are absolutely correct. I have read a number of Gradle-negative articles the past two years or so. Some of them were just rants, some had good points. but one that really drew my attention of late, is this wiki page on Gradle by Jaroslav Tulach.

This page dicusses what the author considers to be a flawed design in Gradle. It is termed the halting problem and in over-simplified terms can be seen that one could start up a build script, which will never terminate.

I am from an engineering background and in engineering we don't care about perfect, we care about what works.My inititial reaction after reading the Tulach article was "So what?" I am not a computer scientist, I'm an engineer. This halting problem is an interesting dicussion, but how much does this computer scientist meandering really affect my delivery of quality software?

We get taught how to drive on public roads, how to be safe in doing it and to care about he safety of other road users. We don't have to follow those laws and guidelines. Sometimes you have to exceed the speed limit, sometime you have to drive in the wrong lane in order to solve a problem. (As a matter of fact we can drive how we want, but at some stage that behaviour will catch up with us either in the form of an accident, punishment by law or death). Does this mean road transport and road loaws are a bad thing? No, it is about what works for suatainable behaviour and having the flexivility to go outside of the rules when needed.

Since the article did like to compare Maven to Gradle, it should be pointed out that  it is very easy to get a Maven build to lock up via a badly crafted plugin (as is the case for Gradle). The only difference is that one that also cause halting in Gradle via a badly written build script. The author seems to argue that that is a good thing in Maven's XML. I would say So What if Groovy contains a while statement? Should I use it a build script? No, probably not, unless I really need to solve a very tricky problem which at the point in time, cannot be solved otherwise. If I search through all the build.gradle scripts on Github & Bitbucket, how many of them would contain a while statement? Probably none.
 
The author does make a reasonable point about security. It is easier to parse Maven XML to obtain a list of dependencies, that it would be to do from a Gradle script, but once more plugins are addd to Maven those bets are off to as one does not know what can be added through Maven third-party plugins without running the build!

For now though, in my context and the contexts and most other Gradle users, the halting problem does not matter. We know the rules of the road for Gradle. If we stick to the patterns, they will usually do what we need and the Gradle DSL will look like a DSL and not a Groovy script.

I think the one thing that many people are missing is that Maven and Gradle are not really comparable. Gradle falls into a new generation of build tools, a generation beyond that of Maven, Ant or Rake. It is trying to break new ground. There are going to be mistakes made and issues in designs will occur, but without those learning will not happen. Without learning there will be no progress. There will also come a day when there a better build tools that the Leiningen and Gradle generation.

In conclusion, please write more critical (not ranting) articles about Gradle, for then the tool (and other build tools in its generation) can be improved, but please make sure you have your facts right.

Mageia 4 on Lenovo G550

I am in the process of upgrading a number of  Lenovo G550 laptops to Mageia 4. I have been a Mandriva user since the 90s, but switched to Mageia around the version 2 release as Mandriva simply could not cut it with regards to stability. I hope that these notes might help others solving some issues on these laptops

Network errors on eth0:

If the ethernet adaptor according to lspci is "Ethernet controller: Broadcom Corporation NetLink BCM5906M Fast Ethernet PCI Express (rev 02)" then set the MTU=1000 (in contrast to default of 1500) as per a previous posting.

BCM4312 wifi driver:

If you are unlucky still to have Broadcom wireless hardware in your G550, then some work lies ahead. This specific hardware identifies itself to lspci as

Network controller: Broadcom Corporation BCM4312 802.11b/g LP-PHY (rev 01)

Mandriva gives you the option to use the new dkms-broadcom-wl package, but it simply did not want to install on the default 2.6.39 kernel due to DKMS build failures. I decided to settle for using the firmware-dependent b43 driver instead. For this the b43-fwcutter package had to be installed and the firmware downloaded from openwrt and unpacked.

Edit /etc/modprobe.conf and add the line

options b43 pio=1 qos=0 

(Src: OpenSuse Forums)

As root, run the following commands

b43-fwcutter -w /lib/firmware /path/to/unpacked-broadcom-wl/linux/wl_apsta.o
modprobe -r b43
modprobe b43

Finally setting MTU=1000 as was the case for the ethernet hardware, solved the dropping of packets.

Lacklustre performance:

If you are running KDE, uninstall kmail (there are better email clients), virtuoso & nepomuk.

Faking it with Gradle Repositories

There are days with Gradle when you need to pull files from a location that is not a Maven or Ivy repository. Gradle does provide a flatDir repository, but that is restricted in that files have to be local and they can only be referenced as :artifactId:version. As a regular Gradle user you probably want to use the group:artifactId:version or group:artifactId:version:classifier@ext format.

What if you want to pull from say Sourceforge, Google Projects or a legacy build artifact server ? There used to be a repositories plugin for Gradle, but it no longer seems to work with later Gradle versions. The good news is that it is possible in many cases to achieve this with Ivy + pattern layout.

Ivy usually requires an XML file (Ivy file) to describe its transitive dependencies, but in Gradle if the Ivy file does not exist, then it is simply assumed that there are no transitive dependencies. This is good for working with arbitrary artifact stores as they will not have such files. Managing transitive dependencoes are now up to you, the build script author, but you probably know this anyway as you are already doing something that is not of the ordinary.

In concept.gradle the artifact pattern string contains a number of fields. Those in square brackets are field names, those in parenthesis are optional sections.

To get a better understanding of what these fields mean, I suggest reading the Ivy terminology page. For now let's just see how the standard Mavane items you are familiar with in Gradle will map to these fields.

• group - This maps to organisation.
• artifactId - Effectively maps to both module and artifact. Strictly speaking not the same, but in practice this is usuaaly how it works out.
• version - Maps to revision
• classifier - This seems to be seldom used, is usually optional, but maps to classifier.
• ext - Refers to the extension of the artifact. Is optional and maps to ext.

How would you go about mapping this in practice? Let's assume you have a project that is dependent on libusb on SourceForge. The latter has a flexible folder structure, so once the Files section has been inspected one can write an appropriate pattern as is shown in sourceforge.gradle

Notice that the ext field is not in parenthesis, meaning that in this case I expect the extension to be passed.

Finally to prove that it work we can create a little build script tryIt.gradle and running it with gradle --info -b tryIt.gradle tryIt should provide output similar to output.txt.

Reviewing User Stories - Rise of the Quality Factors

Many people are familiar with Bill Wake's INVEST acronym when it comes to user stories. Even though I've heard a number of criticisms launched at conferences, from my experience it is still very popular in current teaching material.

Althought INVEST can be used to review user stories, when I look at the quality aspects of a process overall I would like to have a different approach. With user stories in each various incantations being very popular in the Agile world, I would therefore want to be able to review from an outsider's point of view, possibly that of a coach or consultant.

My friends Ashish Misra and Aditya Garg have come up with what I consider to be an excellent list [1]. I have deemed to name it the User Story Quality Factors:

Completeness: Any team or organisation will develop a way of capturing user stories. This might be internalised or published in a similar fashion to a Definition of Done. Therefore does te story have all the information it is supposed to have? Does it have traceability from previous documentation or discussions?

Consistency: The way the specification is formatted, the language used and the way the requirements are presented should be consistent throughout.

Ambiguity: Is it free of any ambiguous statements?

Specific:  Is it free of generalised statements?

Realisable: Does the requirement make sense and is it possible? Has all of the information been included in order to deliver the product?

Testable: If the user acceptance criteria is not testable it cannot be built with confidence.

Traceable: Who wants this requirement and why is it needed, what business strategy does it support?

Measurable: How are you going to measure if the requirements have actually been delivered and are operational to specification? Quantification applies to both user acceptance criteria (that which tell you that you can release) and success criteria (that which tells you that your release is a success in the market). If I don't see this I usually teach people the basic of Planguage - Name, Scale, Meter & Goal.

Acceptable: Is the use of calculations, language, logic and formulas correct?

Achievable: Is the user story achievable within an sprint (iteration-based) or a acceptable number of cadences (iteration-less)? If not, then consideration must be given to splitting the user story in to smaller units that still deliver value.

Independent: It is best if stories are independent. Sometimes this is not completly possible, thus one-way directional dependencies are acceptable. Circular and two-way dependencies are not allowed, otherwise prioritisation and planning problems will occur. The latter will also indicate that there are other issues within the context of a project or business value increment.


[1] By their own admission, Ashish & Adi would not declare full originality to the idea, but rather that it has been a fusion of ideas, influenced by the streams of time. Maybe a similar list already exists in another publication. If anyone reads this and can point me to that, then please do so - appropriate credit needs to be given.

More Advanced Build Flows with Jenkins

A while ago I needed to orchestrate a much more complex build flow in Jenkins. Besides the normal build, which includes unit testing, there was a need to test performance, performance static analysis on the code and well as execute a full functional test run. The problem was that most of these things took hours to complete. As the start of this, a full functional test run, if done sequentially, took 60 hours. Luckily it was possible to partition the testing so that it could be run in parallel.

What was required is to create a flow that looked similar to below.

This is achieved via the Cloudbees Build Flow Plugin, which is free and open-source. The source code is on Github. I am not going to explain more about the basics of the plugin, that can be read for itself on the plugin wiki page. There is enough there to get someone started. However, when you want to achieve the above kind of flow you'll need to know a bit more.

The DSL is Groovy-based, so knowing the basics of the languages and especially Closure syntax is required.  The flow itself is controlled via the FlowDSL class. Furthermore calling build returns a JobInvocation instance and dependant on how parallel is called, it will either return a list or a map containing FlowState instances. The build results are actually stored in a Run instance which, after obtaining a JobInvocation instance is available via .build property-syntax. Knowing this difference is important as it will help you to extract build information later on.

Once you have the JobInvocation object you can obtain a number of useful bits of information

Creating Build Flow

The following code snippet illustrates how to create a build flow similar to the graphic depiction.

 

Define as much as possible as closures

This will delay execution until such time that is required. This is what parallel relies on too. buildFlow, staticAnalysisFlow and performanceFLow are examples.

Capturing build results within the closure

Notice how buildFlow will store results in buildResult. (The latter will become a JobInvocation instance after execution)

Executing multiple instances of the same job

It is relatively easy to execute multiple instances in parallel, each running with different parameters. This can be done with a simple Groovy Range and using the collect operation. parallel requires a list of closures to be passed to it, collect does that for you See how testFlow is defined above. We have even wrapped each build inside and ignore closure, so that unstable builds don't break the build pipeline.

Break down complex flows into smaller flows

Stringing lots of flow sections together, can be tricky to follow. In such cases break down the flows into smaller manageable sections. As staticAnalysisFlow took a very long time to complete, I wanted it to run parallel to everything else. I have therefore created a mainFlow to run alongside. In order to deal with the complexity of running performanceFlow in parallel with another flow which already has more parallel flows in it, I have broken it down into smaller flows, which are then strung back together. Strictly speaking the use of arrays and each are not necessarily, but when you have even for jobs involved, this style can sometimes be easier to read.

Collecting results from multiple parallel builds

For all of the aprtitioned test runs, I wanted to collect the build numbers so that it could be passed downstream. As mentioned earlier parallel returns a FlowState instance. It is a simple case of iterating through all of the builds and finding the lastBuild property which will give you a JobInvocation to work with.

Collecting everything

Once all of the builds have completed, the necessary information can be passed do a final downstream job, which in turn can aggragate information form all of the other jobs.

Build-flow Limitations

For me one of the drawbacks of the current implementation is the lack of Grape support. Sometimes one needs just a bit more decision-making capability than what stock Groovy will offer you. It would be helpful to be able to use @Grab to pull in some extra libraries.

It is also not possible to version the build flow script. It could definitely be useful to store it in source control and then update it before each run.  The best workaround for now is to either

• Create the build flow job via Jenkins CLI and version control the config.xml file
• Create the build via Gary Hale's gradle-jenkins-plugin and version control the build.gradle and template.xml files.
• There is no real way of testing the flow outside of Jenkins at present. This makes it hard to syntax check before hand. The best you can do is create a series of fast-completing mock jobs, to experiment with and once you are happy convert the flow to attach the real jobs that you need to link.

In conclusion

Even given the above mentioned limitations, the Build Flow Plugin is a powerful item in the quiver of any Jenkins crew. I hope that this article will help other towards some more complex real world orchestrations.