We killed the butler: Replacing Jenkins with Concourse

By Anna Kennedy and Holger Ihrig, 2019-12-20

At Working Group Two, we try to use CI/CD pipelines to automate all of our repetitive tasks when it comes to code and infrastructure deployment and testing, such as:

We had been using Jenkins to run such pipelines, but having to configure it by navigating a web GUI made it difficult to maintain, redeploy, and upgrade, so we decided to look for alternatives.

Jenkins

The majority of our code lives in a monorepo, and we use Bazel to manage builds and tests. We try to do all of our infrastructure configuration via gitops so it was important that a continuous integration and deployment system not only play nicely with our existing structures, but be itself configurable from code.

We spent time investigating other options, and eventually settled on Concourse, a cloud-native CI/CD server where tasks are deployed in containers, and config is stored as yaml.

Infrastructure as code

We run Concourse in Kubernetes, so the setup and configuration itself is all done with yaml files and kubectl. It’s fast and easy to upgrade and redeploy.

Within Concourse, the pipeline configuration is entirely yaml-based; there are no buttons in the UI except for an abort/re-run button.

jobs:
- name: run-acceptance-tests-staging
  plan:
  - task: run-tests
    config:
      inputs:
      - name: monorepo
      run:
        path: /bin/bash -c runtests.sh
    on_failure:
      put: notify-slack-ci

Concourse task

Pipelines are made up of jobs that run in series or parallel; jobs consist of tasks. Pipelines, jobs and tasks are described in code and automatically visualised in the UI. Changes to pipelines are applied by updating the yaml file and running Concourse’s fly cli tool.

Containerised deployment

Concourse runs every job in its own container, which means that every job uses an entirely clean, reproducible environment. Any dependencies required for a task can be pre-installed in the image. This is a huge improvement for us over Jenkins, where dependencies were installed to the entire worker node, and artifacts from previous builds were left lying about on the server.

We use docker containers, and we also run Concourse itself as a container, which means a bit of docker-in-docker magic. It look a little work to build an image we were happy with, but beyond that it went surprisingly smoothly for us on the whole. The only drawback is that we have to run images in privileged mode, but in our self-managed Kubernetes cluster this isn’t really too much of an issue.

There were some challenges in figuring out what resources needed to be allocated; we settled on three worker nodes and a maximum of 2 active tasks per worker. We see a little bit of slowness some afternoons when the pull-requests are coming thick and fast, but the cluster remains stable. It would be nice to set some autoscaling here to cope with short-term peaks in load.

Concourse pipeline

Debugging

Since the tasks all run in containers, it’s easy to debug issues locally by running the same image on the laptop as is running in Concourse. Alternatively, the fly execute cli tool runs a local project in a container in Concourse, a nice interim step when trying to get a deploy working.

The fly intercept tool offers a way to log into a running container in concourse to troubleshoot:

$ fly intercept -j ecs-services
1: build #27, step: monorepo, type: get
2: build #27, step: notify-slack-ci, type: get
3: build #27, step: notify-slack-ci, type: put
4: build #27, step: run-acceptance-tests-staging, type: task
choose a container: 4
root@02f69d15-b7be-4f2e-43f7-24f549071bb1:/tmp/build/3a58ea39#

Resource types and extending Concourse

There are a large number of resource types available for Concourse, making it fairly straightforwards to configure pipelines.

As resource types in Concourse are all based on containers, extending Concourse means introducing a new container that can be called from Concourse.

A resource type needs to implement the following executables/scripts:

These binaries need to be placed under /opt/resource in the docker container. Concourse calls these binaries with JSON payload and optional parameters that can all be found here. What is done in those executables/scripts is up to the implementer, but it is really easy to extend concourse using this mechanism.

Learning curve

The move from Jenkins to Concourse has overall been a very positive step for us. If there has been any drawback it’s that implementing any brand new system usually means something of a learning curve, and Concourse is no exception.

The documentation is sometimes a bit minimal, but now that we have a number of pipelines up and running we’re finding it easier and easier to add more. Similarly, it took some time to settle on how to allocate resources such that we cope with load but aren’t wasteful at quiet times.

Summary

The biggest improvements of Concourse over Jenkins have been:

Overall, we’re pretty pleased with Concourse. There are a few features we’re eagerly awaiting in future releases, such as re-running targeted builds, but for the most part it has been relatively straightforwards to import all our existing jobs from Jenkins and add more.

References

Suggest a change

Working Groups Two's website is completely open-source and hosted on GitHub. Anyone is free to suggest changes through GitHub.

Suggest a change