When we started implementing CI at Wolox, every time someone broke the build in Travis CI, they had to bring some pastries for everyone at the office. When the teams started getting bigger, that was not the only thing that was hard to implement: Travis queues started growing and the applications environments became hard to setup and maintain. Consequently, builds in queues started becoming idle and failed permanently. That was the moment when we started research to identify for similar tools, but none of them suited our needs. After some research we came up with Jenkins and tried to implement it in the same way as Travis but using Docker Compose to run the builds without sharing any environment configuration between them.

This is a step by step guide to setting up Jenkins to run builds with Docker Compose and report status to GitHub Pull Requests. We will show a Docker configuration for Ruby on Rails but it can be modified to run with whatever framework you prefer.

See also: Dynamic Jenkins Pipelines

Jenkins

What we ended up doing was installing Jenkins in an Amazon EC2 instance with Nginx. You can find an installation guide here.

We needed several plugins to integrate Jenkins with GitHub and have a Travis CI like a flow:

• GitHub Plugin
• GitHub Authentication Plugin
• GitHub Pull Request Builder: For more information about how to setup this plugin read this and this.
• Post Build Script Plugin
• Parameterized Trigger
• Matrix Project Plugin: This Plugin lets you manage project permissions for GitHub users.

You can install all of them by going to Manage Jenkins > Manage Plugins

To have a similar flow to Travis CI we added two different jobs:

• JobName: A job that builds any Pull Request created.
• JobName-Base: A job triggered by a Pull Request job. It builds the base branch of the project every time the associated Pull Request job has a successful status.

Let’s setup the jobs now!

JobName

• Name: YouProjectName
• GitHub project: https://github.com/Wolox/your-repositoy-name
• Source Code Management: Git

You need to set Refspec with:

+refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*

You can add an SSH credential for your repository. This can be done in Credentials.

• Check GitHub Pull Request Builder and add the following information:

1. Commit Status Context: The name you would like GitHub to show in the Pull Requests when running your Jenkins job.
2. List of organizations. Their members will be whitelisted: You can add your organization’s GitHub name here.
3. Check Allow members of whitelisted organizations as admins.
4. Check Build every pull request automatically without asking (Dangerous!).

• Build Execute Shell:

So, this is the script we are going to execute every time the job is triggered. We have several steps:

1. Remove unnecessary files to free some space.
2. Build a docker compose container. This will run the Dockerfile that we will setup later.
3. Prepare the test database.
4. Run Tests, lints or whatever script you like. We chose Rspec and Rubocop Lint.

Unbuffer will output your script logs with pretty colors. It’s necessary to add set -e at the beginning of the script for Jenkins to fail if any previous step failed. If you avoid adding set -e, then it will only fail if the last step failed (in this case the Rubocop lint).

• Add Set status “pending” on GitHub commit build step.
• Add a Post-build Actions Execute Shell build step to remove some Docker dangling files:

This will remove idle containers and images to free some space. Actually Docker has some issues with space so you can run this script with a cron.

• Add a Trigger Parameterized build on the other projects build step:

This will let us trigger the JobName-Base job when our Pull Request build is successful. This is the only setup that worked for us. There are some other Jenkins Plugins that should resolve this without needing to make a new Job. But they didn't work as we expected.

Now it is time to set the Base Job!

JobName-Base

You can create this Job by cloning the one above. The setup is quite similar.

• Name: YouProjectName-Base or whatever name you like
• GitHub project: https://github.com/Wolox/your-repositoy-name
• Check This build is parameterized and add:

• Source Code Management: Git. Here we added the same configuration as used on the other Job.
• Uncheck GitHub Pull Request Builder. We are not triggering this Job with a Pull Request.
• Build Execute Shell: Same as the other job.
• Post Build Action: Same as the other job.
• Remove Trigger Parameterized build on other projects’ build step.

Every time someone makes a Pull Request, our JobName will be triggered and if the build is successful, then the JobName-Base will be triggered. We chose origin/development as our base branch, but you can change this in the job parameters.

See also: Unicorn Not Trapping USR2

Docker Compose

Our Jobs will use Docker Compose to run our tests. So lets see how to setup this in our Ruby on Rails project:

Dockerfile

This file contains all the environment setup we need for our project to run. Every time we execute the Docker-Compose build it will run this script and create a new image for our code. The following example will work for most Ruby on Rails projects. We added essential ubuntu libraries and Gems installations. You can read more about Dockerfile here.

You can play with the -j parameter to make the gems installation faster.

Docker-Compose.yml

In this file you can setup different services that you will need when running your project. For example: database, web server, redis, etc. You can read more about it here. With the following example you can run and develop a Ruby on Rails application using Docker.

database.yml

If you want to use Docker Compose for CI and for local development, you need to add some setup to your database configuration file:

If you are using MongoId you need to override the mongoid.yml in the build step script:

Also you have to add mongo image to Dockerfile.

Results

After all these setups, you will see something like this in Jenkins:

We managed to run tests and lints in a unique container isolated from all the others, every time someone creates a pull request. Whatever the result of the build, it will be shown in Github like this:

Successful build:

Failed build:

Also with all this setup we can run our containers locally just by running docker-compose run web and have our environment ready to develop in a matter of seconds.

It’s Nice to Have Features

To have a complete Continuous Integration process we are missing the deployment of our application every time someone merges a feature with our base branch, for example. That’s the last part we are working on, we will be giving more details about it in a future post! We have been working hard to setup the isolated containers we told you about and we need to improve the jobs’ triggering process. It will also be nice to have a ci.yml file where we can set some configurations for our builds.

Posted by Esteban Pintos (esteban.pintos@wolox.com.ar)

www.wolox.com.ar