Posts Tagged ‘CI’

Continuous integration at the #EMCDojo CI at the EMC Dojo, Brian Roche Sr Director of Engineering

CI at the EMC Dojo, Brian Roche Sr Director of Engineering

Brian Roche

Brian Roche - Senior Director, Cloud Platform Team at Dell EMC. Brian Roche is the Leader of Dell EMC’s Cloud Platform Team. He is based in Cambridge, Massachusetts, USA at the #EMCDojo.

If you have worked on a project for the last 10 years and you have a nicely written (and probably bound) Product Requirements Document, then you’re probably working on a Waterfall project.  If you’re too busy to even ponder the first sentence because you’re overly concerned with the ’testing phase’ of the project that’s coming in a few months, then you’re definitely on a Waterfall project.

The truth is most of the IT projects in-flight today use the Waterfall methodology of Plan, Do, Check, Release.  These projects measure release cycles in months and sometimes years – not in minutes or seconds like most agile projects.  For them, releasing software is a BIG DEAL.  After all, they’ve been working on this software for 12-18 months.  These teams don’t practice the art of releasing software very often.  So what happens when we don’t build the muscle memory to do something?  We’re not very good at it. Most Waterfall teams are not very good at releasing software to their customers.  Subsequently, the product they produce reflects the fact they’re out of shape when it comes to creating installers, packages and the release process in general.


How to Set up a Concourse Pipeline Xuebin He Dojo Developer

Xuebin He Dojo Developer

Xuebin He

Latest posts by Xuebin He (see all)

How to Set up a Concourse Pipeline

The first step to continuous integration is setting up your own CI pipeline. The #EMCdojo uses Concourse for our own pipeline and we love it! Concourse (the official CI tool for Cloud Foundry) can pull commmitted code and run tests against it, and even create a release after passing tests.

Before I tell you HOW, I’ll tell you WHY

In our workspace, our pipeline monitor is displayed on a wall right next to the team. A red box (aka failed task) is a glaring indicator that something went wrong. Usually the first person who notices shouts out “Ooh! What happened?” and then we roll up our sleeves and start debugging. Each job block can be clicked on to get output logs about what happened. The Concourse CLI lets you ‘hijack’ the container running the job for hands-on debugging. Combining these tools, it’s usually fairly quick to find a problem and fix it.

Having this automated setup, it’s easy to push small features one at a time into production and see their immediate effect on the product. We can see if the feature breaks any existing tests (unit, integration, lifecycle, etc). We also push new tests with the new feature and those are added to the pipeline. At the end of the pipeline, we know for sure if the feature is done, or still needs more work.

Step 1: Set up Concourse

Set up Server

The easiest way to set up concourse is using vagrant

You can access your concourse at

Download Concourse cli

You can only start, pause, and stop pipelines or tasks on the concourse website. If you want to configure the pipeline, you have to download fly from concourse. Fly is the name of concourse cli.

Step 2: Configure Pipeline

Make a CI Folder

You can generate your CI folder under the root of your project. See code block below.

pipeline.yml will define what your pipeline looks like.


So now your pipeline should look like this:


Using groups, we can make a different combination of jobs. Each job can have several tasks. The tasks are located here ci/tasks/*.yml.


This defines a task. A task is like a function from inputs to outputs that succeed or fail. Each task runs in a seperate container that requires you to give the address of the docker image that you want use. You can put Dockerfile under ci/docker/. Inputs are already defined in pipeline.yml. The duplication here is to make it easy for us to run one-off tests. The outputs of the task can be reused by later tasks in the same job.

Make a Secret File

You have to generate a secret file that has all of the environments required by the pipeline. All required environments are in pipeline.yml wrapped by double curly braces.


Set Pipeline

Start Pipeline

The initial state of the pipeline is paused. You have to start it by clicking the menu button on the concourse website OR with fly by:

Run One-off

You can run a one-off test for a specific job. This will not be shown in pipeline.

The lines above fly execute are the environment variables, and lines below are the inputs of that task. Those are already defined in ci/tasks/*.yml.


You can hijack into the container thats running the task that you want to debug by:

If you run one-off, you can just run:

You can find build number by clicking the top right button on your pipeline page.

And you’re done! And remember: Continuous Integration = Continuous Confidence.

If you have any questions, please comment below.