Archive for the ‘Devops’ Category

Concourse Tutorial

Xuebin He

Latest posts by Xuebin He (see all)

Why we are here

If you are reading this blog, you probably already know the benefits of Continuous Integration and Continuous Delivery and Concourse is one of the great options out there. You may not know where and how to start this new DevOps journey. This blog will try to eliminate your concerns through the following Chapters.

Chapter 1

What is Concourse

Concourse was started as a side-project inside Pivotal. It became very popular quickly in the CloudFoudnry Community and Pivotal and now in the DevOps world, because it has benefits like a yml defined pipeline, containerized environment and well defined API for supporting different resources, etc.

Why Concourse

Let’s talk about some of the benefits a little.
The pipeline in Concourse is defined only by one yml file so that it can be placed into the project repository and can be tracked by VCS. In this case, the pipeline would always be alined with the code and tests. It’s also a good chance to put Dev and Ops sitting together.
Having containerized environments for different tests makes sure there is no pollution between tests and testing environments. And users have control over what docker images are used in each test and those Dockerfiles could also be tracked. If things go wrong during the tests, the user can inject into the failing container and perform debugging.
Concourse supports a reasonably large amount of plugins for different abilities like building and pushing Docker images, sending Slack messages to the team if builds failed… You get the idea.

How to deploy

You can always deploy Concourse using bosh if you are familiar with it. If not, don’t worry, Concourse also provides deployment using docker-compose.
Once it’s being deployed, we can download Concourse’s CLI named fly from its home page, and put it into our PATH.
First we need to login using:
This will create a yml file at ${HOME}/.flyrc containing the following lines:
In the above file, cool is an alias for this Concourse instance. Using this, we can use fly without typing IP and password every time. For example, if Concourse is upgraded and we want to upgrade the local fly, we can use the following command:

Chapter 2

Baby Crawl

A pipeline normally consists of many different tasks. Concourse is able to run a task in a container and react to its result. That’s why a task could make the pipeline very colorful. A task is defined as a yml file. This file only defines what is needed to run this task, and doesn’t specify how to get different dependencies into its running container.
For instance, lets say we have a project called cool-project, and this project needs to be run. This is done through a script; in this case, a-cool-task, which is placed in: cool-project/ci/tasks/a-cool-task.sh. This cool task is saying “I need to be running in a container from an Ubuntu image, and I’m expecting there to be a folder called cool-project placed into my working directory with the file cool-project/ci/tasks/a-cool-task.sh inside of it. Lastly, A_VERY_COOL_ENV is required for me to start”.
a-cool-task.yml
Say we have this cool-project at ${HOME}/workspace/cool-project, and our task definition file is colocated with the task script. We can run this task using the fly command like this:
The execute command will ask Concourse to run this task as one-off, not being part of any pipeline. Concourse will create a new container from ubuntu and populate A_VERY_COOL_ENV into this container. It will also copy the whole cool-project folder into the working directory, denoted by ${WORK_DIR} for now. Then, it will trigger ${WORK_DIR}/cool-project/ci/tasks/a-cool-task.sh to start the task script.
Normally, execute will help us a lot during the development stage of the pipeline itself, because it provides a much faster feedback loop.

Baby steps

Now, let’s build a dummy pipeline and put it into cool-project/ci/pipeline.yml. We can put all pipeline related files into the ci folder.
pipeline.yml
A pipeline normally has two parts, resources and jobs. Resources are the definitions for dependencies like a git repository such as cool-project or a Docker image, etc. Concourse is able to download those resources and put them into containers where the jobs are running.
Jobs are the core part of a pipeline, they define the whole workflow of the pipeline. For example, in the above pipeline, we only have one job a-cool-job, and it has two steps that will be running in sequence in the order written in the job definition. The first step is a get that will download the cool-project repo, and it will be triggered by pushing a new commit. The second step is to run the previous task that we defined (a-cool-task).
The last part is the variables wrapped by double braces {{}}. Those variables will be evaluated when we setup the pipeline. The value of these variables could be stored in a separate yml file that is not going to be put inside this repository, because this file normally contains some secrets like the following one:
secrets.yml 
We can setup the pipeline by:
Now if we open our pipeline in a browser, it should look something like this:
a-cool-pipeline
a-cool-pipeline.png
Black boxes stand for resources, while colored boxes stand for jobs.
If we click this job box, we should see the latest build of it.
a-cool-job
a-cool-job.png

Chapter 3

Making trouble

With the pipeline, we could run into 2 different kinds of trouble: the commits failed the pipeline or the pipeline failed itself. Let talk about the first one first.
Hooks
When the commits failed, our pipeline will make the job box red, and will not go any further. At this point, we probably want to do some damage control or “blame” the trouble maker (for example, send him/her a Slack message).
Concourse Hooks will enable us to trigger another script when the task itself has failed or succeeded.
For example, with the following job definition, if any step inside the plan fails, Concourse will run the task under the on_failure hook, which will alert this is not that cool! in this case. We can also do something when the job succeeded with on_success or when the job aborted with on_abort.
job.yml

 

a-cool-failing-job
a-cool-failing-job.png
Hijack
With hooks, we are able to do some damage control, but we still eventually need to know why it failed. We can hijack into the container that ran this particular task by:
Now we just need to select the number of the step where the job failed from the output.
Because Concourse has its own Garbage Collection system, it will remove inactive containers after a certain amount of time. We can actually set the interval of GC doing cleanup by adding the option –gc-interval to the start command of the Concourse web instance.
One-off
Like what we did in Baby Crawl we can debug a task separately with the same inputs from the pipeline. The reason why we do this is because we can get much faster feedback if we are trying to fix it. For the same reason, we also use one-off a lot while we are developing a new task.

Chapter 4

Playdate

Concourse cannot do everything. As users, we still need to write some code to solve some problems ourselves. For example, we are not able to pass artifacts being built from one job to the next job.
However, Concourse provides a well defined pattern for developers to create plugins to allow Concourse to do things that it would not normally be able to do. For example, it has an s3 plugin which makes it easy for us to upload or download artifacts from Amazon S3 Storage to pass artifacts between jobs.
Sometimes, our test involves multiple stages, like: clean existing servers, deploy new servers, then run test. Once a commit goes into these stages, we want to make sure that no other commits go into these stages again until the first commit is finished since we only have limited resources to run servers. To prevent other commits from going into these stages, we can use the plugin pool.
pipeline.yml
The above pipeline has two jobs to complete the acceptance test. The first job will try to acquire a-cool-lock from the resources locks, and will wait forever until it’s been released by the previous holder. If the job itself failed, it should release the lock. The lock should also be released at the end of the acceptance test, whether it failed or succeeded. This assures the lock will be available if no job is currently using it.

Chapter 5

Tricks

Dockerfile
Every task will be running in a fresh container. Most of the time, we have to install some dependencies for our tests or other tasks to be able to run. We shouldn’t let the pipeline spend too much time on something not directly related to the job(s) at hand.
Starting the fresh container that already has those dependencies preinstalled would make the feedback from the pipeline much faster. Because those Dockerfiles for the containers might be very specific, having them placed into ci/docker/ folder will help the team to maintain them and enjoy the benefit of being version tracked.
YML
Similar jobs may share a lot of the same steps or variables, and it might be very painful to keep these steps and variables concurrent throughout the entire pipeline definition as some of them change, because changing one step or variable would then require changing it again everywhere else that it appears.
For example, both of the jobs above are using Kubernetes and require secrets for the Kubernetes instance. &kube-secrets will create a reference to those params, and <<: *kube-secrets in the second job will be replaced by those params from the first job while setting up the pipeline using fly set-pipeline.

Why The Dojo Matters a guide to digital transformation wrapped up in a scipab

a guide to digital transformation wrapped up in a scipab

Emily Kaiser

Emily Kaiser

Head of Marketing @DellEMCDojo #CloudFoundry #OpenSource #TheWay #LeanPractices #DevOps #Empathy

 

The Situation | Digital transformation has become a nuanced term. Somewhat like ice cream in the summer. When you see someone eating your favorite flavor on a hot summer day, it incites an immediate craving. You know that you want the cone, that it will bring you a sense of happiness not only in the taste, but also maybe in the social outing it will revolve around. But then you think about that bathing suit you hope to wear later in the day or the water that will then be needed to quench your thirst, and oftentimes there is hesitation in fulfilling the craving you know ultimately will bring no regret. Okay, maybe that’s a little too distilled of an explanation, but I hope you get the point. Digital Transformation. We know it is important, in fact it is the impending future no matter how much you try to avoid or deny it. But then by embarking on the journey, you also know that it will create work and a probable disruption in your comfortable ‘plan’. So you begin to question the value. And begin to cringe at the term, or try to validate your thought that the term has too much hype.

The Complication | This hesitancy and resistance in diving head first into the process actually hinders long-term success. Companies that are not investing fully all in money, support and effort are running the risk of falling behind competitors.  In order to build more qualitative products at a more rapid speed, there comes a time where the company, each of its teams and its employees need to embrace and overcome true digital transformation. But it’s hard. Really hard.

The Implication | As seen in the image attached, companies are running the risk of losing the opportunity to obtain that expected 30% revenue by 2020 where customers are investing to be a part of the movement. Not only this, but if companies don’t move quickly they will miss the sweet spot in the maturity of digital transformation that their competitors are gaining as time lapses. This most immediately causes depreciation of Net Promoter Score, which arguably now is more important than any vanity metric (i.e. how many lines of code are being written, number of commits, etc.) ever was. And once trust is lost, it is close to impossible to rebuild especially in the Fortune 500 customer base.

Position | It is now more than ever that we need to look past the nuance and move our teams toward modernization. At the Dojo, our mission is two-fold; to practice modern software development methodology (XP, Lean Startup) and to further evangelize ‘The Way’ to internal Dell EMC product teams, and to contribute to Cloud Foundry Foundation sanctioned OS projects. We are very lucky to work for a company that is investing in and understands fully that in order to stay alive, and most importantly thrive as IT leaders, we must continue to scale in this world of Digital Transformation. Our power at the Dojo lies in the buy-in from all levels.

Action | Our power as Dell EMC on the Digital Transformation world stage lies in the buy-in from every member of the company. It has been proven time and time again that customers LOVE the modern way in which we are building software. There are definitely challenges to rewiring the way that we work and the way that we measure the work we produce, but with hard work, comes not only a thrilling journey, but a highly productive one that produces amazingly positive results. There is no better time than now to jump on this Digital Transformation train.

Benefit | Use the Dojo as a testament and witness to all of the aforementioned sentiments and Digital Traction Metrics as seen in the attached image. Join us in paving the path to the Future. And eat an ice cream cone while you are at it.

Doers for Today, Visionaries for Tomorrow and Change Agents for Always! We Truly Are The Trifecta

We Truly Are The Trifecta

Emily Kaiser

Emily Kaiser

Head of Marketing @DellEMCDojo #CloudFoundry #OpenSource #TheWay #LeanPractices #DevOps #Empathy

The Dell EMC Dojo has a mission that is two-fold; we contribute to open source Cloud Foundry, and we evangelize ‘the way’ (XP, Lean Startup, etc) by engaging with internal Dell EMC teams in a purely DevOps manner. Our mission is direct with a scope that some could argue is boundless. By practicing ‘the way,’ hours, days, and weeks fly by as we push code to production at times every few minutes. Not only is our push to market rapid, so is our overall productivity. Oftentimes teams working with us nearly guffaw when they come to our office in Cambridge, MA and are able to see the ‘wizard(s) behind the curtain.’ We are asked how we keep three to five projects on track while also engaging with internal teams, planning large technical conferences, and working in the realm of R&D in our greater TRIGr team with an east coast contingent of only eight people and a west coast contingent of five. The secret? We LOVE what we do!

 

A team with empathy at its core, there is never a moment when a task seems impossible.

Truly, we could be featured on one of those billboards along the highway stating that there is no ‘I’ in ‘Team.’ Two baseball players carrying an opposing team member across Home because she/he has hurt themselves, a photo of all of the characters from Disney’s “The Incredibles,” The Dell EMC Dojo team… Take your pick… TEAMWORK. Pass It On.

In all seriousness, the pace at the Dojo can be absolutely exhausting, and with such a small team, the absence of one person (which let’s face it, vacation and life needs to happen at points) could in theory be a huge deal. But, because DevOps is what we live and breathe, any member of the team can fill this gap at any point, truly putting to practice the idea that there doesn’t have to be and should never be a single-point-of-failure. Albeit the industry or sector, what more emulates the ‘Go Big, Win Big’ message than this? By continually pushing ourselves to pair and to up-level the knowledge of our entire team, we never wait until tomorrow to take action. There is no need or desire to.

 

Agility is not a term we just talk about, but is simply inherent to everything we do.

With the combination of the rapidly changing market (externally and internally) and the pace in which we work, we at the Dojo have learned that we must stay on our toes. For those reading this that are familiar with sports, one of the first lessons learned in soccer is to never plant your feet. Holding such a stance allows for the opposing team to outpace you when the unexpected happens, which is most of the time. Same goes here. Pivoting is now second nature for us, and it doesn’t come with the scares. Instead, it is actually exciting when we are able to take data and identify ways in which we can better align with efficiency and effectiveness of software and methodology; to truly keep the user omnipresent in everything we do. We are happier. The ‘customer’ is happier. It is a (Go Big) win-win (Big) game. The cool thing too is that the more we practice this, the more we also feel somewhat like we can predict the future, because we begin to see trends before they are even a thing.

 

 

Doers for Today. Visionaries for Tomorrow. Change Agents for Always.

Do You Hear That? It’s the sound of Keyboards! Call for Papers | Cloud Foundry Summit Silicon Valley 2017 is quickly approaching!

Call for Papers | Cloud Foundry Summit Silicon Valley 2017 is quickly approaching!

Emily Kaiser

Emily Kaiser

Head of Marketing @DellEMCDojo #CloudFoundry #OpenSource #TheWay #LeanPractices #DevOps #Empathy

Our brains are on fire, our keyboards are hot, and the joke in the office the past few days has been over our extreme excitement for the eventual need to buy sunscreen since our Boston winter leaves us Vitamin D deprived. Why is this the case, you may or may not be asking? Well, I plan on telling you anyway because it is just too exciting not to share!

Our team is preparing for CLOUD FOUNDRY SUMMIT SILICON VALLEY! We felt a social duty to let all of those we care about and want to be there with us for what’s sure to be the summit of the summer (how can it not be when it is being held in June in Santa Clara?!), that the last call for papers is quickly approaching (no seriously, it’s this Friday, February 17th).

Just as a refresher for those on the fence, Cloud Foundry Summit is the premier event for enterprise app developers. This year the Foundation, through market research and feedback, found that interest and industry need is engrained in the focus on innovation and the streamlining of development pipelines. For this reason, Summit 2017 is majorly honing in on microservices and continuous delivery in developers’ language and framework of choice. That is why the session tracks available will be Use Cases, Core Project Updates, Experiments, Extension Projects, and Cloud Native Java. Each session that is chosen for the conference is enabled (1) primary speaker and (1) co-speaker. The primary speaker receives a complimentary conference pass while the co-speaker receives a discounted conference pass. So what’s stopping us from getting involved? Absolutely NOTHING!

As a sneak peak to a few of the topics our team have submitted for approval, see below:

  • Adopting DevOps and Building a Cloud Foundry Dojo (Lessons Learned)
  • Lift & Shift Your Legacy Apps to Cloud Foundry
  • How to Develop Scalable Cloud Native Application with Cloud Foundry
  • Enabling GPU-as-a-Service in Cloud Foundry
  • Blockchain as a Service
  • Avoiding pitfalls while migrating BOSH deployments
  • Spring Content: Cloud-Native Content Services for Spring

 

So, now what’s stopping YOU from getting involved? Submit papers here: https://www.cloudfoundry.org/cfp-2017/ and/or register here: https://www.regonline.com/registration/Checkin.aspx?EventID=1908081&utm_source=flash&utm_campaign=summit_2017_sv&utm_medium=landing&utm_term=cloud%20foundry%20summit&_ga=1.199163247.1732851993.1460056335

Last but definitely not least, let us know if you plan on coming—we are more than happy to share sunscreen 🙂 We cannot wait to see you there!

Dell EMC Dojo at Hopkinton! Reviewing what we covered, and what we learned

Reviewing what we covered, and what we learned

Hey again everyone! We’re writing out today to talk about a few topics that we covered during our time at the Dell EMC Dojo Days in Hopkinton. We met with a lot of great minds and took plenty of input into how we work. We also gave a lot of insight to passersby in what we did and how we worked.

Firstly, Brain Roche and Megan Murawski debuted the famous “Transformation Talk”. We normally give this presentation in preparation for an engagement with another team, but in this case, it was given to allow open criticism and circulation of our methodology to the rest of the company. We cover in this presentation: pairing, and why it’s important to our process, why we use TDD (Test Driven Development) (and why you should too!), and our weekly meetings including Retros, IPM, and Feedback to name a few. We had plenty of great ideas and questions, as usual, and we realized twenty minutes over time that we couldn’t get Brian off the stage.

Xuebin He eventually got Brian off-stage for a talk he conducted on CICD (Continuous Integration, and Continuous Deployment). Xuebin being one of the developers at the Dojo allowed him to be a bit more technical in his talk, and cover some of the programming practices and tools we use to achieve this at the Dojo. Concourse is our tool for running our beautifully constructed tests, along with standard mocking design patterns and the code quality produced with TDD.

We picked up again on Tuesday at 1145 to talk about why a PaaS exists, and why it’s important. That talk, given by yours truly, was focused on some of the common technical roadblocks that keep developers, customers, and managers from being able to work efficiently; as well as the ways using a PaaS can solve those problems to build a better business.

To containerize applications for a PaaS, we would need to learn basics like “What is a 12 factor application, and what’s a container?” Thinh Nguyen stepped in and gave a great description on how we use guiding principles while developing our application environment to be better for us and our customers.

Throughout all of our talks, we worked away on two pair stations very carefully brought from our lair in Cambridge. We gave away some free swag, some free candy, and raffled off some super giveaways. We thank everyone involved in preparing and executing these few days for their hard work. We also want to give a huge thanks to everyone who attended our talks (rambles) and participated in some mind-expanding conversations.

Finally, I want to close with a few notes. We always enjoy fresh perspective. If you had more to say, or you missed us during our time and you want to start a conversation, leave a comment in the comment section! If you don’t want to comment here, then drop us a line in my email. We’d love to hear from you.

Until next time, remember: Cloud Foundry, Open Source, The Way. #DellEMCDojo.

THE EVENT OF THE SEASON Coming to Hopkinton December 5th and 6th

Coming to Hopkinton December 5th and 6th

Emily Kaiser

Emily Kaiser

Head of Marketing @DellEMCDojo #CloudFoundry #OpenSource #TheWay #LeanPractices #DevOps #Empathy

Before reading this blog post, I suggest that you and anyone that may be reading over your shoulder sit down. In your chairs? Great, because the excitement you are about to feel will surely leave you overwhelmed.

 

The event of the quarter is right around the corner! Please consider this your personal invitation to join us for DOJO DAYS next Monday and Tuesday, December 5th and 6th from 10 am to 4 pm in the 176 Café out at Dell EMC in Hopkinton.

official-dojo-days-logo

DevOps is a hot word in the world of technology. It represents a way of working and technique that aims at making the customer omnipresent in everything that is created and delivered. Companies today consider this way the future.

For this reason, Dell EMC and Pivotal paired to create the first ever Dell EMC – Pivotal Cloud Foundry Dojo. Our goals are centered around personal, team and company transformation. Through the practice of modern software development, we spend our days contributing to Cloud Foundry Foundation sanctioned Open Source projects through R&D modernization and methodology known as ‘the way’ (XP, Lean Startup).

Many know us as the team in Cambridge with great food and beverage (no really, it’s amazing- come visit us to see for yourself). We realize, though, that the commute isn’t ideal for most. So, we bring you Dojo Days to experience #adayinthelifeofthedojo, with a replication of our office environment and our team in action representing the steps we took to transform.

In addition to seeing #adayinthelifeofthedojo, you will have the chance to pair with us, join our team for lightning sessions that are driven by the audience’s goals (schedule below) and have the chance to win a Dell EMC Dojo branded Patagonia, S’well water bottles, t-shirts, and more!

 

The schedule of the lightning sessions are as follows. Please come by the Café day of to sign up and/or just drop in:

Monday, December 5th:

11:45 am to 12:45 pm: Steps to Transformation starring Brian Roche and Megan Murawski

2:00 pm to 2:30 pm: TDD and CI/CD starring Xuebin He

 

Tuesday, December 6th:

11:45 am to 12:45 pm: PaaS, and Why your Developers Care starring Gary White

1:30 pm to 2:00 pm: Factor, Cloud Native, and Containers (oh my!) starring Thinh Nguyen

 

If you have any questions, please contact Emily Kaiser at Emily.Kaiser@dell.com. Otherwise, we cannot wait to see you, your team, and all of your friends out in Hopkinton next week!

Bringing Cloud Foundry to You… Exciting Upcoming Events that the Cool Kids are Attending

Exciting Upcoming Events that the Cool Kids are Attending

Emily Kaiser

Emily Kaiser

Head of Marketing @DellEMCDojo #CloudFoundry #OpenSource #TheWay #LeanPractices #DevOps #Empathy

For those of you who follow us on Twitter (@DellEMCDojo for those that don’t) or via this blog, I would guess it is pretty evident that we just love Cloud Foundry. We take pride in being a member of the foundation mostly because it is with this platform that we feel we are delivering and practicing what we preach; living out a true DevOps culture most conducive to productivity and quality product and delivery.

We are lucky enough to just be returning from Cloud Foundry Summit where upwards of 800 developers and other market adopters joined in Frankfurt, Germany to learn from other community members and to be a part of the PaaS’s business direction. With the Native Hybrid Cloud team, we at the Dojo spent our days representing DellEMC through presentations and booth shifts. We left feeling even more confident in our daily choice to use Cloud Foundry and to further evangelize the PaaS for the betterment of the larger technical OS community.

It is because of this that we are excited to announce that we will be hosting and supporting three Cloud Foundry Days here in the near future. Cloud Foundry Days are new, one-day educational events aimed at developers and operators who want to connect with businesses, learn about the platform, and share knowledge. The first that we will be hosting with the support of TIBCO, Pivotal, Telstra, Stark & Wayne, among others, is going to be taking place on October 18th in Sydney, Australia. If you have any interest in attending or know of others who may be, please find the registration link here: https://www.eventbrite.com.au/e/cloud-foundry-day-sydney-australia-tickets-27935602138. The second of these events will be taking place on October 20th in Melbourne, Australia and will also be supported by TIBCO, Pivotal, Telstra, Stark & Wayne, among others. The registration link for this event can be found here: https://www.eventbrite.com.au/e/cloud-foundry-day-melbourne-australia-tickets-27985249635. Last, but definitely not least, is the CF Day in Shanghai, China on October 25th where we will be supporting GE (Registration information to be available within the next day or two. Please contact us via twitter if you are interested).

We truly hope that these events will be fruitful in their reach and effectiveness, and help our technical community discern the best option for themselves and the businesses they affect. Teamwork makes the dream work!

A Fresh Developer’s take on Cloud Foundry Why the third platform is the new way to iterate

Why the third platform is the new way to iterate

Coming to the Dojo a little over two months ago for my first day of work at my first (out of college) job, I was mystified by the amount of understanding and knowledge I still had to gain by using the Cloud Platform as a development and deployment environment. After the first few weeks, we started developing a brand new application for internal use. There was no existing infrastructure, but a clear idea to use a SQL database, and a web portal for the UI. I was thrilled! I worked on my fair share of web applications, so this would be a cake walk.

Great! What were the IP’s? What was the endpoint? What was the downtime for our environment on updates? What was our server technology? They said it was up to me. I began thinking of asking IT for a VM spin up our database, scripts designed to back up the data with cron jobs, registering a DNS for our website, and for a potential duplicate set for our test environment; technical aches and pains that are first steps when building a web application. I then began asking what seemed the obvious question of who to go to for the resources, and I was told we would do it all ourselves. Wait, I thought, we were supposed to deploy and maintain the VM’s set up users for the database, download the applications, and backup technology? That was easily another two or three days short term, and days of complications long term. Or so I thought.

Except…I hadn’t known that we were using Cloud Foundry, and the benefits that came from this. All of the pain points I used to face in setting up a functioning web app were moot. Instead of spending time configuring and maintaining environments, the Dojo, as a team, has put together a full-functioning web application with production environment, architecture, an extensive automated testing harness, and automated deployment. All as painlessly as we might use Github.

Removing Development Roadblocks

When I thought of the Web application, a typical structure comes to mind of the architecture for the app.

Screen Shot 2016-08-31 at 12.20.49 PM

A typical developer headache for a web application

There’s plenty of steps for our team, as developers, to maintain and care about here. But with Cloud Foundry, we can remove many of these concerns.

Lets talk about Databases first. Using service brokers as provided through CF, we can simply bind a SQL service to our application. For extra sweetness, we can use a buildpack that supports that our server is connecting to a service-provided database. The buildpack will, at runtime, connect our application’s backend into a bound SQL database, no environment variables necessary.

Servers and UI? No problem. Pushing applications to Cloud Foundry takes as many commands as pushing to git. We can also update and configure the application directly through a UI included in our environment. This means changing something we forgot to put in our three-four deployment commands doesn’t warrant another deployment. As far as firewalls go, using the proper security groups to avoid exposing secretive endpoints is a few minutes behind the scenes of a UI, or a couple commands in a terminal.

Alright, so our development roadblocks are covered. What does all this mean for our Customers?

Rapid Integration and Feedback Loops

With our Test Driven Development and Continuous Integration mantras, we can rapidly iterate on our products. With Cloud Foundry, those rapid iterations allow us to take customer feedback. Using our real product on something as quick to set up and easily accessible as CF platform is, we can respond to our customer’s concerns much more efficiently.

Allowing our customers a way to directly interact with our product in this way is not only helpful for them, but for us, for a variety of reasons. Firstly, we’re able to get valuable feedback from them in a timely fashion, not when a feature has been done and untouched for two months (less old code touching, the better). Additionally, we can test what our real runtime environment will be like for our production level product, which means faster satisfaction for customers.

Screen Shot 2016-08-31 at 12.48.14 PM

Our development cycle. Made simple with less setup behind the scenes

Finally, we don’t have to worry about maintaining servers and VMS when they may go down. Cloud Foundry can sense when an app crashes, and restart it for us. Same database, same DNS, and limited downtime. Doesn’t get much easier than that.

Final thoughts

Using Cloud Foundry also means we can simplify our development process by having a consistent API and well-documented environment surrounding our application. When we rotate around our projects, people coming into our web application are using similar knowledge and practices they might use when deploying our Minecraft Docker Containers.

It’s for the reasons in this article, and I’m sure for many more to come, that the cloud platform and Cloud Foundry has made development easier for us. It allows us to create better products, faster. We can get immediate feedback from customers and get stories accepted faster. For most developers, that’s a no brainer.

Retrospectives Rock! Our way of ensuring continuous improvement: personally, as a team, and in the industry

Our way of ensuring continuous improvement: personally, as a team, and in the industry

Emily Kaiser

Emily Kaiser

Head of Marketing @DellEMCDojo #CloudFoundry #OpenSource #TheWay #LeanPractices #DevOps #Empathy

There is great philosophical and factual validity behind the concept of empathetic reflection affecting positively people’s lives and interactions with others. Throughout history, people have focused their actions and reactions around this idea both subconsciously and consciously. A famous and often used example of this is the Examen as originally defined by St. Ignatius; the concept of through thoughts, words and deeds, reflecting on where you have been (further what brought you elation, sadness, stress, confusion, etc.), where you are, and aligning these things with where you want to go; continuous improvement of the self.

Here at the EMC Dojo, as our name literally implies, we practice “the way.” A large portion of this form of enlightenment is no doubt found in reflection as practiced through Retrospective, or an exercise our team knows as Retro. The exercise is practiced as explained in the aforementioned example as a form of continuous improvement of our methodology. We use the time to align everyone’s perspective and to ensure we are on the same page in terms of action items and overall goals.

Retro is done every Friday without failure. It is about an hour long and is attended by everyone who plays a role in the Dojo’s operations. Even when members of our Dojo family are working off-site, we ensure that a Google Hangout session is available so that they are virtually in attendance, as this is arguably the most important “meeting” all week. It is the time and space for every employee to speak his/her mind and do so with assumed positive intent, so that feedback can constructively be given and received.

The concept is simple: as a whole team, we are living out the mantra we preach. We are learning by doing; generating weekly action and reaction to our development process through empathetic honesty.    

The person running the exercise for the week (this is not assigned per week, but rather a spur of the moment volunteer) will draw on a white board three columns signified by a smiley face (things that went swimmingly), a neutral face (the so-so items), and a sad face (as your teacher used to proclaim “areas for improvement”). Then all members will add topics/items for discussion from the week to each of these columns as he/she sees fit. These items do not need to be evenly distributed by any stretch of the imagination. That is, no person or team collectively must add the same number of items to the smiley face column as they do to the neutral face column and/or the sad face column (this is part of the honesty piece).

The leader of the exercise will then address each topic/item on the board one item at a time from left to right or right to left (we do this to ensure that we are not making the retro top heavy in any one of the columns). By addressing the topic/item, we mean that the leader will ask the team who wrote the item on the board and why. This is where the importance of this exercise comes to play.

No matter the topic of discussion, in order for the exercise to work and for the team to reap all of its benefits, team members must be completely and empathetically honest about his/her contribution. By doing so, the team learns to navigate conflict by having uncomfortable, but necessary discussions. As with anything, with practice comes perfection. And by perfection, I mean the unattainable kind; something that we are still reaching for 🙂 All kidding aside, it’s true – with every week there are steps back that we learn from paired with giant steps toward our overall goals and methodology.

With the addressing of each topic, the team decides action items that are recorded on the spot. This allows us to ensure that we are offering more than an end of iteration wrap-up, and instead generating real actions and change to ourselves, our team and our organization. We are assessing our ability to break down and solve problems and make improvements –measuring, building and learning constantly in our development process (sound familiar?).

While the exercise was nearly impossible to complete during the birth of our DevOps culture, it is now something that we could not live without. Team issues are no doubt as challenging, if not more, than the technical issues we face. Luckily this exercise allows us to face both and do so with our core value of empathy at its center. We not only are then able to end the week (and iteration) on a high note, but also go into our weekend and following iteration rejuvenated. Retros Rock!

One Week at EMC Dojo

Latest posts by Brian Verkley (see all)

I write this while snacking on free banana chips I obtained from the lunchroom. A week in the Dojo is very different than a week programming at home or in a cube farm, and it has as much or more to do with culture than tools and process.

The EMC Dojo in Cambridge, MA (https://www.cloudfoundry.org/welcome-to-the-emc-dojo/) is a place that contributes code to the open source cloud foundry project using a specific development methodology built around pair programming, test driven development, and lean development, known as “the way”. They also teach the way to anyone interested in contributing to Cloud Foundry as well as other development teams within EMC. They earn the title of Dojo by literally being a “place of the way” .

From the moment I got there, I was a member of the team. I had planned to work in the same area, quietly observing the way while working on my own projects. That didn’t work out, and I’m so thankful it didn’t. On the first day, everyone in the Dojo had introduced themselves to me and were interested in what I was working on. They adopted my story into their backlog and encouraged me to attend their standup the next day and begin pair programming with them. I felt so valued before I had even added value. Before long I found myself pair programming on a PHP app (I have no experience in PHP) and doing anything I could to contribute.

They went to lunch together every day I was there, often participating in an office wide Yoga lunch or Go Programming Book Club. They cared about each other’s quality of life and personal hobbies. They struggled out loud with programming syntax and architectural design choices and often came together to discuss, solve, or vote. Every member of the team was heard and had the same weight in decision-making.

And it works. The speed and efficiency that they are able to take a story idea and turn it in to working code is amazing. And although they are all individually quite talented, there seems to be a “greater than the sum of their parts” thing happing with “the way”. They are completed unfazed by tackling something new largely because of their confidence that as a team they will be able to solve it.

What I witnessed, no, what I was a part of for my week, was a team of developers effectively coding for hours based on shared goals and methodology in an environment that made everyone happy. I’m so thankful to everyone who paired with me, shared with me, challenged debated and listened to me, welcomed me, and taught me.

What an outsider might notice first is the ping-pong table next to the cafeteria with available food and no checkout register, but that would be missing the point. The culture that they have built here and the benefits of developing in “the way” have left me with a lot more than this now empty bag of banana chips.