How long does it take?

Here is the scenario, a user of your system (end-user, business, stakeholder, etc) finds a bug, which gets reported to you. You analyse it, you know how to fix it, you go right ahead and code it in; but, how long does it take for your user to actually see it implemented? Think of that bug as any code change; a real defect, a new feature, whatever; if it takes a few hours to get it coded, why does the user need to wait a week, or two, to actually see it? Oh!, I see, this is why:

• the developer needs to stop doing what he is doing, developing!
• build the code
• change a few configs
• make the deploy package
• submit a request to the deployment team
• call one of your contacts cause deployment team is not responding
• call again, luckily someone is available
• got an email “Deployment completed!”
• open the site to run a smoke test, 500s all over the place :o
• jump on a conference call, bla bla bla
• … (oh my God, I said not to delete that file)
• deployment finished
• let QA know to go ahead and test it
• QA is in the other side of the world, not gonna happen today
• QA finds something, another email, another day
• change the code, another email, another day
• done! staging looks good; what, staging? Yes, but don’t worry, move this to production will be straightforward (ha!)
• (more or less, I think I made my point clear)

Continuous Delivery (CD) has being around from the early days of the agilism (is that a word?) and extreme programming, and before that as well, if we dig into the lean manufacturing principles of the Toyota Production System. In these days, and with the popularity of DevOps and the cloud, CD has exploded and tons of options are available to overcome the aforementioned situation, technology-specific as well as technology-agnostic.

I was not planning on writing this much, I rather show what I got. The following is just one setup that I have for one specific project. If you see it with clinic eyes, you will tell areas of improvement, there are, but it fulfils the needs as is (push your decisions up to the point where you really need to make them, rather than “guessing” what’s gonna happen, as Martin Alaimo told me once).

Alright, this is my deployment pipeline, it is a visual representation of the health of the system and environments. The more boxes (jobs) you have, probably better, since if anyone turns red, you know exactly where to get your hands on.

This is running on a GUI-less Ubuntu server, Jenkins is the continuous integration server, and through the build pipeline plugin is how you get the graphical dashboard.

Lets break down each job to understand its responsibility.

• CEMS_CI
• Git is used as a SCM, so the Jenkins git plugin needs to be installed
• Poll the upstream git repository at night, which is alright in terms of a daily build, but not exactly what I would like, I tell you why. Jenkins exposes a REST webapi that you can leverage to request a given job to run. That said, it is fairly simple to configure a post-receive git hook to run a curl command, so with that, what you would accomplish is to have one build per developer’s commit (even more frequent feedback of the system health). 
  In the case of this project, the git repo is an in-company instance of gitorious, so a development team has no access to manipulate the given hooks. Github does give you that possibility, which is neat, so if Github is your upstream repo, and your Jenkins is accessible through the internet, you could accomplish this.
• Pull the latest changes from the upstream git repository. This will look only after to the master branch, so any code not ready to be integrated needs to be on its own branch (refer to this great branching strategy).
  The job will run under the jenkins user, so you need to make sure that it is able to ssh connect to the repo, load everything needed on its PATH, rvm etc etc (not the scope of this post)
• Run a bash script to install new ruby gems and run common rake tasks. The last task, running your tests, should really be your QA. If all tests pass, you must be able to ship the software without fear.
  

   #!/bin/bash
   bundle install
   bundle exec rake db:migrate
   bundle exec rake db:seed
   bundle exec rake db:test:prepare
   bundle exec rake ci:setup:rspecdoc spec

• Publish rails stats and ruby code coverage results, which in terms of quality, you can set it up to make the build fail if it goes below a given % threshold
  
• Tag the upstream repo for traceability (read more about git tags)
  
• and finally, if all above steps are ok, automatically trigger the next job Deploy to Dev
• Deploy to Dev
• Clone the repo, push it to a bare local one, created to mimic the upstream one (given that one is within our control, we can manage the hooks at our own will)

   #!/bin/bash -e
   echo "Preparing to deploy dev..."
  
   if [ -d "./tmp" ]; then
     echo "Removing old files"
     rm -rf "./tmp"
   fi
  
   mkdir tmp
   git clone git@[repo_url].git tmp
   
   cd tmp
   git remote add [remote] "/opt/git/[local_repo].git"
   
   echo "Pushing source code to dev..."
   git push [remote] master
   
   echo "Done!"

• Now, on your post-receive hook of the local repo, run this (make sure it is executable first)

   $ chmod +x hooks/post-receive 

   # hooks/post-receive
   
   #!/bin/sh
   # Deploy Development environment
   
   echo "Sit back, your deployment is kicking off..."
   
   # Kill server
   kill -9 $(pidof ruby)
   
   APP_PATH="[my_app_path]"
   
   if [ -d "$APP_PATH" ]; then
     echo "Backing up old version..."
     ./backup_site.sh
     rm -rf $APP_PATH
   fi
   
   cd $APP_PATH
   unset GIT_DIR
   git pull origin master
   bundle install
   bundle exec rake db:migrate
   bundle exec rake db:seed
   
   # Start server once again on the background
   nohup rails server &> /dev/null &

• and finally, if all above steps are ok, automatically trigger the next job Dev Readiness
• Dev Readiness
• Its responsibility is to ask the app if it is up and running by executing a curl command to a preconfigured route

   #!/bin/bash
   curl -X GET 'http://localhost:[port]/health' -I 

• finally, its post-build action is to set to manual-trigger a deploy to the next environment (this is the only human interaction to get app on the next server, upgraded and running)

The Deploy to Stage job will pretty much do the same as Deploy to Dev. We can then add jobs like this one for any other environment we want to manage, with pretty much very low effort, repeatable, repeatable, repeatable.

Wrapping up, I don't want to leave without mentioning Nico Paez, who was the one who has shown me the Jenkins build pipeline for the first time, in one of the many agile events we connect in.