Practical continuous deployment

Balaji Vajjala

Principal DevOps Consultant/Solution Architect

Feb 2014

Created by Balaji Vajjala / @bvajjala

Hi, my name is Balaji Vajjala and I'm the Chief Architet and DevOps Solution Architect. I will talk about implementing a Practical Continuous Deployment pipelines.

Who Am I?

Balaji Vajjala!

  • A Full Stack DevOps Engineer/Solution Architect for 10+ years!
  • Original company sysadmin!
  • Developer for last 4 years!
  • Now working out of East Coast
  • Not a professional speaker

What is DevOps?

  1. Continuous Deployment
  2. Continuous Delivery
  3. Continuous Integration

Continuous Delivery Pipeline Stages

Balaji Vajjala

Principal DevOps Consultant/Solution Architect
Feb 2014

Who Am I?

Balaji Vajjala!

  • A Full Stack DevOps Engineer/Solution Architect for 10+ years!
  • Original company sysadmin!
  • Developer for last 4 years!
  • Now working out of East Coast
  • Not a professional speaker

Continuous Delivery pipeline stages

  1. Commit Phase
  2. Automated Accetance Testing (AAT) Phase
  3. Exploratory Testing Phase (ET)
  4. User Acceptance testing Phase (UAT) Phase
  5. Staging and Pre-Prod Phase
  6. Production Phase

Commit Phase

Commit Phase

  1. Syntax Check
  2. Code Metrics
  3. Compile (For Compiled Languages)
  4. Auto Merge for Development Branches (when using Git-Flow etc.)
  5. Unit Tests

Automated Accetance Testing (AAT) Phase

Automated Accetance Testing (AAT) Phase

  1. Integration Tests
  2. Stubbed & Mocked Endpoints & Data
  3. BDD Framework tests
  4. Component tests
  5. Story-Level tests
  6. Feature-Level tests

Exploratory Testing Phase Phase

Exploratory Testing Phase Phase

  1. Usability Tests
  2. Visual (UI) Tests

User Acceptance testing Phase (UAT) Phase

User Acceptance testing Phase (UAT) Phase

  1. SHOWCASES
  2. FEATURE-LEVEL Testing by the client/customer

Staging and Pre-Prod Phase

Staging and Pre-Prod Phase

  1. Smoke Tests
  2. Performance tests
  3. Capacity tests
  4. Network tests
  5. Post-Deployment tests

Production Phase

Production Phase

  1. Post-Deployment Tests
  2. Ongoing Live Transaction Tests
  3. Rollback & Re-Deploy
  4. Smoke Tests

Continuous Deployment Pipeline #1

Continuous Deployment Pipeline #2

Practical continuous deployment

Balaji Vajjala

Principal DevOps Consultant/Solution Architect
Feb 2014

What I’ve been up to…

  • Last 6 months converting our order systems to high-availability and continuous deployment.
  • Why 6 months?
    • Because the concept is straightforward, but it’s implications affect a lot of your organisation.

What is DevOps?

  • Continuous Deployment
  • Continuous Delivery
  • Continuous Integration

“Deployment”? “Delivery”?

  • Continuous integration is continuous, automated build and test.!
  • Continuous delivery is the next obvious step; be continuously release-ready !
  • Continuous deployment is the final step, the continuous delivery of software to production.

“Deployment”? “Delivery”?

  • Constant QA is the common theme.!
  • In practice there’s a continuous spectrum of options, each organisation has different needs and constraints.!
  • But if you trust your testing and process you can adopt the level appropriate for you.

Why Continuous deployment?

  • We want to release features, not “what ever happens to be done”!
  • Automation: Releasing is hard, automation makes it repeatable!
  • Remove organisational bottlenecks to releases

Why Continuous deployment? Stakeholder benefits

  • To customers: You’ll get your requested feature faster!!
  • To management: You’ll get results faster and clearer progress.!
  • To devs: No more death-marches, mad- dashes, clean-up after releases.!
  • To admins: You know which change broke the system!

Continuous deployment: So how do you actually do it?

  • Continuous deployment guides tend to focus on the high-level philosophy!
  • But how do you actually get a feature from a customer request to your servers?

Development workflow

  • Continuous deployment implies a clearer development process.!
  • You need to know what is going out when you release, not a dump of the current state.!
  • Hence release by feature

Development

  • Track your feature requests in a bug tracker!
  • Branch on each feature, automatically test!
  • Pull requests for code-review/merge!
  • Automatic release to staging on each merge!
  • Promote from staging to production

Step 1: Track your requests

  • Each feature/update request should have a unique ID.!
  • This allows tracking the state of a feature from request to deployment.!
  • Bug-trackers are a good choice for this.

Step 2: Work on this feature in a branch

  • Create a branch for just this feature!
  • Name it after the feature request!
  • Jira/Stash integration will do this!
  • The branch will be merged when complete!
  • You need a sane version control system!
  • We use git, Mercurial is good to.

Step 3: Automatically test the branch

  • Run a continuous integration tool that will automatically run tests against the branch.!
  • Features may not be merged until all tests are passing.!
  • Stash has some features to support this.

Step 4: Code review

  • No code may be merged to the release branch until reviewed by other members of the team.!
  • Team members have a responsibility to ensure quality.

Step 4.1: Stash testing integration



Step 5: Merge and release

  • Once all reviews and tests are passed them merge to release branch!
  • At this point we have a separate Bamboo plan that performs a full release.

Step 6: Deploy to staging

  • Allows testing of more advanced interactions and against production samples.!
  • More testing can occur at this point, including testing by humans.

Step 7: Release to production

  • Valid staging builds may be promoted up to production.

Segue: “Continuous downtime”?

  • So if you’re doing all these releases, what about uptime?!
  • For public-facing service clustering/HA is important.!
  • Ideally you should be able to automate cluster configuration as part of the deployment

Last mile



Practical issue

  • How do you actually get releases onto your staging and production servers?!
  • AKA “the last-mile problem”

Last mile

  • Puppet/Chef are not appropriate!
  • For simple/single-node applications you can use a Bamboo agent directly!
  • For more complex setups use an automation tool

Last mile

  • Puppet/Chef are not appropriate!
  • .. if timing is critical!
  • .. if cross-host coordination required

Last mile

  • Roll your own!
  • Bamboo SSH plugin + bash scripting!
  • Number of existing automation solutions!
  • func, capistrano, SaltStack, Ansible, mcollective, Fabric…

Last mile

  • Bamboo agent per-node!
  • SSH not required!
  • Works for simple (single node) apps! * Coordination is tricky

Last mile

  • Agent-based frameworks!
  • Powerful and flexible!
  • Can parallelise deployments!
  • Requires setup on all nodes!
  • If you already have it setup then use it

Last mile

  • SSH scripting!
  • Requires management of SSH keys on agent!
  • Bamboo SSH plugin!
  • Scripting (Bash, Python, Ruby, etc.)!
  • Automation frameworks (Ansible, SaltStack, Func, Fabric)

Last mile

  • Our solution!
  • Ansible for automation (explicit support for load-balancer integration)!
  • Minimal requirements, SSH+Python!
  • Bamboo pulls Ansible directly from their source repository!
  • Ansible playbooks checked into git

Practical issue

  • How do you manage what has been released, and to where?!
  • How do you control who performs deployments?

Bamboo deployment environments

  • The release build plan can be associated with certain environments!
  • Normal ones are dev, staging (QA) and production

Bamboo deployment environments



Bamboo deployment environments

  • Environment has tasks, like a build plan!
  • Tasks perform the actual deployment!
  • Environments have permissions, limiting who may perform deployments!
  • Generates releases, which are deployed!
  • Has some nice integrations…

Bamboo deployment release



Bamboo deployment JIRA integration



Procedural issues

  • Where’s the oversight in all this?!
  • What about SoX, PCI, SEC requirements?!
  • Who is allowed to do releases?!
  • Who signs off?

Procedural issues

  • Our solution – separate the infrastructure!
  • Dedicated Bamboo server for business software!
  • Dedicated agents for building!
  • Separate, dedicated agents for deployment

Procedural issues

  • Access controls!
  • Build team/admins control the server!
  • Business analysts define features!
  • Devs code, review, merge and release!
  • Features pushed to staging for BA review!
  • BAs can promote releases to production

Questions?

Balaji Vajjala @bvajjala http://bvajjala.github.io/

Thanks!