Balaji Vajjala's Blog

A DevOps Blog from Trenches

05

  • 5.1 Create a CloudFormation template
    • Learning Objectives
    • CloudFormation
    • Create and run a CloudFormation template
    • Launch CloudFormation stack based on template

5.1 Create a CloudFormation template

Learning Objectives

By the end of this lesson you will be able to:

  • Create and run a CloudFormation template.

CloudFormation

Slide1

Create and run a CloudFormation template

CloudFormation is described as a JSON (JavaScript Object Notation) template. It’s a model-driven template in that the AWS infrastructure is instantiated according to its own specification of proper order of execution. It is not a procedural language. If you are writing a CloudFormation template, you only need to follow the rules of the CloudFormation external Domain Specific Language (DSL) in JSON notation.

  1. https://github.com/stelligent/devopsinthecloud/blob/master/infrastructure/templates/jenkins.template
  2. From your local GitHub repository, go to devopsinthecloud/infrastructure/templates/ and open the my myjenkins.template file. A copy of the file is located at https://github.com/stelligent/devopsinthecloud/blob/master/infrastructure/templates/

  3. Within the Parameters section, add the following parameters to the CloudFormation myjenkins.template file. Parameters are custom fields that users of the CloudFormation template enter to configure their environment. You can enter these parameters through the CloudFormation wizard available through the AWS Management Console, the command line interface or through the CloudFormation API. Within each parameter you can setup constraints for these parameters. Constraints may describe things like the Min and Max Size, whether or not to echo back what the user is entering in the parameter field, etc. There’s a syntax to describing a Parameter. First, you define the name Parameters in quotes followed by a colon. Then, you put an open curly brace. You use commas to delimit each parameter with the exception of the last parameter. Each parameter name is a custom name that you come up with. Each parameter name is in quotes. Then, you define each of the properties for the parameter. A parameter can be defined as a String, a Number of a CommaDelimitedList

    “` “KeyName” : { “Description” : “Name of an existing EC2 KeyPair to enable SSH access to the instances”, “Type” : “String”, “Default” : “ditc”, “MinLength”: “1”, “MaxLength”: “64”, “AllowedPattern” : “[–_ a-zA-Z0-9]*”, “ConstraintDescription” : “Can contain only alphanumeric characters, spaces, dashes and underscores.” }, “InstanceType” : { “Description” : “WebServer EC2 instance type”, “Type” : “String”, “Default” : “c1.medium”, “ConstraintDescription” : “Must be a valid EC2 instance type.” }, “PrivateBucket” : { “Description” : “S3 bucket for storing credentials”, “Type” : “String”, “Default” : “ditcpmedcreds”, “ConstraintDescription” : “Must be a valid S3 Bucket” }, “PublicBucket” : { “Description” : “S3 bucket for storing build artifacts”, “Type” : “String”, “Default” : “stelligentlabs”, “ConstraintDescription” : “Must be a valid S3 Bucket” }

    “`

  4. Within the Mappings section, add the following parameters to the CloudFormation myjenkins.template file. Mappings are key/value pairs. In this template, I’m defining the AWSInstanceType2Arch to use a t1.micro 64-bit instance with this AMI. Typically, you might have multiple AMI’s based on a region. This is referred to later when defining a launch configuration.

    "AWSInstanceType2Arch" : { "t1.micro": { "Arch" : "64" }, "m1.large": { "Arch" : "64" }, "m1.xlarge" : { "Arch" : "64" }, "m2.xlarge" : { "Arch" : "64" }, "m2.2xlarge" : { "Arch" : "64" }, "m2.4xlarge" : { "Arch" : "64" }, "c1.medium" : { "Arch" : "64" }, "c1.xlarge" : { "Arch" : "64" }, "cc1.4xlarge" : { "Arch" : "64" } }, "AWSRegionArch2AMI" : { "us-east-1" : { "32" : "ami-7f418316", "64" : "ami-7341831a" }, "us-west-1" : { "32" : "ami-951945d0", "64" : "ami-971945d2" }, "us-west-2" : { "32" : "ami-16fd7026", "64" : "ami-10fd7020" }, "eu-west-1" : { "32" : "ami-24506250", "64" : "ami-20506254" }, "ap-southeast-1" : { "32" : "ami-74dda626", "64" : "ami-7edda62c" }, "ap-northeast-1" : { "32" : "ami-dcfa4edd", "64" : "ami-e8fa4ee9" } }

  5. Within the Resources section, add the following configuration to the CloudFormation myjenkins.template file. The Resources section is where most of the “real work” is done in a CloudFormation template. Resources define how the various AWS resources are configured as part of your infrastructure. This includes most of the AWS resources such as EC2 Instance, EC2 Security Group, Auto Scaling LaunchConfiguration and Amazon CloudWatch Alarm

    “` “CfnUser” : { “Type” : “AWS::IAM::User”, “Properties” : { “Path”: “/”, “Policies”: [ { “PolicyName”: “Admin”, “PolicyDocument”: { “Statement”: [ { “Effect”:“Allow”, “Action”:“”, “Resource”:“” } ]}
    } ] } },

    “PrivateBucketPolicy” : { “Type” : “AWS::S3::BucketPolicy”, “Properties” : { “PolicyDocument”: { “Id”:“PrivateBucketPolicy”, “Statement”:[ { “Sid”:“ReadAccess”, “Action”:[“s3:GetObject”], “Effect”:“Allow”, “Resource”: { “Fn::Join” : [“”, [“arn:aws:s3:::”, { “Ref” : “PrivateBucket” } , “/*” ]]}, “Principal”:{ “AWS”: { “Fn::GetAtt” : [ “CfnUser”, “Arn” ]} } } ] }, “Bucket” : {“Ref” : “PrivateBucket”} } },

    “HostKeys” : { “Type” : “AWS::IAM::AccessKey”, “Properties” : { “UserName” : {“Ref”: “CfnUser”} } },

    “WebServer”: {
    “Type”: “AWS::EC2::Instance”, “DependsOn” : “PrivateBucketPolicy”, “Metadata” : { “AWS::CloudFormation::Init” : { “config” : { “packages” : { “yum” : { “java-1.6.0-openjdk” : [], “tomcat6” : [], “git” : [], “make” : [], “gcc” : [], “sqlite-devel” : [], “libxml2-devel” : [], “libxslt-devel” : [], “libyaml-devel” : [] } },

        "files" : {
      "/usr/share/tomcat6/webapps/jenkins.war" : { 
        "source" : "http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war",
        "mode"   : "000500", 
        "owner"  : "tomcat",
        "group"  : "tomcat" 
      },

      "/usr/share/tomcat6/sqs_receive_message.rb" : { 
    "source" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/", { "Ref" : "PublicBucket" }, "/scripts/sqs_receive_message.rb"]]},
        "mode"   : "000500", 
        "owner"  : "tomcat",
        "group"  : "tomcat",
        "authentication" : "S3AccessCreds"
      },

"/usr/share/tomcat6/sqs_send_message.rb" : { 
        "source" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/", { "Ref" : "PublicBucket" }, "/scripts/sqs_send_message.rb"]]},
        "mode"   : "000500", 
        "owner"  : "tomcat",
        "group"  : "tomcat",
        "authentication" : "S3AccessCreds"
      },

"/usr/share/tomcat6/terminate.rb" : { 
        "source" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/", { "Ref" : "PublicBucket" }, "/scripts/terminate.rb"]]},
        "mode"   : "000500", 
        "owner"  : "tomcat",
        "group"  : "tomcat",
        "authentication" : "S3AccessCreds"
      },
      "/usr/share/tomcat6/.ssh/known_hosts" : {
        "source" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/", { "Ref" : "PrivateBucket" }, "/known_hosts"]]},
        "mode"   : "000644",
        "owner"  : "tomcat",
        "group"  : "tomcat",
        "authentication" : "S3AccessCreds"
      },
"/usr/share/tomcat6/.ssh/id_rsa" : {
        "source" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/", { "Ref" : "PrivateBucket" }, "/id_rsa"]]},
        "mode"   : "000600",
        "owner"  : "tomcat",
        "group"  : "tomcat",
        "authentication" : "S3AccessCreds"
      },

"/etc/cron.hourly/jenkins_versioning.sh" : {
        "source" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/", { "Ref" : "PublicBucket" }, "/scripts/jenkins_versioning.sh"]]},
        "mode"   : "000500", 
        "owner"  : "tomcat",
        "group"  : "tomcat",
        "authentication" : "S3AccessCreds"
}
    }
  }
},

    “AWS::CloudFormation::Authentication” : { “S3AccessCreds” : { “type” : “S3”, “accessKeyId” : { “Ref” : “HostKeys” }, “secretKey” : {“Fn::GetAtt”: [“HostKeys”, “SecretAccessKey”]}, “buckets” : [ { “Ref” : “PrivateBucket” }, { “Ref” : “PublicBucket”} ] } } }, “Properties”: { “ImageId” : { “Fn::FindInMap” : [ “AWSRegionArch2AMI”, { “Ref” : “AWS::Region” }, { “Fn::FindInMap” : [ “AWSInstanceType2Arch”, { “Ref” : “InstanceType” }, “Arch” ] } ] }, “InstanceType” : { “Ref” : “InstanceType” }, “SecurityGroups” : [ {“Ref” : “FrontendGroup”} ], “KeyName” : { “Ref” : “KeyName” }, “Tags” : [{ “Key” : “Name”, “Value” : “Jenkins” }], “UserData” : { “Fn::Base64” : { “Fn::Join” : [“”, [ “#!/bin/bash -v\n”, “date > /home/ec2-user/starttime\n”, “yum update -y aws-cfn-bootstrap\n”,

      "# Install packages\n",
  "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackName" }, " -r WebServer ",
  "    --access-key ",  { "Ref" : "HostKeys" },
  "    --secret-key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},
  "    --region ", { "Ref" : "AWS::Region" }, " || error_exit 'Failed to run cfn-init'\n",

    “# Copy Github credentials to root ssh directory\n”, “cp /usr/share/tomcat6/.ssh/* ~/.ssh/\n”,

    “# Update Jenkins with versioned configuration\n”, “rm -rf /usr/share/tomcat6/.jenkins\n”, “git clone git@github.com:stelligent/devopsinthecloudjenkins.git /usr/share/tomcat6/.jenkins\n”,

    “# Installing Ruby 1.9.3 from RPM\n”, “wget https://s3.amazonaws.com/stelligentlabs/resources/rpm/ruby-1.9.3p0-2.amzn1.x86_64.rpm\n”, “rpm -Uvh ruby-1.9.3p0-2.amzn1.x86_64.rpm\n”,

    “# Install Jenkins Plugins\n”, “wget -P /usr/share/tomcat6/.jenkins/plugins/ http://updates.jenkins-ci.org/download/plugins/git/1.1.16/git.hpi\n”, “wget -P /usr/share/tomcat6/.jenkins/plugins/ http://updates.jenkins-ci.org/download/plugins/s3/0.2.0/s3.hpi\n”, “wget -P /usr/share/tomcat6/.jenkins/plugins/ http://updates.jenkins-ci.org/download/plugins/jenkins-cloudformation-plugin/0.9/jenkins-cloudformation-plugin.hpi\n”, “wget -P /usr/share/tomcat6/.jenkins/plugins/ http://updates.jenkins-ci.org/download/plugins/build-pipeline-plugin/1.2.3/build-pipeline-plugin.hpi\n”, “wget -P /usr/share/tomcat6/.jenkins/plugins/ http://updates.jenkins-ci.org/download/plugins/github/1.2/github.hpi\n”, “wget -P /usr/share/tomcat6/.jenkins/plugins/ http://updates.jenkins-ci.org/download/plugins/dashboard-view/2.2/dashboard-view.hpi\n”,

    “# Install Bundler\n”, “gem install bundler\n”, “gem install aws-sdk\n”, “gem install cucumber\n”, “gem install net-ssh\n”, “gem install capistrano\n”,

    “# Add Tomcat user to sudoers and disable tty\n”, “echo "tomcat ALL=(ALL) NOPASSWD:ALL\” >> /etc/sudoers\n”, “echo "Defaults:%tomcat !requiretty\” >> /etc/sudoers\n”, “echo "Defaults:tomcat !requiretty\” >> /etc/sudoers\n”,

    “# Add AWS Credentials to Tomcat\n”, “echo "AWS_ACCESS_KEY=”, { “Ref” : “HostKeys” }, “\” >> /etc/sysconfig/tomcat6\n”, “echo "AWS_SECRET_ACCESS_KEY=”, {“Fn::GetAtt”: [“HostKeys”, “SecretAccessKey”]}, “\” >> /etc/sysconfig/tomcat6\n”, “echo "AWS_CLOUDFORMATION_HOME=/opt/aws/apitools/cfn/\” >> /etc/sysconfig/tomcat6\n”,

    “# Add CloudFormation CLI tools\n”, “wget -P /opt/aws/apitools/ https://s3.amazonaws.com/stelligentlabs/CloudFormation-CLI.tar.gz\n”, “tar -C /opt/aws/apitools/ -xf /opt/aws/apitools/CloudFormation-CLI.tar.gz\n”,

    “# Setup deployment directory\n”, “mkdir /var/www/rails\n”, “sudo chown -R ec2-user:ec2-user /var/www/rails\n”,

    “# Tomcat Setup\n”, “chown -R tomcat:tomcat /usr/share/tomcat6/\n”, “service tomcat6 start\n”,

    “/opt/aws/bin/cfn-signal”, “ -e 0”, “ ‘”, { “Ref” : “WaitHandle” }, “’”,“\n”,

    “date > /home/ec2-user/stoptime” ]]}}
    } },

    “IPAddress” : { “Type” : “AWS::EC2::EIP” },

    “IPAssoc” : { “Type” : “AWS::EC2::EIPAssociation”, “Properties” : { “InstanceId” : { “Ref” : “WebServer” }, “EIP” : { “Ref” : “IPAddress” } } },

    “FrontendGroup” : { “Type” : “AWS::EC2::SecurityGroup”, “Properties” : { “GroupDescription” : “Enable SSH and access to Apache and Tomcat”, “SecurityGroupIngress” : [ {“IpProtocol” : “tcp”, “FromPort” : “22”, “ToPort” : “22”, “CidrIp” : “0.0.0.0/0”}, {“IpProtocol” : “tcp”, “FromPort” : “80”, “ToPort” : “80”, “CidrIp” : “0.0.0.0/0”}, {“IpProtocol” : “tcp”, “FromPort” : “8080”, “ToPort” : “8080”, “CidrIp” : “0.0.0.0/0”} ] } },

    “WaitHandle” : { “Type” : “AWS::CloudFormation::WaitConditionHandle” },

    “WaitCondition” : { “Type” : “AWS::CloudFormation::WaitCondition”, “DependsOn” : “WebServer”, “Properties” : { “Handle” : { “Ref” : “WaitHandle” }, “Timeout” : “1200” } }

    “`

  6. Within the Outputs section, add the following parameters to the CloudFormation myjenkins.template file. Outputs is an optional section where you return values to the user of the CloudFormation stack. Outputs are displayed in the Outputs section in the Management Console and when running the cfn-describe-stacks command.

    “` “InstanceIPAddress” : { “Value” : { “Ref” : “IPAddress” } }, “JenkinsURL” : { “Value” : { “Fn::Join” : [“”, [“http://”, { “Ref” : “IPAddress” }, “:8080/jenkins”]] }, “Description” : “URL for newly created Jenkins app” } }

    “`

  7. Resources for using CloudFormation:CloudFormation Getting Started. CloudFormation API Reference. CloudFormation Command Line Reference. AWS Resource Types Reference.

Launch CloudFormation stack based on template

  1. Click this link to launch CloudFormation.
  2. Click the Create New Stack button
  3. Enter a name in the Stack Name field.
  4. Click the Upload a Template File radio button and the Choose File button. Upload the myjenkins.template you created and click the Continue button.
  5. Modify any of the parameters as necessary and complete the rest of the wizard.