Continuous Integration for Golang Applications with Terraform, Codebuilds and ECR

I wrote this post for sharing, how we can do continuous integration for Golang APIs with some tools. In this post I’m going to use terraform for making a codebuild resource, Code build to build the Golang API and generate a docker image and ECR for keeping the docker image.

The workflow:

The continuous integration workflow is very easy to understand, every time a new tag is created the webhook will start the job, Codebuild will compile the Golang API and will generate the docker image, then will run a docker push to AWS ECR,

For this exercise, I pushed the code at https://github.com/braybaut/Golang-API This repository has the Golang API, Terraform files to deploy the AWS resources and buildspec to generate the steps

This code can be consulted at Github: https://github.com/braybaut/Golang-API

├── books.go
├── config
│   ├── aws.tf
│   ├── buildspec.yml
│   ├── codebuild.tf
│   ├── terraform.tfvars
│   └── variables.tf
├── Dockerfile
├── handlers.go
├── main.go
├── models.go
├── README.md
├── router.go
└── routes.go

The Golang Application is a minimal API that just show a little information of some books information that I’ve read:

Routes:


func init() {
	routes = Routes{
		Route{
			"Index",
			"GET",
			"/",
			"Index",
			Index,
		},
		Route{
			"TodoShow",
			"GET",
			"/books/{bookId}",
			"Get specific book with ID",
			GetBook,
		},
		Route{
			"TodoIndex",
			"GET",
			"/books",
			"Get all Books",
			GetBooks,
		},

		Route{
			"info",
			"GET",
			"/info",
			"Get Paths info",
			GetPaths,
		},
	}
}

I won’t talk much about the Golang code if you want to know more about the code, you can see the code at GitHub (I can write a post about golang in the future).

In orchestration Environment where many microservices run (for example golang) we need to pack our microservices, yes… the Golang binary can be our artifact but in this scenario, the real artifact is the Docker images that keep the Golang binary. For this case codebuild must compile the source code and generate the docker image, I wrote this buildspec.yml that run a docker build and docker do magic!!

Dockerfile:


FROM golang:alpine AS  builder 

RUN apk update && apk add --no-cache git

RUN mkdir /app
ADD . /app
WORKDIR /app
RUN go get -d -v  
RUN go build -o library

FROM golang:alpine

COPY --from=builder /app/library /

EXPOSE 8080

ENTRYPOINT ["/library"]

The Dockerfile generates the Golang binary (library) y run this like entry point, also use the feature (multistage build) where we can make more optimized images (less size, fewer layers, etc)

Codebuild will handle the process of generating the docker image, for this, we can define the instruction (phases) with a yml file ( buildspec.yml)

The buildspec.yml must run these steps:

login to ECR
Docker build and tag the docker image
Docker push to ECR

buildspec.yml

version: 0.2

git-credential-helper: yes

phases:
    install:
        runtime-versions:
            docker: 18
    pre_build:
        commands:
            - echo "loggin to AWS ECR"
            - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)

    build:
        commands:
            - echo "Getting Tag.."
            - |
              if [ -z $CODEBUILD_WEBHOOK_TRIGGER ]; then  
                export IMAGE_TAG=$CODEBUILD_SOURCE_VERSION
              else 
                export IMAGE_TAG=$(echo $CODEBUILD_WEBHOOK_TRIGGER | cut -d "/" -f2)
                fi 
            - echo "verify tag.."
            - echo $IMAGE_TAG 
            - echo "Building the docker image.."
            - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
            - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG 
    post_build:
        commands:
            - echo "Pushing Docker image"
            - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG

How determinate the tag?

Our pipeline can start manually or automatically, if start manually we can set the repository tag to build, this tag is keeping an environment variable CODEBUILD_SOURCE_VERSION, but if start automatically the value is keeping in CODEBUILD_WEBHOOK_TRIGGER to know this, the build stage has a conditional to get the correct tag


if [ -z $CODEBUILD_WEBHOOK_TRIGGER ]; then  
   export IMAGE_TAG=$CODEBUILD_SOURCE_VERSION
else 
   export IMAGE_TAG=$(echo $CODEBUILD_WEBHOOK_TRIGGER | cut -d "/" -f2)
fi

We have the golang code and dockerfile to build the docker image and buildspec.yml, now we need the terraform file to deploy codebuild and ECR.

Deploying Codebuild:

For this point, we have some terraform files, for example:

terraform.tfvars: Set the needed variables for example AWS_ACCOUNT_ID

AWS_ACCOUNT_ID=
AWS_REGION="us-east-1"
IMAGE_REPO_NAME="library-api"
URL_REPO="https://github.com/braybaut/Golang-API.git"

variables.tf: make the variables with terraform.tfvars values


variable "URL_REPO" {
  type = string

}

variable "AWS_REGION" {
  type = string
}

variable "IMAGE_REPO_NAME" {
  type = string
}

variable "AWS_ACCOUNT_ID" {
  type = string
}

aws.tf: Set the provider to use

provider "aws" {
  region = "${var.AWS_REGION}"
}

codebuild.tf: policies, roles, codebuild, ECR to create

This file has S3, ECR, policies, role, and other definitions that are needed to this workflow, but I just paste the codebuild resources, this definition only set some environment variables needed to Buildspec, set the log-stream configuration and define the source

resource "aws_codebuild_project" "library-build" {
  name = "library-build"
  description = "Easy build to generate docker image for Golang Applications"
  build_timeout = "5"
  service_role = "${aws_iam_role.code-build-library.arn}"

  artifacts {
    type = "NO_ARTIFACTS"
  }

  cache {
    type = "S3"
    location = "${aws_s3_bucket.library-build.bucket}"
  }

  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image = "aws/codebuild/standard:2.0"
    type = "LINUX_CONTAINER"
    image_pull_credentials_type = "CODEBUILD"
privileged_mode = true

   environment_variable {
      name = "AWS_DEFAULT_REGION"
      value = "${var.AWS_REGION}"
    }
   environment_variable {
      name = "AWS_ACCOUNT_ID"
      value = "${var.AWS_ACCOUNT_ID}"
    }

    environment_variable {
      name = "IMAGE_REPO_NAME"
      value = "${var.IMAGE_REPO_NAME}"
    }

  }

  logs_config {
    cloudwatch_logs {
      group_name = "log-group"
      stream_name = "log-stream"
    }

    s3_logs {
      status = "ENABLED"
      location = "${aws_s3_bucket.library-build.id}/build-log"
    }
  }

  source {
    type = "GITHUB"
    location = "${var.URL_REPO}"
    git_clone_depth = 1
    buildspec = "${file("buildspec.yml")}"
  }
 
}

Enabling webhook:

This definition will create a webhook to start the job, this webhook has filters that exclude everything that meets the “^refs/heads/.” pattern ( new commits, new branches, etc ) and the hook will trigger when the pattern “^refs/tags/.” is met ( create new tags )

resource "aws_codebuild_webhook" "library-webhook" {
  project_name = "${aws_codebuild_project.library-build.name}"

  filter_group {
    filter {
      type = "EVENT"
      pattern = "PUSH"
    }

   filter {
      type = "HEAD_REF"
      pattern = "^refs/tags/.*"
    }
   filter {
      type = "HEAD_REF"
      pattern = "^refs/heads/.*"
      exclude_matched_pattern = "true"
    }


  }
}

Now that we have all files to make the pipeline, we need to deploy all resources, to Deploy terraform files is needed set the AWS keys as environment variables, then run terraform init and terraform apply commands:

Terraform will create 7 resources, if the creation process was good, we can see these resources created:

    aws_codebuild_project.library-build
    aws_codebuild_webhook.library-webhook
    aws_ecr_repository.library-ecr
    aws_iam_role.code-build-library
    aws_iam_role_policy.ecr-policy
    aws_iam_role_policy.s3-policy
    aws_s3_bucket.library-build

we can see the new Codebuild project and the ECR created

Codebuild project:

ECR

The resources is created, we can test the webhook create a new tag from github:

we create a new release:

Verify that the build is running:

The build was started, also we can see the logs

When the job finished, we can see the new docker image at ECR

Our Continuous integration workflow finish there, we can see that the docker image tag is the same to the tag set as release at github:

This was a little example that how we can create an easy continuous integration for Golang APIs with Codebuild, GitHub and Terraform, In the next post I will write how can create a continuous delivery workflow with CodeDeploy and Fargate and other post joining both workflows to make a pipeline with continuous integration and delivery workflow.