## Introduction When working in large distributed environments, Docker can simplify the use of service discovery tools by packaging each service in separate containers. There are multiple tools built around the Docker ecosystem that make using Docker in your infrastructure an advantage in regards to service discovery. In this post I'm going to use Consul, Terraform, Docker, and Ansible to build a small distributed web application. We'll see how to use Terraform and Ansible to build and provision the infrastructure, and how to use Docker and Consul to make different components discover one another. ## Consul Consul has multiple components, but as a whole, it is a tool for discovering and configuring services in your infrastructure. It provides several key features: * __Service Discovery:__ Clients of Consul can provide a service, such as api or mysql, and other clients can use Consul to discover providers of a given service. Using either DNS or HTTP, applications can easily find the services they depend upon. * __Health Checking:__ Consul clients can provide any number of health checks, either associated with a given service ("is the webserver returning 200 OK"), or with the local node ("is memory utilization below 90%"). This information can be used by an operator to monitor cluster health, and it is used by the service discovery components to route traffic away from unhealthy hosts. * __Key/Value Store:__ Applications can make use of Consul's hierarchical key/value store for any number of purposes, including dynamic configuration, feature flagging, coordination, leader election, and more. The simple HTTP API makes it easy to use. * __Multi Datacenter:__ Consul supports multiple datacenters out of the box. This means users of Consul do not have to worry about building additional layers of abstraction to grow to multiple regions. Consul is designed to be friendly to both the DevOps community and application developers, making it perfect for modern, elastic infrastructures. Another great tool created by Jeff Lindsay is called Registrator, which automatically registers and derigesters services for Docker containers. It inspects the Docker container and register its address and exposed ports. By using these tools we can solve most of the problems around service discovery with Docker containers. ## Terraform Terraform is used to build, create, and version infrastructure effortlessly with some basic code in its configuration file and it's a great example of infrastructure as code. Terraform is used to create and manage different infrastructure resources, like physical machines, VMs and even Docker containers. It supports different Cloud service providers like AWS and DigitalOcean, and it can also manage low level components such as instances, storage, and networking. It can even manage DNS entries in Cloudflare, databases in MySQL, and Consul key/value store, all of which can be managed by a simple configuration file that defines several resources related to different services and providers. Even though Terraform is a multi-purpose tool that can manage low and high level components, it can't offer the full control over configuration management which is where Ansible comes in. ## Prerequisites This article assumes that you have basic knowledge about the following technologies: * Docker * Ansible * AWS ## Let's Get Started In this setup we're going to deploy a simple, yet powerful publishing platform called Ghost. Ghost is a NodeJS application that uses SQL database backend (PostgreSQL, MySQL, or SQLite). The application is configurable through one file called config.js in its root directory and can be used to set the values for the database and mail configuration. The setup will consist of the following: * Load balancer: Nginx. * Four application servers with Ghost installed. * Database server: MySQL * Consul Server All the applications including Nginx, Ghost and MySQL are dockerized, and we'll use AWS as cloud service provider. Each server will be created using Terraform, each application will require knowledge of database server address. The load balancer will need to know about the Ghost application server's addresses and ports. ![Screen Shot 2016-01-11 at 7.21.41 AM](https://cdn.levvel.io/blog_content/Screen-Shot-2016-01-11-at-7.21.41-AM.png) Each server will contain a Registrator container that will add the services to the Consul server: ``` root@lb1:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e64708af135c nginx:latest "nginx -g 'daemon off" 23 hours ago Up 22 hours 0.0.0.0:80->80/tcp, 443/tcp nginx-lb1 9797b30402df gliderlabs/registrator:latest "/bin/registrator -ip" 23 hours ago Up 23 hours registrator-lb1 ``` The registrator will register the Nginx container with consul so that Consul will be aware of Nginx service that has port 80 open: ``` root@lb1:~# docker logs registrator-lb1 2016/01/08 20:42:04 Starting registrator v6 ... 2016/01/08 20:42:04 Forcing host IP to 52.25.58.215 2016/01/08 20:42:04 consul: current leader 172.17.0.2:8300 2016/01/08 20:42:04 Using consul adapter: consul://52.34.89.135:8500 2016/01/08 20:42:04 Listening for Docker events ... 2016/01/08 20:42:04 Syncing services on 1 containers 2016/01/08 20:42:04 ignored: 9797b30402df no published ports 2016/01/08 20:42:40 ignored: e64708af135c port 443 not published on host 2016/01/08 20:42:40 added: e64708af135c lb1:nginx-lb1:80 ``` ## Building the Servers The servers will be built on AWS using Terraform and the following Is the configuration file that will spin up 6 servers. First it will configure the AWS provider using the following section: ```ruby # Configure the AWS Provider provider "aws" { access_key = "${var.access_key}" secret_key = "${var.secret_key}" region = "us-west-2" } ``` This will tell Terraform that I will use AWS with the variables access_key and secret_key, and all servers will be in us-west-2 region. ```ruby # Configure Security group resource "aws_security_group" "default" { name = "aws_default" description = "Default security group" # inbound access from anywhere ingress { from_port = 0 to_port = 65535 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } # outbound internet access egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } ``` The last step before creating the servers is to define the authentication key that will be deployed to each server: ```ruby resource "aws_key_pair" "auth" { key_name = "key" public_key = "ssh-rsa xxxxxxxx" } ``` The following section will define the aws_instance resource and will create the load balancer server. Note that I used “group” tag to be used later with Ansible, I will use Ubuntu-14.04 LTS OS on each server. ```ruby # Create load balancer resource "aws_instance" "lb" { connection { user = "ubuntu" } instance_type = "t2.micro" tags { Name = "lb1" Group = "lbs" } security_groups = ["aws_default"] ami = "ami-5189a661" key_name = "key" } ``` In the same manner, the following will describe how to create application servers, database server, and consul server. ```ruby # Create apps resource "aws_instance" "app1" { connection { user = "ubuntu" } instance_type = "t2.micro" tags { Name = "app1" Group = "apps" } security_groups = ["aws_default"] ami = "ami-5189a661" key_name = "key" } resource "aws_instance" "app2" { connection { user = "ubuntu" } instance_type = "t2.micro" tags { Name = "app2" Group = "apps" } security_groups = ["aws_default"] ami = "ami-5189a661" key_name = "key" } resource "aws_instance" "app3" { connection { user = "ubuntu" } instance_type = "t2.micro" tags { Name = "app3" Group = "apps" } security_groups = ["aws_default"] ami = "ami-5189a661" key_name = "key" } resource "aws_instance" "app4" { connection { user = "ubuntu" } instance_type = "t2.micro" tags { Name = "app4" Group = "apps" } security_groups = ["aws_default"] ami = "ami-5189a661" key_name = "key" } # create db resource "aws_instance" "db1" { connection { user = "ubuntu" } instance_type = "t2.micro" tags { Name = "db1" Group = "dbs" } security_groups = ["aws_default"] ami = "ami-5189a661" key_name = "key" } # Create Consul resource "aws_instance" "consul" { connection { user = "ubuntu" } instance_type = "t2.micro" tags { Name = "consul" Group = "consul" } security_groups = ["aws_default"] ami = "ami-5189a661" key_name = "key" } ``` Before creating the servers, you should run `$ terraform plan` which will describe the execution plan of creating the instances and resources: ``` $ terraform plan Refreshing Terraform state prior to plan... The Terraform execution plan has been generated and is shown below. Resources are shown in alphabetical order for quick scanning. Green resources will be created (or destroyed and then created if an existing resource exists), yellow resources are being changed in-place, and red resources will be destroyed. Note: You didn't specify an "-out" parameter to save this plan, so when "apply" is called, Terraform can't guarantee this is what will execute. + aws_instance.app1 ami: "" => "ami-5189a661" availability_zone: "" => "" ebs_block_device.#: "" => "" ephemeral_block_device.#: "" => "" instance_type: "" => "t2.micro" key_name: "" => "key" placement_group: "" => "" private_dns: "" => "" private_ip: "" => "" public_dns: "" => "" public_ip: "" => "" root_block_device.#: "" => "" security_groups.#: "" => "1" security_groups.3578963758: "" => "aws_default" source_dest_check: "" => "1" subnet_id: "" => "" tags.#: "" => "2" tags.Group: "" => "apps" tags.Name: "" => "app1" tenancy: "" => "" vpc_security_group_ids.#: "" => "" + aws_instance.app2 ami: "" => "ami-5189a661" availability_zone: "" => "" ebs_block_device.#: "" => "" ephemeral_block_device.#: "" => "" instance_type: "" => "t2.micro" key_name: "" => "key" placement_group: "" => "" private_dns: "" => "" private_ip: "" => "" public_dns: "" => "" public_ip: "" => "" root_block_device.#: "" => "" security_groups.#: "" => "1" security_groups.3578963758: "" => "aws_default" source_dest_check: "" => "1" subnet_id: "" => "" tags.#: "" => "2" tags.Group: "" => "apps" tags.Name: "" => "app2" tenancy: "" => "" vpc_security_group_ids.#: "" => "" + aws_instance.app3 ami: "" => "ami-5189a661" availability_zone: "" => "" ebs_block_device.#: "" => "" ephemeral_block_device.#: "" => "" instance_type: "" => "t2.micro" key_name: "" => "key" placement_group: "" => "" private_dns: "" => "" private_ip: "" => "" public_dns: "" => "" public_ip: "" => "" root_block_device.#: "" => "" security_groups.#: "" => "1" security_groups.3578963758: "" => "aws_default" source_dest_check: "" => "1" subnet_id: "" => "" tags.#: "" => "2" tags.Group: "" => "apps" tags.Name: "" => "app3" tenancy: "" => "" vpc_security_group_ids.#: "" => "" + aws_instance.app4 ami: "" => "ami-5189a661" availability_zone: "" => "" ebs_block_device.#: "" => "" ephemeral_block_device.#: "" => "" instance_type: "" => "t2.micro" key_name: "" => "key" placement_group: "" => "" private_dns: "" => "" private_ip: "" => "" public_dns: "" => "" public_ip: "" => "" root_block_device.#: "" => "" security_groups.#: "" => "1" security_groups.3578963758: "" => "aws_default" source_dest_check: "" => "1" subnet_id: "" => "" tags.#: "" => "2" tags.Group: "" => "apps" tags.Name: "" => "app4" tenancy: "" => "" vpc_security_group_ids.#: "" => "" + aws_instance.consul ami: "" => "ami-5189a661" availability_zone: "" => "" ebs_block_device.#: "" => "" ephemeral_block_device.#: "" => "" instance_type: "" => "t2.micro" key_name: "" => "key" placement_group: "" => "" private_dns: "" => "" private_ip: "" => "" public_dns: "" => "" public_ip: "" => "" root_block_device.#: "" => "" security_groups.#: "" => "1" security_groups.3578963758: "" => "aws_default" source_dest_check: "" => "1" subnet_id: "" => "" tags.#: "" => "2" tags.Group: "" => "consul" tags.Name: "" => "consul" tenancy: "" => "" vpc_security_group_ids.#: "" => "" + aws_instance.db1 ami: "" => "ami-5189a661" availability_zone: "" => "" ebs_block_device.#: "" => "" ephemeral_block_device.#: "" => "" instance_type: "" => "t2.micro" key_name: "" => "key" placement_group: "" => "" private_dns: "" => "" private_ip: "" => "" public_dns: "" => "" public_ip: "" => "" root_block_device.#: "" => "" security_groups.#: "" => "1" security_groups.3578963758: "" => "aws_default" source_dest_check: "" => "1" subnet_id: "" => "" tags.#: "" => "2" tags.Group: "" => "dbs" tags.Name: "" => "db1" tenancy: "" => "" vpc_security_group_ids.#: "" => "" + aws_instance.lb ami: "" => "ami-5189a661" availability_zone: "" => "" ebs_block_device.#: "" => "" ephemeral_block_device.#: "" => "" instance_type: "" => "t2.micro" key_name: "" => "key" placement_group: "" => "" private_dns: "" => "" private_ip: "" => "" public_dns: "" => "" public_ip: "" => "" root_block_device.#: "" => "" security_groups.#: "" => "1" security_groups.3578963758: "" => "aws_default" source_dest_check: "" => "1" subnet_id: "" => "" tags.#: "" => "2" tags.Group: "" => "lbs" tags.Name: "" => "lb1" tenancy: "" => "" vpc_security_group_ids.#: "" => "" + aws_key_pair.auth fingerprint: "" => "" key_name: "" => "key" public_key: "" => "ssh-rsa xxxxxxxxx" + aws_security_group.default description: "" => "Default security group" egress.#: "" => "1" egress.482069346.cidr_blocks.#: "" => "1" egress.482069346.cidr_blocks.0: "" => "0.0.0.0/0" egress.482069346.from_port: "" => "0" egress.482069346.protocol: "" => "-1" egress.482069346.security_groups.#: "" => "0" egress.482069346.self: "" => "0" egress.482069346.to_port: "" => "0" ingress.#: "" => "1" ingress.1403647648.cidr_blocks.#: "" => "1" ingress.1403647648.cidr_blocks.0: "" => "0.0.0.0/0" ingress.1403647648.from_port: "" => "0" ingress.1403647648.protocol: "" => "tcp" ingress.1403647648.security_groups.#: "" => "0" ingress.1403647648.self: "" => "0" ingress.1403647648.to_port: "" => "65535" name: "" => "aws_default" owner_id: "" => "" vpc_id: "" => "" Plan: 9 to add, 0 to change, 0 to destroy. ``` ## Ansible + Terraform Ansible can be used with Terraform by providing a dynamic inventory (terraform.py) which will connect to systems by reading terraform's .tfstate file. It currently supports AWS and you can find the dynamic inventory script [here](https://github.com/CiscoCloud/terraform.py). The dynamic script can be used to group each set of servers in its own group according to the tags we specified earlier with each server. ![Screen Shot 2016-01-11 at 7.04.56 AM](https://cdn.levvel.io/blog_content/Screen-Shot-2016-01-11-at-7.04.56-AM.png) Before running `$ terraform apply`, we should define the Ansible roles that will provision each component on the servers. Each role in the Ansible playbook will describe how to get the components up and running and the Ansible playbook will look like this: ```yaml --- - hosts: all sudo: yes roles: - { role: common, tags: ["common"] } - { role: docker, tags: ["docker"] } - hosts: "aws_tag_Group=consul" sudo: yes roles: - { role: consul, tags: ["consul"] } - hosts: "aws_tag_Group=dbs" sudo: yes roles: - { role: db-run, tags: ["db-run"] } - hosts: "aws_tag_Group=apps" sudo: yes roles: - { role: app-run, tags: ["app-run"] } - hosts: "aws_tag_Group=lbs" sudo: yes roles: - { role: lb-run, tags: ["lb-run"] } ``` The **common** and **docker** role will be installed on all the servers, and each group of servers will have its own role, i.e. consul group will contain consul role and so on. ## Consul Role This role will simply run the [gliderlabs/docker-consul](https://github.com/gliderlabs/docker-consul) Docker image and it will expose the corresponding ports of the Consul server. It's not required to run a cluster of Consul servers, but it's recommended in a production environment to provide HA systems: ```yaml - name: Pull and run Docker consul docker: name: consul-{{ ansible_hostname }} image: progrium/consul restart_policy: always state: started command: -server -bootstrap -ui-dir /ui ports: - 8500:8500 - 8400:8400 - 8600:53/udp ``` This image provides a web UI interface which will display the services and nodes registered with Consul server. ## Lb-run Role This role will essentially do three main tasks: - run the registrator container. - run the Nginx container. - run consul-template tool. The Registrator task will be common between the roles (lb-run, db-run, and app-run): ```yaml - name: Pull and run registrator container docker: name: registrator-{{ ansible_hostname }} net: host image: gliderlabs/registrator:latest volumes: - /var/run/docker.sock:/tmp/docker.sock command: -ip {{ public.ip }} consul://{{ consul_url }} ``` Note that the registrator image uses consul_url variable to connect to consul server, and uses public.ip to tell the consul server the public ip of the registered service. The second task will run the create the configuration directory for Nginx, and will pull the nginx configuration and the ctmpl file which is the template that will be used by consul-template: ``` - name: Create configuration path file: path={{ nginx_docker_path }} state=directory mode=750 owner=root group=root - name: Download Nginx configuration for the apps git: repo={{ nginx_config_repo }} dest={{ nginx_docker_path }} clone=yes ignore_errors: yes - name: Pull and run Nginx Docker container docker: name: nginx-{{ ansible_hostname }} image: nginx:latest restart_policy: always ports: - 80:80 volumes: - "{{ nginx_docker_path }}/nginx.conf:/etc/nginx/nginx.conf" ``` An important note here is to mount the configuration file in an nginx container to the configuration file on the host, this will ensure that the consul-template will change the configuration file regularly whenever any information changed regarding the services. The configuration template will like the following: ``` worker_processes 1; pid /run/nginx.pid; events { worker_connections 1024; } http { sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; server_tokens off; include /etc/nginx/mime.types; default_type application/octet-stream; access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; upstream apps{ {{range service "ghost"}} server {{.Address}}:{{.Port}};{{end}} } server { listen 80 default_server; server_name ghost.me.com; location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://apps; } } } ``` Note that this creates an automatic reverse proxy by using this template with consul-template tool, and restarts Nginx every time the upstream updated. The last task is to run consul-template itself: ``` - name: Download consul-template get_url: url=https://releases.hashicorp.com/consul-template/0.12.0/consul-template_0.12.0_linux_amd64.zip dest=/opt/consul-template.zip - name: Unarchive consul template unarchive: src=/opt/consul-template.zip dest=/usr/local/bin/ copy=no mode=u+rwx - name: Run consul template shell: creates=/run/consul-template.pid consul-template -pid-file /run/consul-template.pid -consul {{ consul_url }} -retry "5s" -template "{{ nginx_docker_path }}/nginx.ctmpl: {{ nginx_docker_path }}/nginx.conf:docker restart nginx-{{ansible_hostname }}" & ``` This will use nginx.ctmpl file described earlier to update nginx.conf file which is in turn mounted inside the Docker container. ## App-run Role This role will run registrator as well, and it will run the ghost application and consul-template. The ghost task will look like the following: ``` - name: Create configuration path file: path={{ ghost_docker_path }} state=directory mode=750 owner=root group=root - name: Download Ghost configuration git: repo={{ ghost_config_repo }} dest={{ ghost_docker_path }} clone=yes ignore_errors: yes - name: Build Ghost image docker_image: name=denmanjd/ghost path={{ ghost_docker_path }}/ state=build - name: Pull and run Ghost Docker container docker: name: ghost-{{ ansible_hostname }} image: denmanjd/ghost restart_policy: always volumes: - "{{ ghost_docker_path }}/config.js:/var/www/ghost/config.js" ports: - 2368:2368 env: CONSUL_URL={{ consul_url }} ``` This creates a directory to store the configuration for the ghost application and will build the docker image for the ghost application. Then it will mount the config.js file inside the application. Finally the role will run consul template which will process the following template: ```javascript var path = require('path'), config; config = { // ### Production // When running Ghost in the wild, use the production environment. // Configure your URL and mail settings here production: { url: 'http://denman.ghost.com', mail: {}, database: { client: 'mysql', connection: { {{range service "mysql"}} host : '{{.Address}}',{{end}} user : 'ghost_user', password : '{{key "mysql/ghost_password"}}', database : 'ghost_db', charset : 'utf8' }, debug: false }, server: { host: '0.0.0.0', port: '2368' } } }; module.exports = config; ``` Beside using the .Address, it will also fetch the MySQL password from consul, which will be stored later using mysql role. ## Db-run Role This role will run registrator and then a mysql role: ``` - name: Pull and run MySQL Docker container docker: name: mysql image: mysql:5.5 restart_policy: always env: MYSQL_DATABASE=ghost_db, MYSQL_USER=ghost_user, MYSQL_PASSWORD={{ mysql_ghost_password }}, MYSQL_ROOT_PASSWORD={{ mysql_root_password }} ports: - 3306:3306 volumes: - /var/lib/mysql:/var/lib/mysql - name: Store mysql password into consul command: curl -XPUT -d "{{ mysql_ghost_password }}" http:// {{ consul_url }}/v1/kv/mysql/ghost_password ``` The mysql ghost password and mysql root password will be generated randomly and will store the ghost password into key/value store. ## Go Time Now its time to run `$ terraform apply` and then run Ansible playbook on the created servers: ``` $ terraform apply $ ansible-playbook -i terraform.py -s -u ubuntu playbook.yml ``` You can make sure that everything went OK, by checking the consul server using the following url: `http://consul-ip:8500/` ![Screen Shot 2016-01-11 at 7.18.51 AM](https://cdn.levvel.io/blog_content/Screen-Shot-2016-01-11-at-7.18.51-AM.png) To make sure that nginx is aware of the 4 ghost application servers created, we can check the nginx configuration which should've been updated with the server's ips: ``` root@lb1:~# cat /opt/nginx-ghost/nginx.conf upstream apps{ server 52.35.119.235:2368; server 52.35.31.204:2368; server 52.32.104.9:2368; server 52.24.29.58:2368; } ``` Finally access the application using the load balancer's ip. We can also configure Terraform to adjust the DNS records using AWS Route53 record resource. Here at Levvel, we're excited about creating unique solutions, leveraging existing toolsets and helping our clients revolutionize the way they develop applications, manage complex infrastructure and drive business needs through the IT organization. If you’d like to talk about how to use such tools in your team, please reach out to me at james.denman@levvel.io or at hello@levvel.io for more general questions. ## About Levvel Levvel is an IT consulting firm that combines the innovative DNA of a start up with the wisdom, scalability, and process rigor of a Fortune 100 company. We offer both technical implementation services as well as strategic advisory services. Levvel offers you an “unfair advantage” over your competition with comprehensive services including DevOps, Cloud, Mobile, UI/UX, Big Data, Analytics, Payment Strategy, and more.

Docker + Ansible + Consul + Terraform

Docker + Ansible + Consul + Terraform

Read the Blog