Consul - Launch, Backup and Restore a cluster in AWS
This is a tutorial on how to launch, backup and restore a fully working Consul cluster in AWS.
The CloudFormation code is posted on on GitHub and contains all you need to have the stack up and running. Simply clone the repository and use the cloudformation-template.yaml file to launch your stack.
What is Consul?
Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality. Each of these features can be used individually as needed, or they can be used together to build a full service mesh. Consul requires a data plane and supports both a proxy and native integration model. Consul ships with a simple built-in proxy so that everything works out of the box, but also supports 3rd party proxy integrations such as Envoy.
The key features of Consul are:
- Service Discovery: Clients of Consul can register a service, such as api or mysql, and other clients can use Consul to discover providers of a given service. Using either DNS or HTTP, applications can easily find the services they depend upon.
- Health Checking: Consul clients can provide any number of health checks, either associated with a given service ("is the webserver returning 200 OK"), or with the local node ("is memory utilization below 90%"). This information can be used by an operator to monitor cluster health, and it is used by the service discovery components to route traffic away from unhealthy hosts.
- KV Store: Applications can make use of Consul's hierarchical key/value store for any number of purposes, including dynamic configuration, feature flagging, coordination, leader election, and more. The simple HTTP API makes it easy to use.
- Secure Service Communication: Consul can generate and distribute TLS certificates for services to establish mutual TLS connections. Intentions can be used to define which services are allowed to communicate. Service segmentation can be easily managed with intentions that can be changed in real time instead of using complex network topologies and static firewall rules.
- Multi Datacenter: Consul supports multiple datacenters out of the box. This means users of Consul do not have to worry about building additional layers of abstraction to grow to multiple regions.
Consul is designed to be friendly to both the DevOps community and application developers, making it perfect for modern, elastic infrastructures.
Deploy the stack
Run the CloudFormation template in the AWS Console
- Login to the AWS console and browse to the CloudFormation section
- Select the cloudformation-template.yaml file
- Before clicking "Create", make sure that you scroll down and tick the “I acknowledge that AWS CloudFormation might create IAM resources” checkbox
- ...drink coffee...
- Go to the URL in the output section for the environment that you want to access
Main resources created
- 1 AutoScaling Group
- 1 Elastic Load Balancer
- 1 S3 bucket (for data backup)
- 1 DNS record (for ease of access)
Autoscaling
The autoscaling groups uses the CpuUtilization alarm to autoscale automatically. Because of this, you wouldn't have to bother making sure that your hosts can sustain the load.
Monitoring
The stack launches NodeExporter (Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors) on each host inside the cluster.
The stack also includes a custom StatsdExporter which is used to send Consul-specific metrics into Prometheus.
Data persistency
Due to the mechanics Consul, you don't need to set up any sort of data persistency, as the application itself is using the RAFT protocol to send data to-from all its clients and servers, hence maintaining a continuous state of up-to-date.
Backup and Restore
A cronjob has been set up to run every day on the ASG hosts that dump the data in an S3 bucket that is created inside the template.
There's also a restore script to help you refresh keys and ACLS in case you need to as well as a script to run the backup manually.
Security concerns
If you start using Consul, you'll notice that as the time goes, it will become more and more an indispensable tool in your day-to-day life. Because of this, we strongly advise you to:
- Use an INTERNAL load balancer (set Exposure parameter = internal)
- Use subnets WITHOUT public IP addresses
- NOT share your master token with anyone
- ALWAYS create tokens and policies that limit as much as possible the access
Final notes
Need help implementing this?
Feel free to contact us using this form.