Ensemble¶
This is really the sugar on the doughnut. headintheclouds.ensemble is an orchestration tool for Docker that will manage dependencies between containers, intelligently figure out what needs to change to meet the desired configuration, start servers and containers, and manage firewalls. It uses a simple YAML-based config format, and it’s doing as much as possible in parallel.
I built ensemble on top of headintheclouds to manage thisismyjam.com. We’re using it now for our production setup and it seems to hang together so far. The configuration format is heavily influenced by Orchard’s Fig.
Tasks¶
- headintheclouds.ensemble.up¶
Create servers and containers as required to meet the configuration specified in _name_.
- Args:
- name: The name of the yaml config file (you can omit the .yml extension for convenience)
- Example:
- fab ensemble.up:wordpress
Configuration YAML schema¶
# The name of the server. If count > 1, the names will be
# SERVER_NAME, SERVER_NAME-1, SERVER_NAME-2, [...].
# If SERVER_NAME is an IP address, it is implied that it is
# "unmanaged".
SERVER_NAME:
# An optional template, see below.
template: TEMPLATE
# Provider is required unless SERVER_NAME is an IP address.
# Valid options are currently ec2 and digitalocean
provider: PROVIDER
# Optional. The number of copies of this server will
# be created. Default=1.
count: COUNT
# Provider-specific settings, see the section on
# provider-specific create flags in the Tasks section
# Examples for an EC2 instance:
size: m1.small
image: ubuntu 14.04
security_group: web_ssh
# The containers to run
containers:
# The name of the container. Again, if container count > 1,
# names will be suffixed with '', '-1', '-2', etc.
CONTAINER_NAME:
# Required. E.g. orchardup/redis.
image: IMAGE
# Optional. A list of ports to open in the format
# CONTAINER_PORT[:EXPOSED_PORT][/PROTOCOL]
ports:
# Examples:
- 80
- 3306:3366
- 1234/tcp
- 1234:2345/udp
# Optional hash of environment variables to pass to
# docker run
environment:
# Optional template for env vars
template: TEMPLATE
# Examples:
FOO: BAR
hello: 123
# Optional hash of volumes to bound mount in the
# format HOST_DIR:CONTAINER_DIR
volumes:
# Examples:
/docker-vol/web/tmp: /tmp
/data/logs: /var/log
# The number of instances of this container to run.
# Default=1
count: COUNT
# Optional firewall configuration. If defined, only the
# ports specified here will be open, all others will be
# closed.
firewall:
# firewall also accepts an optional template
template: TEMPLATE
# The open ports are defined as a hash of PORT[/PROTOCOL]
# to IP or list of IPs or "*" or $internal_ips, e.g.:
3306: 10.1.1.12
8125/udp: 10.1.1.15
# "*" opens a port to the world
22: "*"
# $internal_ips is a special variable (see Variables and
# dependencies below) that will expand to a list of all
# internal IPs for the servers in the same configuration
# file, effectively opening a port to all of them.
6379: $internal_ips
# Ports can also be wildcarded, like this
"*/*": $internal_ips
templates:
TEMPLATE_NAME:
# anything goes here
Templates¶
To avoid having to write the same chunk of YAML over and over again, templates can be used as a sort of preprocessor macro. Anything that is defined in the main configuration will override the value in the template. For example, if you have a config that looks like this
myserver:
template: foo
containers:
template: bar
yourserver:
template: foo
containers:
template: bar
image: hello/world:other
environment:
template: baz
templates:
foo:
provider: digitalocean
size: 1GB
bar:
image: hello/world
ports:
- 80:9000
environment:
HELLO: 123
baz:
WORLD: 456
it will expand to
myserver:
provider: digitalocean
size: 1GB
containers:
image: hello/world
ports:
- 80:9000
environment:
HELLO: 123
yourserver:
provider: digitalocean
size: 1GB
containers:
image: hello/world:other
ports:
- 80:9000
environment:
WORLD: 456
Variables and dependencies¶
Often you want to connect containers and servers, but you probably don’t know the address of the server or container in advance. Enter variables and dependency management!
Here’s an example:
web:
provider: ec2
containers:
web:
image: hello/web
ports:
- 80
environment:
REDIS_HOST: ${redis.ip}
redis:
provider: ec2
containers:
redis:
image: orchardup/redis
ports:
- 6379
When you “up” this ensemble manifest from a vanilla setup with no running servers, the order of operations will be:
- Start “web” and “redis” servers in parallel
- Resolve ${redis.ip} to the actual IP of the redis server
- Start the redis and web containers in parallel
If the web container would need to wait for the redis container to start, you could put in an environment variable like
# [snip]
web:
environment:
REDIS_HOST: ${redis.ip}
_DEPENDS: ${redis.containers.redis.ip}
headintheclouds.ensemble abstracts all the scheduling and will complain if you try to set up cyclical dependencies, so you can set up pretty complex dependency graphs without thinking too much about what’s going on behind the scenes.
As a side note, headintheclouds doesn’t use docker links, instead you point containers to the IPs of other servers and containers.
Idempotence and statelessness¶
The only state that headintheclouds keeps is the internal caches, and these can be wiped without any negative side effects. Instead of storing state locally, the state of servers and containers is interrogated on the fly by logging in to the servers and checking what is actually running.
When you run fab ensemble.up:myensemble, it will log in to any existing servers with the same names as in the manifest, and check if they’re equivalent to what the configuration says. Then it will check the Docker containers and firewall rules on each host to see if they match the manifest.
This is how headintheclouds.ensemble is idempotent. You can run fab ensemble.up:myensemble any number of times with no effect on your servers, provided you don’t change myensemble.yml.
Before going out starting servers and containers, headintheclouds will prompt you to confirm the changes that will be made.
The only caveat is that headintheclouds doesn’t currently delete servers and containers if you remove them from the manifest, you have to do that manually with the terminate and docker.kill commands. That’s just so you don’t go and tear things don’t by accident.
Server names and roles¶
If you have a conf YAML file like this
foo:
provider: ec2
[snip]
bar:
[snip]
containers:
blah:
image: some/image
environment:
FOO_NAME: ${foo.name}
count: 2
foo and bar are the names of the servers. But when using Fabric, role is synonymous with name. So you could do
fab -R bar ping
to ping both of the bar server. To access a single one, you’d have to use the -H Fabric flag, e.g.
fab -H 123.45.67.89 ssh
(assuming 123.45.67.89 is the IP of one of the bar servers).