Blog Projects
Programming►Helm108 Server

Provisioning Images on Vagrant and Digital Ocean With Puppet

Provisioning Helm108

At this point in building my server for Helm108 I had a Vagrantfile running Ubuntu 16.04. My next step was to learn how to provision my server, and then get that provisioned image on to Digital Ocean. I initially followed these instructions for deploying to Digital Ocean from Vagrant but that didn’t seem like the right solution for a production deployment. At work we use Hashicorp’s suite of tools and as they’re all open source (and I could bother the devops team if I needed help) I decided to do the same.

After talking to said devops team I was recommended Puppet for my provisioning needs. Having no experience with this kind of thing that seemed reasonable so I went with it. I could have done more research and tried to judge the best tool for the job but with anything I do where I have no experience I typically just pick one and try that. I might hate my choice, but at least I now have better information with which to judge alternatives.

Provisioning in Vagrant

You can define provisioners in your Vagrantfile, and Vagrant will run them on vagrant up and vagrant provision. Pretty simple.

Provisioning on Digital Ocean

Getting a provisioned image running on Digital Ocean using Hashicorp’s suite of tools involves two steps:

  1. You use Packer to bring up a standard Digital Ocean box using the image you specify. Packer then runs the provisioners that you have specified, saves a snapshot of the VM, and then shuts down and destroys the VM. You can specify one of Digital Ocean’s existing base images, or you can run Packer against one of your own existing snapshots.
  2. You then use Terraform to take your snapshot, turn it into an actual running VM and point a domain at it.

In this post I’m just going to focus on step one. I’ll talk about the Terraform step, and all the other things that Terraform does, in a later post.

Making the box work with Puppet

Packer and Vagrant are the entry points to my server in their respective environments. They both support Puppet, so all I need to do is write a Puppet manifest and point both Vagrant and Packer at it. This is nice, as it means that I can write code once, test it locally, and when I’m happy with it I can deploy it to Digital Ocean just by running a different command.

Vagrant and Packer run Puppet manifests on the VMs that they control by copying the manifest files to the VM and running Puppet. What they do not do is install Puppet for you, so you need to run a shell provisioner first that installs it. Here is my shell provisioner that I run first:

provision-base.sh
1
2
3
4
5
6
7
#!/bin/sh -x
export DEBIAN_FRONTEND=noninteractive
sudo locale-gen en_GB.UTF-8
cd ~ && wget https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb
dpkg -i puppetlabs-release-pc1-trusty.deb
apt-get update
apt-get install -y puppet-agent

export DEBIAN_FRONTEND=noninteractive allows you to perform unattended installs. As I understand it the subsequent commands will just assume you’re saying yes to everything.
sudo locale-gen en_GB.UTF-8 was just to get rid of a warning saying my locale wasn’t set.
The remaining four lines are how you install Puppet 4, as per the documentation.

Specifying Provisioners in Vagrant

Nothing really special here, this is straight from the Vagrant documentation.

Vagrantfile
1
2
3
config.vm.define "dev", primary:true do |dev|
  config.vm.provision "shell", path: "provision-base.sh"
end

Specifying Provisioners in Packer

Nothing special here either.

packer.json
1
2
3
4
5
6
7
8
9
{
  "provisioners": [
    {
      "type": "shell",
      "script": "provision-base.sh",
      "pause_before": "5s"
    }
  ],
}

Adding Puppet to the mix

With that done, my server can now run Puppet manifests. I ended up splitting my Puppet work into two separate stages, one for building the server, setting up the user and installing the packages I want on it, the other for installing and running my projects on it. This means that I can create a base snapshot with my dependencies on it on Digital Ocean, and then when I want to add a new project to my server I can just run the second project provisioner without having to rebuild the entire server. To facilitate this I have two separate packer configs, but Vagrant runs both provisioners as I generally want to go through the entire process fresh when testing locally.

My folder structure looks like this:

1
2
3
4
5
6
7
8
9
10
puppet-env/
  base/
    manifests/
      site.pp
  deploy/
    manifests/
      site.pp
Vagrantfile
packer-base.json
packer-deploy.json

Here’s what my final Vagrantfile looks like regarding provisioning:

Vagrantfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
config.vm.define "dev", primary:true do |dev|
  config.vm.provision "shell", path: "provision-base.sh"

  config.vm.provision "puppet" do |puppet|
    puppet.environment_path = "puppet-env"
    puppet.environment = "base"
    puppet.module_path = "modules"
  end

  config.vm.provision "puppet" do |puppet|
    puppet.environment_path = "puppet-env"
    puppet.environment = "deploy"
    puppet.module_path = "modules"
  end
end

You can see how the various Puppet config options map to the folder structure. By default Puppet looks for a site.pp in the manifests directory under the environment_path path, so that’s where I put everything.

As mentioned above I have two separate Packer configs:

packer-base.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "provisioners": [
    {
      "type": "shell",
      "script": "provision-base.sh",
      "pause_before": "5s"
    },
    {
      "type": "puppet-masterless",
      "manifest_file": "puppet-env/base/manifests/site.pp",
      "module_paths": "modules/",
      "puppet_bin_dir": "/opt/puppetlabs/bin/"
    }
  ],
}
packer-deploy.json
1
2
3
4
5
6
7
8
9
10
{
  "provisioners": [
    {
      "type": "puppet-masterless",
      "manifest_file": "puppet-env/deploy/manifests/site.pp",
      "module_paths": "modules/",
      "puppet_bin_dir": "/opt/puppetlabs/bin/"
    }
  ]
}

There’s more in those json files than the provisioners, but the rest isn’t relevant to this post. packer-deploy.json does not reference the shell script because packer-base.json has already installed Puppet. Also note that both Packer configs specify where the Puppet bin directory is. /opt/puppetlabs/bin is the default location for Puppet 4.