Easy deployment of custom debian packages with Puppet
The context
In modern computer science, applications often need to withstand large amount of traffic or process a lot of data. The way to achieve required performance is distributed computing, which employs many machines to carry out tasks in parallel. Large number of nodes means that there could be problems with keeping the production environment consistent on all of them.
The problem
We would like to be able to deploy a custom created debian package across multiple machines. This could be achieved by using scp
to transfer a .deb
file to every node and then running dpkg
to install the package. Such manual installation could work well for a few machines, but as the number of nodes grows, an automatic solution is a must.
The solution
Deployment of custom packages can be achieved by creating a debian repository. Client would connect to this repository and install the latest version of given package:
$ sudo apt-get update
$ sudo apt-get install custom-package
Having in mind, that our package is going to be deployed to multiple machines, we would like to automate that process. Our solution will employ:
- Puppet - installing components and managing configuration
- Aptly - publishing and updating debian repository
- Apache HTTPD - exposing Aptly repository to clients
Server side
The apt_repository::server
class provides a Puppet recipy to host a debian repository on a given machine. This has been achieved in a few steps:
Installing Aptly
Atply can be easily installed using module from puppetforge:
class { 'aptly':
config => {
rootDir => $aptly_rootdir,
}
}
This step is fairly straightforward, apart from one gotcha. Aptly is not available in default debian repositories of the operating system, so we need to add Aptly repository to /etc/apt/sources.list
. Luckily, the Aptly module does this job, but does not trigger apt-get update
. We ensure such update by adding a Puppet command:
Class['apt::update'] -> Package<| |>
Publishing repository
To publish a repository, we need to call:
vagrant@server:~$ sudo /usr/bin/aptly -architectures=amd64 -skip-signing=true publish repo pracuj-debs
Sadly, there is a problem. Even though the first command call works flawlessly:
...
Local repo pracuj-debs has been successfully published.
...
The following call of the same command fails:
...
ERROR: prefix/distribution already used by another published repo: ./precise [amd64] publishes {main: [pracuj-debs]}
To prevent such error, a simple shell script, publish_repo.sh
has been provided in the module. It checks if given repository exists, and executes publish
command if it does not. Execution of publish_repo.sh
is idempotent, just like any command executed by Puppet should be.
The server module also provides a command to update Aptly repository with packages included in directory specified by debian_package_dir
parameter:
vagrant@server:~$ sudo /usr/bin/update-apt-repository
Serving repository
Another puppetforge module, apache, is used to create a webservice to serve the contents of the aplty repository:
class { 'apache': }
apache::vhost { 'bigdata.aptrepo':
port => $server_port,
docroot => "${aptly_rootdir}/public",
require => Package['aptly'],
}
Client side
Thanks to the apt module, the apt_repository::client
class is quite simple. All that needs to be done is adding the server host to /etc/hosts
file and list the Aptly server as debian source:
host { 'apt-repository-server':
ip => $server_address,
}
apt::source { 'atplyrepo':
location => "http://apt-repository-server:${server_port}/",
architecture => $repo_architecture,
release => $repo_release,
repos => 'main',
allow_unsigned => true,
require => Host['apt-repository-server'],
}
The signing of the repositories was skipped when Aptly repository was created (using the -skip-signing=true
flag), so for the client the allow_unsigned => true
option has been set. This is fine as long as the Aptly repository will not be made public.
Configuration
Whole apt_repository
module is configured with Hiera. Single set of parameters configures both client and server, which guarantees that they will work well together. Parameters are placed in .yaml
files, that assign configuration to given nodes in hierarchical fashion.
Demo Time!
To follow the example presented below, Vagrant 1.7.4+ and SSH client are needed. Source code can be found on Github
Let’s check if the apt_repository
module works as expected! Two simple debian packages have been created. pracuj-example_1.0-1_all.deb
is placed in files/debian_package_dir
(and hiera parameter debian_package_dir
has been set to that directory). Other file, pracuj-example_1.2-1_all.deb
has been placed outside of the package directory. Both those packages install simple bash command that prints version of package on console.
Vagrant needs to know, how to configure each node. For server we would like to install apt-repository::server
and trigger update-apt-repository
afterwards:
node 'server' {
...
class { 'apt_repository::server': }
exec { '/usr/bin/update-apt-repository':
require => Class['apt_repository::server'],
}
}
So, let’s run the server:
> vagrant up server
After a while, the server should be up and running. Client configuration is also simple.
node 'client' {
...
class { 'apt_repository::client': }
package {
'pracuj-example':
ensure => latest,
require => Class['apt_repository::client'],
}
}
After running vagrant up client
the latest version of the pracuj-example
package should be installed:
> vagrant ssh client
vagrant@client:~$ pracuj-example
This is version 1.0-1
vagrant@client:~$ logout
Now, let’s add a newer version of the pracuj-example
package to the Aptly repository:
> vagrant ssh server
vagrant@server:~$ cd /vagrant/files/
vagrant@server:/vagrant/files$ mv pracuj-example_1.2-1_all.deb debian_package_dir/
vagrant@server:/vagrant/files$ sudo update-apt-repository
...
Publish for local repo ./precise [amd64] publishes {main: [pracuj-debs]} has been successfully updated.
vagrant@server:/vagrant/files$ logout
Let’s simulate refresh of Puppet configuration on client. Such task can be achieved by running vagrant provision
:
> vagrant provision client
> vagrant ssh client
vagrant@client:~$ pracuj-example
This is version 1.2-1
vagrant@client:~$ logout
Hey! It works :)
Now only thing left is to cleanup:
> vagrant destroy -f
Wrap-up
We have demonstrated how to deploy a custom package in small, virtual two machine environment. Thanks to Puppet, deploying such package on hundreds of nodes is as easy. Presented solution could be used to propagate debian packages generated using continuous integration tools e.g. Jenkins and provide a consistent production environment across many nodes.