In modern computer science, applications often need to withstand large amount of traffic or process a lot of data. The way to achieve required performance is distributed computing, which employs many machines to carry out tasks in parallel. Large number of nodes means that there could be problems with keeping the production environment consistent on all of them.
We would like to be able to deploy a custom created debian package across multiple machines. This could be achieved by using
scp to transfer a
.deb file to every node and then running
dpkg to install the package. Such manual installation could work well for a few machines, but as the number of nodes grows, an automatic solution is a must.
Deployment of custom packages can be achieved by creating a debian repository. Client would connect to this repository and install the latest version of given package:
Having in mind, that our package is going to be deployed to multiple machines, we would like to automate that process. Our solution will employ:
- Puppet - installing components and managing configuration
- Aptly - publishing and updating debian repository
- Apache HTTPD - exposing Aptly repository to clients
apt_repository::server class provides a Puppet recipy to host a debian repository on a given machine. This has been achieved in a few steps:
Atply can be easily installed using module from puppetforge:
This step is fairly straightforward, apart from one gotcha. Aptly is not available in default debian repositories of the operating system, so we need to add Aptly repository to
/etc/apt/sources.list. Luckily, the Aptly module does this job, but does not trigger
apt-get update. We ensure such update by adding a Puppet command:
To publish a repository, we need to call:
Sadly, there is a problem. Even though the first command call works flawlessly:
The following call of the same command fails:
To prevent such error, a simple shell script,
publish_repo.sh has been provided in the module. It checks if given repository exists, and executes
publish command if it does not. Execution of
publish_repo.sh is idempotent, just like any command executed by Puppet should be.
The server module also provides a command to update Aptly repository with packages included in directory specified by
Another puppetforge module, apache, is used to create a webservice to serve the contents of the aplty repository:
Thanks to the apt module, the
apt_repository::client class is quite simple. All that needs to be done is adding the server host to
/etc/hosts file and list the Aptly server as debian source:
The signing of the repositories was skipped when Aptly repository was created (using the
-skip-signing=true flag), so for the client the
allow_unsigned => true option has been set. This is fine as long as the Aptly repository will not be made public.
apt_repository module is configured with Hiera. Single set of parameters configures both client and server, which guarantees that they will work well together. Parameters are placed in
.yaml files, that assign configuration to given nodes in hierarchical fashion.
To follow the example presented below, Vagrant 1.7.4+ and SSH client are needed. Source code can be found on Github
Let’s check if the
apt_repository module works as expected! Two simple debian packages have been created.
pracuj-example_1.0-1_all.deb is placed in
files/debian_package_dir (and hiera parameter
debian_package_dir has been set to that directory). Other file,
pracuj-example_1.2-1_all.deb has been placed outside of the package directory. Both those packages install simple bash command that prints version of package on console.
Vagrant needs to know, how to configure each node. For server we would like to install
apt-repository::server and trigger
So, let’s run the server:
After a while, the server should be up and running. Client configuration is also simple.
vagrant up client the latest version of the
pracuj-example package should be installed:
Now, let’s add a newer version of the
pracuj-example package to the Aptly repository:
Let’s simulate refresh of Puppet configuration on client. Such task can be achieved by running
Hey! It works :)
Now only thing left is to cleanup:
We have demonstrated how to deploy a custom package in small, virtual two machine environment. Thanks to Puppet, deploying such package on hundreds of nodes is as easy. Presented solution could be used to propagate debian packages generated using continuous integration tools e.g. Jenkins and provide a consistent production environment across many nodes.