Automation and Orchestration
Continuous Integration (CI)
Continuous Integration tools can be used to automate the compilation of code into binaries, or other artifacts. Of course these tools are not limited to just compiling code. They can also be used to provision servers and install solutions, etc. The structure of these tools typically follows a master and agent architecture whereby the agents can be instructed to execute any set of instructions delivered to it by the master. Jenkins, Hudson, CruiseControl, and TeamCity are examples of some of the more popular on-site continuous integration tools in use today.
There are some more cutting-edge continuous integration tools becoming available online. Travis CI and Springloops.io are cloud-based CI tools that encourage you to write code in a remote repository and use their tool to compile and deploy it to your choice of platform.
Configuration Management (CM)
Of late there is a heightened level of interest in Configuration Management tools. Many organizations are now beginning to understand the time-savings and consistency afforded by this category of products. Solutions such as Chef, Puppet, Ansible, and Salt were designed to provision virtual/cloud servers and then installing/configuring software and services atop them. These tools provide an indempotent environment.
Before these tools existed most organizations used continuous integration tools (CruiseControl, Jenkins) to execute scripts custom made for each infrastructure, platform and solution combination necessary. This is not only tedious to create, it is even harder to maintain.
Many organizations have built and continue to maintain these custom scripts for the sake of protecting investment. However, more teams are discovering the power of contemporary configuration management tools like Chef, Puppet and Ansible.
Chef is a tool that uses ‘recipes’ and ‘cookbooks’ along with some help from another tool called ‘knife’. Recipes are written in Ruby and contain all the directives needed to configure a node (virtual/cloud/physical server). Nodes are identified and targeted, either statically or dynamically (can be accomplished using knife) and then cookbooks/recipes are delivered to configure them. Variable information can be extracted from the dispatching Chef server or the system being configured and then used during recipe execution. Chef can be used in ‘solo’ mode where no server is needed, or in enterprise mode where a chef server is required. Finally, Chef defaults to synchronous execution of recipes defined by a run-list, and all nodes to be configured require a pre-existing Chef agent.
[snippet id=”42″]
Puppet is a somewhat similar to Chef in that an agent must be available on the targeted node before it can be configured. Puppet uses modules and manifests instead of recipes and cookbooks. The DSL (domain specific language) for Puppet has evolved into a very straightforward, very easy-to-use set of instructions. ‘File’, ‘Package’, ‘Service’ is the basic template that can be used to configure a node. ‘File’ is the directive used to identify the location of an artifact that may be used later, or may need to be absent before continuing execution. ‘Package’ installs the contents of the ‘File’. ‘Service’ insures that certain processes are running, not running, or in some other state of operation.
There are many attributes that can be used to fine tune Puppet instructions, however the good thing is that you can get started quickly and keep your configuration scripts readable and easy to maintain.
Puppet defaults to asynchronous execution of instruction which is something you should know up-front. Without this knowledge you will likely fail to build a working script if you are coming from a Chef environment where instructions execute synchronously. However, puppet does provide dependency checking, ‘operation chaining’ and ‘classes’ to build more order into a script. Asynchronous execution is a great differentiator but must be handled with care.
[snippet id=”43″]
Ansible is very much like Chef and Puppet however it differs in one very important aspect; no agent is necessary on the target node. Ansible uses ssh to execute commands (stored in ‘playbooks’ in YAML format) remotely on the target node. This difference make Ansible simple in nature.
[snippet id=”41″]
IaaS, PaaS, and SaaS
In the past we installed software on computers. Now we refer to computers as infrastructure. As we have evolved, infrastructure has migrated into the cloud. We now see more virtualization than physical computers in our organizations. Cloud Computing vendors like Amazon , Rackspace and others have built large clouds of infrastructure. We call the service these vendors offer Infrastructure as a service (IaaS).
Naturally we want to install software on these systems and use it like we would on a server in a datacenter. The ability to share an application amongst a great number of users without interfering with each other is called Software as a service (SaaS). Multi-tenancy and security (amongst many other things) are great concerns when building this type of software.
Over time it was recognized that many software packages shared many of the same attributes, common services, languages, dependencies, etc. Instead of building these common parts into software repetitively, they were installed onto infrastructure and labeled ‘platform.’ Offering infrastructure with pre-configured packages and settings is considered providing a Platform as a service (PaaS). Openshift, Heroku, and Cloud Foundry are examples of popular PaaS environments.
Vagrant
Virtual machines are not difficult to install and use these days. However, a tool that can make the configuration of a virtual machine even easier is called Vagrant. This single tool can stitch together configuration management tools (Chef and Puppet) with virtual infrastructure providers (VirtualBox, VMWare, EC2) in a single set of commands that, when executed, builds a platform. This is powerful because the platform can be provisioned an infinite number of time; each time returning the same results.
Vagrant, out-of-the-box, comes with integration to VirtualBox images and Chef and Puppet configuration management scripts. Very powerful package of functionality that is great for prototyping and rapid deployment of testing environments on short notice.
One of the more recent platforms available is called ‘Docker.’ This solution provides a virtual software container that is lightweight and easy to configure.
Artifacts and Artifactory
Any output of a development lifecycle step is defined as an ‘artifact’ that can further be used (or not) as input to the next step in the lifecycle. Artifacts that result from continuous integration tools are usually in the form of binary files or some other version controlled file object. Essentially, artifacts are where the existing code, new features, functionality and bug fixes are compiled.
Artifactory is a software tool that provides a repository that can store physical and virtual pointers to as many versions/revisions of an artifact as you define. Artifactory also supports special ‘reserved words’ which will allow a query to be used to find a specific ‘class’ of an artifact’s revision. The term ‘Latest’ can be used to always locate and provide the metadata regarding the most recent revision of an artifact known to Artifactory. You can define as many of these ‘classes’ as you like.
Artifactory can be queried with tools like Gradle or Maven for artifacts and their dependencies. This makes it possible to marshall all the software packages (artifacts) to be installed on a platform to represent the final solution (SaaS offering).
Build Automation
A software solution is typically created by assembling two or more artifacts into a package that can be installed onto a platform. This is called build automation. Ant is a tool that has been readily used to build packages and their dependencies as part of a post-build workflow. An alternative to Ant was created called Maven. This tool restricted the instruction set in an attempt to simplify the process of compiling and building dependencies. You can compare the evolution of Ant to Maven in the same way you might compare Chef to Puppet. Both tools do the same work but the later has a proprietary DSL (Domain-Specific Language) intended to simplify a process.
Gradle is one of the latest tools in the build automation category. As with Ant, Gradle was created to give the ability to fully customize and fine-tune every aspect of the build process.
[snippet id=”44″]
Python, Perl, Ruby
Scripting languages such as Python, Perl, and Ruby are used to create the ‘magic’ of continuous integration. With these languages you can build the recipes, cookbooks, playbooks, manifests, and other items needed to support a successful automation strategy.