A beginner’s guide to building DevOps pipelines with open source tools

by Ismail.EL
0 comment

If you’re new to DevOps, check out this five-step process for building your first pipeline.

1. A brief introduction to DevOps and the CI/CD pipeline

If you ask several people, “What is DevOps? you’ll probably get several different answers. DevOps, like agile, has evolved to encompass many different disciplines, but most people will agree on a few things: DevOps is a software development practice or a software development lifecycle (SDLC) and its central tenet is cultural change, where developers and non-developers all breathe in an environment where formerly manual things are automated; everyone does what they are best at; the number of deployments per period increases; throughput increases; and flexibility improves.

While having the right software tools is not the only thing you need to achieve a DevOps environment, some tools are necessary. A key one is continuous integration and continuous deployment (CI/CD). This pipeline is where the environments have different stages (e.g., DEV, INT, TST, QA, UAT, STG, PROD), manual things are automated, and developers can achieve high-quality code, flexibility, and numerous deployments.

This article describes a five-step approach to creating a DevOps pipeline, like the one in the following diagram, using open source tools.

Complete DevOps pipeline

Without further ado, let’s get started.

Step 1: CI/CD framework

The first thing you need is a CI/CD tool. Jenkins, an open source, Java-based CI/CD tool based on the MIT License, is the tool that popularized the DevOps movement and has become the de facto standard.

So, what is Jenkins? Imagine it as some sort of a magical universal remote control that can talk to many many different services and tools and orchestrate them. On its own, a CI/CD tool like Jenkins is useless, but it becomes more powerful as it plugs into different tools and services.

Jenkins is just one of many open source CI/CD tools that you can leverage to build a DevOps pipeline.

NameLicense
JenkinsCreative Commons and MIT
Travis CIMIT
CruiseControlBSD
BuildbotGPL
Apache GumpApache 2.0
CabieGNU

Here’s what a DevOps process looks like with a CI/CD tool.

CI/CD tool

You have a CI/CD tool running in your localhost, but there is not much you can do at the moment. Let’s follow the next step of DevOps journey.

Step 2: Source control management

The best (and probably the easiest) way to verify that your CI/CD tool can perform some magic is by integrating with a source control management (SCM) tool. Why do you need source control? Suppose you are developing an application. Whenever you build an application, you are programming—whether you are using Java, Python, C++, Go, Ruby, JavaScript, or any of the gazillion programming languages out there. The programming codes you write are called source codes. In the beginning, especially when you are working alone, it’s probably OK to put everything in your local directory. But when the project gets bigger and you invite others to collaborate, you need a way to avoid merge conflicts while effectively sharing the code modifications. You also need a way to recover a previous version—and the process of making a backup and copying-and-pasting gets old. You (and your teammates) want something better.

This is where SCM becomes almost a necessity. A SCM tool helps by storing your code in repositories, versioning your code, and coordinating among project members.

Although there are many SCM tools out there, Git is the standard and rightly so. I highly recommend using Git, but there are other open source options if you prefer.

NameLicense
GitGPLv2 & LGPL v2.1
SubversionApache 2.0
Concurrent Versions System (CVS)GNU
VestaLGPL
MercurialGNU GPL v2+

Here’s what the DevOps pipeline looks like with the addition of SCM.

Source control management

The CI/CD tool can automate the tasks of checking in and checking out source code and collaborating across members. Not bad? But how can you make this into a working application so billions of people can use and appreciate it?

Step 3: Build automation tool

Excellent! You can check out the code and commit your changes to the source control, and you can invite your friends to collaborate on the source control development. But you haven’t yet built an application. To make it a web application, it has to be compiled and put into a deployable package format or run as an executable. (Note that an interpreted programming language like JavaScript or PHP doesn’t need to be compiled.)

Enter the build automation tool. No matter which build tool you decide to use, all build automation tools have a shared goal: to build the source code into some desired format and to automate the task of cleaning, compiling, testing, and deploying to a certain location. The build tools will differ depending on your programming language, but here are some common open source options to consider.

NameLicenseProgramming Language
MavenApache 2.0Java
AntApache 2.0Java
GradleApache 2.0Java
BazelApache 2.0Java
MakeGNUN/A
GruntMITJavaScript
GulpMITJavaScript
BuildrApacheRuby
RakeMITRuby
A-A-PGNUPython
SConsMITPython
BitBakeGPLv2Python
CakeMITC#
ASDFExpat (MIT)LISP
CabalBSDHaskell

Awesome! You can put your build automation tool configuration files into your source control management and let your CI/CD tool build it.

Build automation tool

Everything is good, right? But where can you deploy it?

Step 4: Web application server

So far, you have a packaged file that might be executable or deployable. For any application to be truly useful, it has to provide some kind of a service or an interface, but you need a vessel to host your application.

For a web application, a web application server is that vessel. An application server offers an environment where the programming logic inside the deployable package can be detected, render the interface, and offer the web services by opening sockets to the outside world. You need an HTTP server as well as some other environment (like a virtual machine) to install your application server. For now, let’s assume you will learn about this along the way (although I will discuss containers below).

There are a number of open source web application servers available.

NameLicenseProgramming Language
TomcatApache 2.0Java
JettyApache 2.0Java
WildFlyGNU Lesser PublicJava
GlassFishCDDL & GNU Less PublicJava
Django3-Clause BSDPython
TornadoApache 2.0Python
GunicornMITPython
Python PasteMITPython
RailsMITRuby
Node.jsMITJavascript

Now the DevOps pipeline is almost usable. Good job!

Web application server

Although it’s possible to stop here and integrate further on your own, code quality is an important thing for an application developer to be concerned about.

Step 5: Code testing coverage

Implementing code test pieces can be another cumbersome requirement, but developers need to catch any errors in an application early on and improve the code quality to ensure end users are satisfied. Luckily, there are many open source tools available to test your code and suggest ways to improve its quality. Even better, most CI/CD tools can plug into these tools and automate the process.

There are two parts to code testing: code testing frameworks that help write and run the tests, and code quality suggestion tools that help improve code quality.

  • Code test frameworks
NameLicenseProgramming Language
JUnitEclipse Public LicenseJava
EasyMockApacheJava
MockitoMITJava
PowerMockApache 2.0Java
PytestMITPython
HypothesisMozillaPython
ToxMITPython
  • Code quality suggestion tools
NameLicenseProgramming Language
CoberturaGNUJava
CodeCoverEclipse Public (EPL)Java
Coverage.pyApache 2.0Python
EmmaCommon Public LicenseJava
JaCoCoEclipse Public LicenseJava
HypothesisMozillaPython
ToxMITPython
JasmineMITJavaScript
KarmaMITJavaScript
MochaMITJavaScript
JestMITJavaScript

Note that most of the tools and frameworks mentioned above are written for Java, Python, and JavaScript, since C++ and C# are proprietary programming languages (although GCC is open source).

Now that you’ve implemented code testing coverage tools, your DevOps pipeline should resemble the DevOps pipeline diagram shown at the beginning of this tutorial.

2. Optional steps

Containers

As I mentioned above, you can host your application server on a virtual machine or a server, but containers are a popular solution.

What are containers? The short explanation is that a VM needs the huge footprint of an operating system, which overwhelms the application size, while a container just needs a few libraries and configurations to run the application. There are clearly still important uses for a VM, but a container is a lightweight solution for hosting an application, including an application server.

Although there are other options for containers, Docker and Kubernetes are the most popular.

NameLicense
DockerApache 2.0
KubernetesApache 2.0

3. Conclusion

In this post, i have explained how to build DevOps pipelines with open source tools.
If you have any questions or feedback, feel free to leave a comment.
As always, if you found this post useful, then click like and share it 🙂

19020cookie-checkA beginner’s guide to building DevOps pipelines with open source tools

Related Posts

Leave a Comment