Blog Insights
Modernizing Developer Operations Infrastructure: Continuous Integration with Buildkite
Forum One’s technical services team has been hard at work this year overhauling its developer operations toolchain. We have taken a critical look at the tools and services we are using to build and deploy sites and applications for our clients, and we are excited about the improvements we are making in the name of efficiency, modernization and, most of all, security.
For many years, Forum One has relied on Jenkins, an open-source automation tool that helps us to integrate, test and deploy code changes during software development, in a process called continuous integration (CI). Like Drupal and WordPress, open-source Content Management Systems that we frequently work with, Jenkins is supported by a community of developers and its core functionality is extensible via plugins. Currently, there are more than 1,400 plugins available for Jenkins.
Over the last couple of years, however, we started to experience growing pains with Jenkins. Some of those biggest pain points were around the fact that we were not storing configuration in code by default, so it was difficult and time consuming to carry identical configuration between projects and between different jobs within the same project. We were able to mitigate this somewhat by using the Pipeline plugin to define our workflows in code, but we found support for that plugin very uneven, and we sometimes had to resort to invoking raw Java classes from the API. We also ran into issues trying to support different versions of software, namely Node.js and Ruby, which were time-consuming to resolve.
While there are lots of plugins available for Jenkins, most are written by third-party developers, and some are of questionable quality. We found that many plugins we relied on stopped receiving maintenance or support. Jenkins began to require much more dedicated maintenance from Forum One’s team. We were suffering from Cobblers Children Syndrome and lacked a clear plan of responsibility for maintaining the application.
From there, it was just a matter of time, and we experienced a disruptive security incident on one of our Jenkins servers in April 2019. An attacker gained access through an insecure plugin, and it left us embarrassed and exposed. We dedicated more than 600 staff hours in the aftermath as we mitigated the risk to our clients and their data. After careful consideration and testing of our alternatives, we also made the decision to move to a new build automation tool: Buildkite. We wrapped up the transition between Jenkins and Buildkite last month, and we could not be happier or more optimistic about where our DevOps platform is headed.
What is Buildkite?
Buildkite is a Software as a Service (SaaS) platform for running CI pipelines, or defined workflows, on your own infrastructure through its open-source agents. During a build, those agents perform a checkout from a source control repository — in our case, GitHub — and execute the hooks and other tasks we have defined, which we have done in code in straightforward shell scripts. We are currently using hooks to integrate secrets management, which allows us to securely access sensitive information that is critical to our pipelines. We are also using hooks to expose environment-specific variables when necessary. Our build servers under Buildkite are ephemeral; we can build a project based solely on the contents of the source control repository itself, and we will never run into collisions from artifacts produced during earlier builds. The build servers are also efficient. They are provisioned based on build metrics and scaled up or down on demand. Buildkite has a web-based interface that allows us to monitor and control our pipelines. The agents report metrics and other aspects of build activity to this UI, so our developers get near-instant feedback on the state of their projects during the build process. It also includes team-based permissions to pipelines, so we can tightly control access for enhanced security. We have been able to easily integrate Buildkite into the other tools we use, including GitHub and Slack. Forum One is in the process of moving away from virtual machines and toward containers via Docker. The Buildkite agent has plugins available for Docker, so we anticipate a smooth transition. Another advantage of Buildkite is that it supports multi-language builds, which is critical for our projects. Most of our projects rely on PHP, Node.js, and Ruby during different points in the build process. Forum One is diversifying the technical solutions we are deploying for clients, so we expect to expand the list of languages we will need support for in our pipelines. We are also able to build projects that require different software versions. For example, some of our legacy projects need older versions of Node.js than our newer, Webpack-based ones.What is on our roadmap?
We have big plans to introduce new features and functions into our DevOps toolchain, and we will leverage Buildkite in the process. A few items that are on our roadmap include:- Buildkite integration with project scaffolding. We have built a project scaffolding tool called Webstarter with Yeoman, and we plan to expand the tool to automate the generation of the pipeline scripts.
- Automated tests for code quality and security. We will add steps in our pipeline to perform linting, code quality analysis, security-related checks, and enforcement of coding standards.
- Automated testing. We will add support in our pipelines for unit, end-to-end and visual regression testing, and we will be flexible on when that testing occurs to meet the project’s needs: on commit, on merge of a pull request, or on the deployment of a specific branch.