Blog Insights
The Web Starter Kit: Building a Brand New Site the Smart Way
Creating the backend of a new website is a more complex process than one would think. There is a lot of base work that needs to be implemented before development can even begin, and getting to that point both efficiently and smartly is crucial to success.
A few weeks ago, I wrote about local development in the age of containers. As internal systems’ sizes and complexities grow, containers can help to manage growth (and anxiety!). Along similar lines, when developing a new website from the start, there are a lot of great and new ways to get a new project off the ground without having to, as the old ways went, to copy/paste a previous instance in the hopes that you’ll save at least some time.
A brief history of
Forum One uses [Aka:
The very earliest project to be called [Aka:
The [Aka:
The generator served us well, but as time moved on, we began to see some issues. The generator was written in a plugin style, and each plugin was shipped as a separate package on npm. Most users never noticed this because we pre-installed most plugins, but adding a new plugin quickly became a chore. If it wasn’t already included, people had to search for it in order to run it. Since each plugin was built in its own repository, we often found hard-to-diagnose issues due to updates accidentally breaking code in other plugins.
When it came time to re-evaluate our local development stack, one decision was a no-brainer: we had to keep using the generator. Having standardized templates is important when you’re a team of over 40 developers. We took all of the lessons learned from the previous iterations of
A brief history of generator-web-starter
Forum One uses generator-web-starter to stand new websites up in a sleek and reliable way. However, to fully appreciate the complexities around how website projects instances are first created, it’s helpful to look back at how we’ve tackled this over time. We as developers are always searching for ways to make things easier and more efficient, albeit with a number of bumps along the way.
Back when copy/pasting was the way to go
[Aka: $ git clone forumone/web-starter]
The very earliest project to be called web-starter was a single GitHub repository containing the necessary configuration to provision a Vagrant VM. Developers started new projects by cloning the repository and starting a project on top of it. The “update” process, if it can be called that, was to download Web Starter’s files again and paste them into the project. While this was a good way to save time, it sometimes led to breakages when developers forgot to re-apply their customizations to the project.
We abandoned this approach relatively quickly due to this drawback. Our projects have a variety of requirements. For example, not all projects required a search backend, and some projects had to install specific versions of software to match what was installed on the production environment. Developers couldn’t remove or modify what came with the web-starter project without the next update reverting those changes. Imagine upgrading PHP, only to find that it had been downgraded by another developer by accident!
Developing a more customized Web Starter solution
[Aka: $ yo web-starter]
The web-starter project taught us a lot about our own needs and how developers needed to be able to interact with the installed project. We took web-starter’s default configuration and broke it down into a more modular structure using the Yeoman scaffolding tool. Thus, the generator-web-starter project was born.
When the generator is invoked, it prompts the user with a small number of options, such as which CMS to install, which PHP version is required and which additional services, such as Solr or Memcached, should be installed. Responses to these prompts are saved in a file and tracked in the project’s repository, ensuring that re-running the generator produces the same result for different users. This all but eliminated the possibility that users accidentally revert any customizations they may introduce to the project.
As mentioned in my previous post on containers, the first iteration of our Vagrant setups used Puppet. We used Hiera to expose simple configuration settings that a user could change once the generator had run. This allowed some degree of self-service for things such as the MySQL server’s configuration. By using a generator, we were able to migrate to Salt by simply creating a new plugin to apply Salt templates instead of Puppet, and deprecating the old Puppet plugin. We kept a high-level options file to continue to allow users to apply small configuration changes without needing support in code.
Under the hood, the first iteration of generator-web-starter used what we’d now call old school code. The codebase was first written against an old version of Node.js, and thus the code and modules relied fairly heavily on community support for what are now built-in functions – for example, the Bluebird library instead of native Promise functionality.
Not resting on our laurels: creating an even better solution
[Aka: $ yo web-starter@2]
The generator served us well, but as time moved on, we began to see some issues. The generator was written in a plugin style, and each plugin was shipped as a separate package on npm. Most users never noticed this because we pre-installed most plugins, but adding a new plugin quickly became a chore. If it wasn’t already included, people had to search for it in order to run it. Since each plugin was built in its own repository, we often found hard-to-diagnose issues due to updates accidentally breaking code in other plugins.
When it came time to re-evaluate our local development stack, one decision was a no-brainer: we had to keep using the generator. Having standardized templates is important when you’re a team of over 40 developers. We took all of the lessons learned from the previous iterations of generator-web-starter to build a new, more sustainable architecture for the 2.0 release.
We kept the plugin style, but brought all the plugins into the main repository, allowing us to ship updates much more effectively and consistently. The codebase received a significant overhaul as well. On the frontend, we’ve had success building projects with TypeScript for years. We used that for the new version here and have found the efficiency gains to be remarkable. Using a type checker when building an application adds a great deal of confidence both when working with existing code or adding new features.
The 2.0 release coincided with our switch to Docker Compose for project management so that each plugin shares access to a class that manages the Compose file that is generated for the project. Self-service maintenance became substantially simplified with Docker Compose: users can now inject their own configuration for MySQL or Solr by editing the Compose file.
We have also found the use of Docker Compose to be a useful way to “field test” new configurations before committing to supporting them in the generator. We can verify that specific setups (e.g., Python-based Django projects) work before standardizing the template. Some contributions to the generator’s templates began as tweaks made to a project to address a quirk or add a new feature.
Overall, we find the current generator-web-starter to be accessible for both users and contributors, and are excited to see where we can take this tool into the future.
Here and next
Thegenerator-web-starter project has been a core part of our development ecosystem for years. Its capabilities have grown in order to help us take on the challenges of larger and more complex websites. The modern Web is an ever-shifting shifting landscape for both users and developers alike. As a result, starting from scratch isn’t an option anymore – we need smart and effective means to get building quickly. It’s important to us that all of our projects start from a stable foundation, and we’re always looking ahead to find ways to make our tools the best they can be. It’s with these tools that we bring our clients the best possible experiences for their Web presence.
And stay tuned: in my next blog, I’ll tackle the forumone-cli package and how it addresses additional pain points when spinning up a new website.
