development

SimplyTest.me From The Ground Up

admin — Wed, 22 Sep 2021 22:58:00 +0000

SimplyTest.me From The Ground Up

When I took over as the project lead for SimplyTest.me, the previous lead shared three primary things with me:

The system had a non-trivial amount of technical debt and was rising more with time
Significant changes were coming with Drupal 8, Composer, Drush, and more
His interest was elsewhere

As part of my proposal, I promised to do what I could to revitalize this. While this effort has taken years, we have hit another major milestone in our journey:

We launched a completely new version of SimplyTest.me. We rebuilt the system from the ground up in an effort to revitalize the project. It’s time to tell the story.

The Old System

Architecturally, the SimplyTest I inherited had three basic systems: the web server front-end, the worker server backend, and a proxying system that glued everything together. Everything was manually maintained and largely custom.

The servers were hand-built and required a ton of maintenance. All of the scripts to run the system were custom.
The worker backend was a glorified, monolithic LAMP stack that would provision and deprovision virtual hosts, databases, and hostnames on the fly.
Cleanup routines would fail commonly on system updates, disks would fill, logs wouldn’t rotate, and the maintenance burden was high.
Major upgrades to components or the operating system would have caused an indeterminate amount of breakage. If you tried to fix and subsequently broke any part of the system, this would impact all of the running instances.
And, there was no failover or load balancing.
The application was running on Drupal 7, which means it was almost three major versions behind.
Backups did not readily exist.

SimplyTest did not have an infrastructure that afforded easy development or encouraged contribution from others in a safe and straight-forward way (without basically using the fragile production systems). Making changes often broke things and the systems would go down routinely. It was not good.

People would complain. A lot. We tried to keep up. But, the service wasn’t stable, nor was it easy to keep up with the latest innovations our community was developing. People routinely let me know about it. They also would routinely blame SimplyTest for issues with their own projects by expecting me, the maintainer of the service, to debug their issues. I recall a vocal community member persistently accusing the system of being broken because “it works on their local system” only for us to find one of the dependent third-party assets had a broken link in their module’s library file (which they already had downloaded locally and they didn’t properly test it).

It was an incredible effort to just keep the lights on. And, it wasn’t sustainable. It was important that we created a new future.

Rebuilding SimplyTest

While a fresh approach was desperately needed, we have finally reached a point where SimplyTest.me has been rebuilt from the ground up. We spared no opportunity to modernize the project, which allowed us to shift our attention back to serving the community. The following sections highlight our accomplishments around revamping SimplyTest.

Goals

Revitalizing the project needed to be prioritized around what our users would get value from. As an example, lowering technical maintenance or eliminating technical debt affords more time to develop the features of SimplyTest.me or more quickly introduce changes tied to new community innovation (see: Gitlab workflows in drupal.org, updated PHP version support in Core, Composer/Drush improvements, and more). Time spent working on the system would not go toward keeping the lights on, but actual value built into the project our users would benefit from.

Increasing contribution and available contributors is a long-standing goal. Not only do I want SimplyTest to be used for lowering the technical barriers of contribution, it is the perfect platform for learning and contributing something meaningful back. We made conscious architectural decisions with an eye for contributor growth tied to technologies we felt people wanted to learn. And, revitalizing the systems happened with an eye toward a cleaner, modern architecture that would afford more straight-forward contributions without the overhead of legacy systems.

A better, more stable service offers more value. If our users get value and they continue to get value, I believe they will help sustain the project through donations or contributions. And, they can focus on the great things Drupal offers. Put differently, if people routinely experience challenges with the service or have trouble getting value from it, this lowers confidence and they are less likely to invest. SimplyTest must seek and be in a position to drive more contribution to remain viable. As such, we hope our work to build a new SimplyTest will hopefully position it to reach its fullest potential.

Another goal is to have SimplyTest be a showcase for the best that Drupal offers. It wouldn’t just be built for Drupal but with Drupal as well. If we found ways to make Drupal or it’s dependencies better through our efforts, we did so. This is how the ecosystem works and we wanted to be fully responsible for this.

Any modernization would be adding value, not losing it. We aimed to have at least 100% feature parity with the old version. I’m proud to say we achieved this goal.

Finally, we stuck to our core tenets. We remain committed to open source. The service would be and will forever be free to our users. And, our actions reflect our fundamental goal to lower the barrier of entry for those who want to use Drupal and learn about the wonderful community we’ve established.

Donations and Paid Labor

I don’t expect people to work for free. Some choose to volunteer and I’m really grateful for that. But, when I took over, I had no money to pay for anything. I established an Open Collective to help with this. I began accepting donations. Initially, this helped pay for expenses like SSL certificates and various attempts to make development infrastructure. But, I saved up money to help pay for sponsored contributions. I was proud to offer some people work when they were out of jobs. I hope to do a lot more of this in the future.

Most importantly, this allowed me to spend some funds toward the revitalization. The project started to get some much needed help and velocity toward our goal. I was able to pay people with specific skills to come in and do work faster and of higher quality.

A lot of this progress was made possible by donations of small and large from both individuals and organizations. It was really cool to also see Drupal Camps, who often leverage SimplyTest for contribution days, make donations with some of their funding. And, companies like Centarro, sponsored development of specific features (the one click demos).

PaaS Hosting

With a volunteer team and limited time/budget, you need to be very careful about what you maintain with a project like this. We learned the hard way how difficult it is to maintain your own infrastructure. In my mind, you can’t waste time reinventing the wheel. While it may seem to conflict with our open source roots, we chose a service and vendor that was closely aligned with our identity: Amazee Lagoon. We are grateful they sponsored the use of their service to help run our Simplytest Drupal 9 website. While their PaaS service is a paid offering, their product is open source. And, it had relevant features and a high degree of customizability we needed to be effective. Amazee also offered use of it’s CDN services, which help with both performance and security. Worrying less about a website affords time for us to build more value into the system. We’re fortunate to have access to this service to allow us to improve the service itself.

Drupal 9 Website

What a better way to showcase Drupal by using Drupal itself. We wanted to harness the best Drupal had to offer as part of our revitalization. We actually started our efforts with Drupal 8 when we began to create the new application, but we then upgraded to Drupal 9. We have leveraged many cutting edge features, like the Typed Data system and the web service capabilities, which help modern Drupal applications shine. We’ve been able to leverage core features and the underlying framework to create a set of services, data models, and API endpoints that pair perfectly with our React-based front-end.

SimplyTest has also innovated significantly around the Drupal community project metadata. Because our efforts were “experience first”, we ran into challenges implementing the ideal experience in support of semantic versioning, compatibility of major versions of core within projects, and more. Major props to Matt Glaman and Benji Fisher for helping research and innovate around these challenges.

Fresh Design

Comic Sans is a sign of the nineties, and the nineties want it’s website back. We’re happy to oblige. With a totally revamped system, we needed a modern design that best represented the technical innovations under the hood. Ana Laura Coto helped with this by producing designs for us, including a new logo! This will help our experience, allow our work to shine, and best represent how cool and modern the new system actually is.

Updated Theme

Iris Ibekwe set up our initial efforts when we implemented the Drupal 8 theme. It was responsive, leveraged modern grid systems, and established build tooling with Gulp, NPM, and more. This work helped get the ball rolling toward implementing our new designs.

A lot of time passed from when the original theme was set up and when we got the new backend fully built. Some newer tools came out that allowed us to upgrade some of the older tools and innovate further. Matt Glaman implemented Laravel Mix, which allowed for live refresh and replaced the theming framework with Tailwind CSS.

QED42 stepped up majorly to finish the job. Their team did a significant amount of theming work and QA. They applied some updated design revisions made by Ana Laura to address identified accessibility issues. As the backend was evolving, the theme was being completed in parallel by their team. The QED42 team got the job done and their team was a pleasure to partner with. Front-end work is one of my technical limits. I was grateful I had such great help with this area.

ReactJS Front-end

The interface for SimplyTest has a lot of conditional behavior. It’s highly interactive. This presents our users with an experience that simplifies much of the underlying Drupal complexities. Users enter a project, they pick a branch, they then only see relevant options to them based on what they want to test. Drupal has evolved a lot and putting the experience first for our users has meant a lot of work behind the scenes to abstract complexities. This work is never done, but our new ReactJS front-end is a major step forward.

Matt Glaman created the React front-end and the connecting backend APIs from Drupal. ReactJS is a great compliment for the web services sponsored by Drupal. The React app maintains state, which can help to address conditional needs and guide users through the experience we want to offer. This was a complex task and Matt’s leadership helped make something from nothing. I am just learning ReactJS and was struggling to make the kind of progress on this we needed. Matt’s expertise was critical to connect both systems and make this a huge success. I’m far more confident that others can contribute to this part of the system and use the React app as a reference for learning (I certainly will be).

Automated Testing

We were making sweeping changes throughout the last several months. When you add in development activities in parallel, it is much easier to break things. Thankfully, automated testing was implemented throughout our process.

As Matt was implementing the Drupal backend and the ReactJS front-end, automated testing continued to be developed at the same time. PHPUnit tests were helping with the Drupal endpoints. Cypress was used to test all of the front-end and interactivity. Having automated tests allow us to have a stable development workflow that ensures we’re less likely to break things when pushing changes.

Website Repository and Quickstart

SimplyTest is a Drupal profile maintained on drupal.org. But, our application runs like a conventional Drupal website managed by Composer. Having a repository with SimplyTest installed allows for more key outcomes. First, the repository maintains the Drupal website running SimplyTest.me in a conventional manner to other Drupal applications. This repository serves as the integration with our new PaaS hosting and it looks and feels just like a standard Drupal application (because, it is). Second, the repository is a quickstart for contributors to load the website locally with minimal work. Community members can readily contribute just like any other Drupal project. This helps save time for those who want to participate and don’t want to waste their time with all of the manual setup. We also maintain both DDEV and Lando support within this Quickstart, with some basic documentation we aim to continue to improve with time.

Violinist

Again, we don’t want to waste time with manual work if we don’t have to. Matt installed Violinist on our SimplyTest website repository, which scans Composer.json files on the main branch for Composer updates. If updates are found, Violinist creates a pull request for the update. This works for Drupal core, all of the contributed modules, and even the SimplyTest code on Drupal.org. Pull requests trigger our automated tests to get immediate feedback. And, it sounds wild, but I can basically update the site from my phone. This is immensely cool and something that comes in handy for rapidly applying security updates.

Modern CI/CD

Our new CI/CD workflow offers smaller, incremental changes we can test iteratively before deploying to production. Our workflow with Violinist, pull requests, Github Actions, and on-demand environments on Lagoon allow us to get testing feedback, deploy small changes frequently, test them before pushing to production, and deliver with high quality. We have moved from having a monolithic infrastructure that wasn’t change friendly to being able to rapidly test and deploy changes in an ephemeral, highly automated, and change-friendly infrastructure. It’s been awesome and it’s such a sharp contrast from where we were before.

Monitoring and Alerts

I leveraged the free version of Pingdom to put basic alerts on the website and worker backend when I took it over. I got alerted for outages, which were frequent before. All monitoring and alerting has been switched to the new PaaS website and leverage TugboatQA service notifications (and Linode as their vendor) that could impact our backend. I believe there are more opportunities to improve this in the future, but the basics are there.

Live Coding Streams

Not only did Matt Glaman help with development and architectural guidance, he had an idea to do live streaming sponsored by his company, Bluehorn Digital. I learned a lot from these streams and am confident many others have as well. Streams cover development, testing, issue queues, development tools, and more. It’s great having someone talk through the process the entire time to understand the details. The streams were great for smashing bugs, working on new features, diving into the architecture, and advocating and promoting SimplyTest itself. They continue to be something I look forward to and were an immense help to launching the new site.

Partners Program

Between Bluehorn Digital, QED42, TugboatQA, Centarro, and Lagoon, it is very clear SimplyTest would be in rough shape without these partners. SimplyTest would not even exist without Druid and Maloon who sponsored the legacy systems. It is equally important to say thanks and highlight their contributions to the project, so I established a partners program. For companies who continue to invest in SimplyTest, we will continue to recognize you and offer incentives as we are able. One incentive is the “first right of refusal” for sponsored SimplyTest work. Partners get first dibs when we choose to pay for enhancements or bug fixes. Please see my previous blog post on the topic to learn more about the kinds of partnerships we’re offering.

Social Media Presence and Talks

It is incredibly important to engage with the community. We sincerely want to know what you think, where we can do better, and how we can deliver the best service possible. Thanks to AmyJune Hineline, we have established a Twitter presence, we are active on our Drupal Slack channel, and we have given talks at several camps that have highlighted our plans and why we’re doing this. AmyJune routinely uses SimplyTest as part of new contributor workshops. Because of this, we see much more engagement. Users routinely file issues, send messages, offer ideas, and help us be better each day. We’ve seen donations from the community. It’s such a stark difference from when we had no presence when I took over the project. And, AmyJune has been a critical part of helping establish and build this community from basically nothing.

Contributors

There are too many people to thank that helped us get where we are today. And, this was a collection of things that led up to the big reveal of our brand new Simplytest.me application. None of it would have been remotely possible without several people and companies stepping up to help. I mentioned several people already in this post and there are countless more who cheered us on, offered advice, pitched in, made donations, and so much more. This effort is no longer “my” side project, it’s now “our” community project. Thank you, thank you, thank you.

Parting Thoughts

We have finally put SimplyTest back in a good place. This is worth celebrating. Our technology is best-in-class and is appealing for anyone to use or learn. We managed to drastically reduce the code and simplify the application, lowering the barrier of entry for SimplyTest itself. We reduced maintenance activities down to nearly zero for both the application and the underlying infrastructure to allow focus on value creation. We’ve built a community that has promises of growth. And, we have the tools in place to sustain it. In retrospect, all of these small steps have added up to something significant in the few years I’ve led the project. It truly was a team effort. I couldn’t be more excited to finally be at this point and share this with everyone. I am grateful for those who pitched in and even more happy to say, “we did it.”

This whole thing has taught me a lot. In a lot of respects, SimplyTest is both the best and the worst of open source. It’s not only a free service but all of the code is open sourced. It’s a great example of both companies and individuals pitching in toward a common cause. But, after having led these efforts, I have seen first hand how people can treat open source maintainers when they get behind a keyboard. Some users expect a lot but fewer are willing to pitch in and make it a success. Our system remains largely volunteer-led, donation funded, and services sponsored. We’ve been highly resourceful with what we have (which is very little in the grand scheme of things). At times people just seem ungrateful. On a personal level, I wish this was more harmonious. I’d love for more help (financially or through direct contribution). I’d love for more awareness for those who expect more than we can reasonably offer. Regardless, we try to put our users' needs at the forefront of the service. We welcome the challenge to serve everyone. And, we are in a much stronger position to serve people now but it took so much time and effort to get there.

Another lesson in open source is how hard it is to have a big impact without resources. Working alone requires investing a lot of time and effort outside of your day job. It’s a fast way to burn out. SimplyTest.me historically relied on volunteer labor, sponsored systems, and had no money. Druid and Maloon graciously paid for all of the web, worker, and proxy servers. But, without funds, we couldn’t pay to get help from subject matter experts when it was needed. We just had to roll up our sleeves, work hard, figure things out, and be really scrappy. We picked up the phone to call friends when it was urgently needed to keep things running. But, one can only do this so long. And, I became fully aware of why the previous maintainer decided to move on. A large portion of time was focused on incidents and “keeping the lights on” while wanting to build the future. Wholesale change was needed and it makes it even more rewarding to launch our brand new system.

One more side note… the largely volunteer Drupal community is an amazing example of what can be accomplished at scale. A small number of contributions by a lot of people can add up to something significant. And, yes, SimplyTest is a part of the community. But, SimplyTest certainly has not seen the vast contribution that the Drupal ecosystem benefits from. We will continue to aspire to reach and achieve the kind of scale the broader community benefits from.

The Future

We need to grow into this new project we just launched. We just shed a ton of technical debt and can finally breathe easier after a rough few years. We still have a lot to learn and improve upon, and we need to work out the bugs. It was a lot of change in a (relatively) small amount of time, but we feel confident in the decisions made and to put us in a position of success. It’s incredible what a small number of highly motivated and capable people can do with clear goals, clear strategies, and clear purpose.

In the coming weeks, I will be publishing a quarterly roadmap talking about key objectives we wish to hit moving forward. We will start with a pause in new features to fix bugs, stabilize, and identify improvements. We will then announce a prioritized set of features which will be informed by the feedback we get.

We thank all of you who supported the service through our transition and when the system did not live up to your expectations. We’re confident that the future is bright, and we look forward to having SimplyTest serve the needs of the community for a very long time to come.

admin Wed, 09/22/2021 - 18:58

Tags

Drupal CI/CD with TugboatQA and Github Actions

admin — Tue, 29 Dec 2020 23:54:38 +0000

Drupal CI/CD with TugboatQA and Github Actions

Vacation this year has been amazing. I've caught up on some of my long-standing to do list, like launching this new nerdstein site and digging into some of the newer technology I've wanted to explore. SimplyTest has some great new contribution I am excited about. I've spent a ton of time with the kids coloring, playing, and just enjoying these moments while they are young. It is shocking how quickly they have grown. I've cooked a lot and those close know how much I enjoy doing that. I've blogged more - finally - after a long hiatus and lack of motivation. I am currently enjoying an Eight & Sand Brewing - Loco Mo Sim DDH and listening to the Gravity album from Our Lady Peace. The kids are currently asleep. I wanted to share some things I've explored today.

Automation is important for consistency and for ease of use. It's one of the tenets of DevOps. Automation is the cornerstone of creating a great experience with SimplyTest.me. I've spent years creating continuous integration solutions for customers to help them be more effective. Yet, I never did that for my personal website... until now.

Building off of my new website launch, I wanted to keep the momentum going. Recall from my blog post yesterday, I set up the basics of TugboatQA to replace my development server with something more modern. I also set up a new Linode for my production server. I wanted to learn Github Actions for deployment automation, as I had never used it. I desired to have an end-to-end CI/CD pipeline built completely with GitOps. And, I wanted as much of this automated as possible.

I was pleasantly surprised with how smoothly things went. Maybe I shouldn't be, as I'm basically building a pipeline for a simple website. Let me share a bit more about this experience should you find it beneficial.

Pipeline

This pipeline basically happens in the following workflow:

I work locally and push changes to a branch to Github (Composer, custom code, config, etc)
I make a pull request for this change off of the branch
Pull requests are subsequently deployed into their own Tugboat environment (executes a fresh environment that pulls production databases and files with automated deployment)
I review the code for anomalies or missing things like config
Manual testing occurs on the Tugboat environment (note: I need to work on automated testing; I plan to implement the visual regression test of TugboatQA in the future)
The pull request is merged after it passes my thorough review ;)
A production deployment happens once the code is merged

For the sake of what I consider novel, I'm going to focus on two main areas of this pipeline: TugboatQA and Github Actions. I assume a production server exists, but I will describe what is needed to manage assets from production (the source of truth for content in Drupal). And, I assume people are good with the well-documented pull request workflow methods. Please reach out if any further details would be helpful.

Commentary: Drupal Assets

My production site is a basic LAMP stack with some specifics for Drupal, like Composer and Drush (site local, of course). Any true Drupal development workflow needs to readily be able to get databases and files from production as the source of truth. Files are relatively easy with the Stage File Proxy module in Drupal, which can replace local files with those served remotely. Drush Aliases are a great way to do the database backups. But, I wanted to avoid putting credentialed data that into a public repository (note: these features are often offered by Drupal PaaS providers and you should use it). I opted to generate a script that could be invoked upon deploy and used mysqldump to create a gzip'd SQL file on-demand. Easy peasy bash script-ezy (sorry, I'm watching too much Food Network during this pandemic hell).

TugboatQA

TugboatQA replaced my development environments with automated environments per change (pull request). TugboatQA natively has Github integration and the Tugboat repository settings allow for new environments to be provisioned upon pull request.

Tugboat's infrastructure-as-code capabilities resemble Docker Compose (I denote this as the services layer), but Tugboat has specific hooks that allow for automation scripts to run at specific events within the lifecycle of an environment. The offering is catered for building great environments in a variety of platforms. As such, the services layer is readily configurable, allowing it to have parity with the Linode server tied to a set of Docker-based images. I'll probably try this out for a NodeJS-based workflow in the future.

I was able to add in the basic Drupal deployment steps into these hooks for the desired automation. This can and should be specific for your project. Mine did the basics: Composer install, drush config import, drush update database, and drush cache clear.

Finally, TugboatQA generates a public and private key for every project. I was able to create a locked-down, Tugboat-specific user on my production server where I copied the TugboatQA public key into the authorized_keys for that user. This allowed me to explicitly control what Tugboat could and could not do within the server. This allowed me to only have Tugboat run a script on production that made a database backup and leveraged scp to retrieve the generated backup into the Tugboat environment.

Example

You can see how I've configured the TugboatQA behavior through its config.yml directive:

Github Actions

I wanted something to kick off a production deployment. Much like AWS Lambda, this type of capability only needs to execute a script which is in sharp contrast to a more persistent, permanent server like production. This deployment automation only needed to happen after I reviewed a change in Tugboat and after I merged a pull request. Github Actions had just what was needed to do this.

Secrets Capability

One of the "secret" sauces of Github Actions is the integration of their secrets feature. Think of this as a private, secure key value store. Instead of committing scripts with a bunch of private information (keys, server IPs, usernames, passwords, etc), you can store your secrets in the repository settings and pull them within committed, ever changing scripts. Very cool.

Events

I only need my production deployment to happen after a pull request is merged into master. This signifies that a change has been reviewed, tested on Tugboat, and it is not expected to break a bunch of stuff. Github does not explicitly have a pull request creation or merge event in Github Actions. However, on further thought, what I really care about is when I push code to master. Suppose I have a hotfix, like a security update in Drupal. Yes, I should push this through normal channels. But, time is of the essence. So, I may want the opportunity to commit code directly to master. Yes, this can break things but I can't rule out the possibility of needing to do this if I feel like it needs to be done (process for the sake of process is not effective). Pushing any code to master actually covers me for any deployment I care about - a hotfix or a pull request. I focused on this code push event for the basis of a production deployment, which was supported by Github Actions.

Automation

Much like Jenkins is commonly leveraged as a CI tool to automate remote deployments, I didn't want Github Actions to do much more than serve as a proxy to my production server. In a similar fashion to Tugboat, a Docker image can be leveraged in each step of a Github Action "used" to execute steps of your workflow within a specific runtime. I chose a simple "SSH Action" image that allowed me to run commands remotely with a clean abstraction to SSH into my production server. I was able to readily embed my secrets into this script thanks to this abstraction, which had parameters for the SSH credentials that I embedded the secrets into.

Like what I did with Tugboat, I added deployment commands for the basics of a Drupal deployment. I updated Composer and leveraged Drush to update the database schema, import the config, and clear the cache. One notable difference was pulling in the most up-to-date code from the master branch. If I had something more complex, this could have accounted for environment-specific differences with tools like Config Split.

Example

You can see how I've implemented the Github Actions configuration through it's provided interface in Github:

Conclusion

It is astonishing that people routinely accept the norm when there are better options available. How do you choose to invest your time? Personally speaking, I have better things to do than manually deploying code when I want to update my website and subsequently scrambling to fix things that break when my QE process was to blame. Imagine those that have to deal with this type of thing frequently and in their professional career. These "manual" approaches add up in terms of technical debt and required investment. As technologists, we need to be better and push to make the systems we build serve us too. Ultimately, it benefits those we're trying to serve by freeing us up to do better with the time and opportunity we have.

admin Tue, 12/29/2020 - 18:54

Tags

Finally! A website refresh

admin — Tue, 29 Dec 2020 03:43:46 +0000

Finally! A website refresh

Let me welcome you to the new nerdstein.net. I'd like to thank Ana Laura Coto for help with the design. I'd like also thank Jonathan Daggerhart for being there when I needed an assist. I am really happy to finally share this project with the world.

This has been a long time coming. Three years. Yes, you read that correctly. Three years. Why? I prioritized learning. And, I am slow and methodical. This website is a culmination of many things I learned over a three year span.

But, three years?!? Yep. I have a rewarding and challenging full time job at Acquia. I have a family with two young kids. This project was all done in my spare time and at my convenience. Most of my time and learning is invested in SimplyTest.me. But, this project gave me a high degree of freedom on my own timeline. I wanted to learn things well and understand things properly. I did it piece by piece. Drupal evolved a lot in three years, as well. There was a lot to learn. And, then 2020 happened (I'll hopefully blog more on that soon).

For the rest of this blog post, I'd like to reflect on these efforts and learning.

From Drupal 7 to Drupal 8 and then Drupal 9

Drupal evolved significantly from Drupal 7 to Drupal 9. My original goal was to launch this with Drupal 8. Then life happened. Then Drupal rolled more features and made it even easier and better to launch my site. I'll share some specifics below. Regardless, Drupal has established itself as a great tool for progressive enhancements.

One notable change on this journey is how the community adopted Composer. I recall the very basic support when Drupal 8 was launched. Major changes were made when I upgraded the site to Drupal 8.7, which totally revamped the Composer work. I did the same when Drupal 9 came out. The community made some significant progress, but I'd love to see this stabilize now. I made heavy use of Composer and it's DevOps capabilities to script some of the integrations with my design system. This area of work was not well documented, especially in moving between versions and resolving issues.

New server

I was long overdue for a tech refresh. I had the same VPS running my Drupal 7 site for over seven years. It served me well. I spun up a VPS for a few Drupal 8 sites I hosted for local non-profit orgs that couldn't afford it. My Drupal 8 development site ran there for several years. The non-profits moved on due to grant funding and poor technical support (hint: if you can't do something well, don't do it). And, that Drupal 8 server rapidly dated itself.

I was able to shut down both VPS servers and create a brand new Ubuntu 20.04 on Linode. Linode has been a sponsor of SimplyTest.me, helping offset the cost of it's development infrastructure. I'd hardly call the experience of Ubuntu 20.04 learning; it was a breeze setting up a Drupal 9 environment. These systems have come a long way even in seven years. Shutting down the VPS servers is bittersweet. I've removed some significant tech debt but those servers were rock solid. No regerts.

Design Systems and Drupal Integrations

The biggest emphasis of learning was the front-end work of the design system. I am still so far from an expert. But, I definitely understand it now. I started with Pattern Lab. I broke down Ana Laura's design into a series of atomic components. I developed them. I learned more about SCSS and building of front-end assets. I mocked up these patterns with JSON data objects and finally built out representative pages with the pattern.

Once the design system was done, I started on the Drupal integration. My goal was to pull as much as possible from the design system (no cheating!). Composer made it easy for me to keep the design system and theme separated. I leveraged the Components module and proceeded to site build and theme simultaneously. Some of this was challenging. It was a fight between properly structuring content, overriding Twig themes, leveraging the Components module to include pattern markup, and parsing Drupal-related data objects.

I blogged about this several times, both in terms of the concepts I used and the architecture I leveraged. Please feel free to read more about this if you are interested.

Twig. All the Twig.

Developing Twig was hands-down the most time consuming part of this project unquestionably. I spent countless hours digging through Drupal rendered objects, raw data, attributes of that data, and much more. I tweaked the display settings, created new view modes, did more tweaks, changed more Twig, and this went on endlessly. Data objects passed to Twig were massive and nearly impossible to debug without printing keys, refreshing, and traversing deeply nested objects. The syntax changed with every different entity and then even more with Views. Yes, Drupal is incredibly powerful and extensible. But, I continue to be surprised by the developer experience around Twig. I just wish this was simpler. I know the community has developed several utility modules. I need to dig into those more.

Media System: image links

I had aspirations of contributing a composite field type module back to Drupal core. Developing this module, which never got complete, was a huge rabbit hole. I thought it would be relatively straight forward to inherit the core-provided image and link field types into one new field type. This was significantly more complex, especially when dealing with cardinality, managing data storage/schemas, and dealing with field widgets that were built atomically. This approach was rapidly becoming complex and it didn't seem like it needed to be. Then, help was on the way...

Daggerhart shared with me the new Media entity delivered as part of the Media system in Drupal 8. I was able to create an Image Link media type that was composed of an image field and a link field. This was super easy. From this, I got all of the expected benefits: Views support, easy integration with other entities, and the perks of the new media features in Drupal. Lesson learned: spend some time figuring out what parts of Drupal can be used before diving into code.

Migration from Drupal 7

My website is quite simple. It's a basic blog. I wanted a simple way to migrate the content. Drupal offered that with it's native Drupal to Drupal migrate UI.

Set up was simple. I was able to set up a secondary database within settings.php for my Drupal 7 database. I archived to the files directory on Drupal 7, extracted them in a specific file path, and entered that path in the UI.

When I ran it, I got a nice confirmation page of potential issues with the Drupal 7 configuration, modules, and structure. I worked through it, ran the migration, and it basically worked. I had a few nodes that did not properly map their type. I cleaned this up through the database by manually editing the node type in the node and node revisions tables, followed up with a cache clear. I also had a few nodes that did not display the body field content. I simply re-saved the node and it worked.

I had to run this migration multiple times over three years as I refreshed the content on my current site. The UI didn't offer a rollback (this would be a nice feature through the UI) and had a big red note of caution when running multiple migrations. I did it anyways and it seemed to work fine for me (again, with my simple use case). Buyer beware.

To learn what happened behind the scenes, I audited all of the migration classes and configuration that were created. Kudos to those who worked on making the migration experience a positive one. I was really surprised how much was automated, how insightful it was, how smooth it went, and how easy it would have been to override the underlying config it created.

TugboatQA

As I previously shared, I wanted a tech refresh. I thought about spinning up a development server, but this isn't really a modern CI/CD workflow. Having such a positive experience with TugboatQA on SimplyTest.me, I wanted to rapidly see changes before posting them live. And, I wanted something that didn't require me to mess around with another server. TugboatQA for the win.

Set up was straightforward. This may be potentially because I understood most of the concepts from SimplyTest.me, even though the use case couldn't be more different for my personal website. I followed the documentation and configured it within a few hours (love their IaC set up). I actually found a bug in how I implemented Composer paths, which I subsequently fixed.

Ported Block Type Templates module to Drupal 9

Again, I spend a lot of my personal time on SimplyTest.me. I still moonlight with some module maintenance, but I do a pretty poor job at this if I am being honest. I did use the Block Type Templates module I developed to have more fine grain control of block Twig. This was immensely helpful with the design system integration. But, I had to port it to Drupal 9. I also cleaned up the issue queue and added some nice features. I hope this helps others, too.

Future Goals

This website is definitely not done. I have some open issues I've logged. I need to clean up some content, fix even more styling, and finish the Tugboat implementation now that I've launched. It's done enough for me to not maintain two websites any longer. Please feel free to let me know if you find any issues.

I also want to add more DevOps. When a PR gets merged, I want to set up a GitHub action to trigger the deployment on prod: pull updates from Git, load the config, run the database updates, and clear the cache. My confidence is a lot higher with Tugboat and my CI/CD workflow in the mix. This should be fun.

I'll be finally adding SSL support for my site, but I'd like to do this by learning more about the edge layer. I need more research but leaning towards using CloudFlare for this.

Pattern Lab has evolved majorly since I first set up the design system. I'll be upgrading this with time and I have the Drupal integration versioned to allow for changes like this to be done in the future.

Final Thoughts

I'm glad to finally be able to launch this thing. I learned a lot doing this. Drupal is still a fantastic option for digital experiences. It does so much. Drupal 9 brings all of the previous years of learnings, new features, great performance, modernized dev workflows, and continues to deliver a highly extensible framework. I thought a lot about developing this in something like Gatsby, but would have still wanted a FOSS backend like Drupal. It did everything I needed natively.

admin Mon, 12/28/2020 - 22:43

Tags

Getting Started with Drupal Rector Development

admin — Tue, 05 May 2020 16:53:14 +0000

Getting Started with Drupal Rector Development

Drupal 8 introduced a lot of changes. Its usage of best-of-breed dependencies, like Symfony, represented a major shift for Drupal. Drupal 8 also moved to a more nimble release cycle with its major and minor releases, allowing a major version of core to get new features in minor releases. Between dependencies releasing updates and changes to core being more common, Drupal needed to evolve to manage change. One way this happens is through deprecations. This provides a way for Drupal to evolve, identify old code as deprecated, and offer guidance on a replacement. Deprecations were especially common as Drupal evolved from a largely procedural approach to one leveraging more object orientation and some of the design patterns Symfony uses, like controllers, services, and more. But, the great thing about this approach for deprecations is that deprecations do not get removed until the next major release of Drupal. This buys the community time to progressively fix deprecations within their code and is somewhat expected between major releases. But, what if addressing deprecations could be automated?

Rector, in its simplest format, is an automated, fancier "find and replace" tool for deprecations. It tries to identify the correct symbols and conditions to perform substitution within a library of rules. Each rule has two primary parts: what it is seeking to replace (statement, method, function, argument, and more, represented as a class) and a processing method to perform the replacement. Each rule returns the object with the proper replacement. This processing is great for nuances tied to deprecations, like optional arguments. Rector’s underlying tooling has fine-grained symbolic control that allows for the highest degree of tuning around deprecations.

Did you know that Drupal is able to automate updating deprecated code? Did you know that these updates represent the bulk of the code development work required to upgrade from Drupal 8 and 9? Did you know that this approach works for core, contributed projects, and for custom code? And your help is needed.

This blog post demonstrates how to create and run Rector rules, with some tips along the way. If we all paused from our hectic lives and made just one Rector rule, we could hit a really high percentage of refactoring (note: it may be impossible today to do this kind of automation and hit a flawless 100%). The future goal of this should be upfront time savings with a follow-up manual review and testing to ensure the automation worked.

Background

In my undergrad, we learned about compilers, particularly Lex and Yacc. In grad school, we leveraged symbolic execution for vulnerability analysis. I didn’t find either experience all that satisfying. Both represent a broader umbrella of symbolic computing, with different intended purposes. In short, the code in programming languages can be broken down into its smallest, atomic parts, line by line, part by part.

Let’s break down a simple statement, $number = 8;. A variable in PHP starts with a "$" character and the context of the variable can change based on the location of a variable. The "=" sign refers to the assignment operator, where the variable set is to the left of the equals sign. "8" represents a literal value, not generated like a function. All of this is considered a statement, which is terminated with a ";" character.

The key takeaway is that even a line of code that is trivial to type is a series of symbols with specific semantic meaning tied to the program language itself. Now, imagine expanding this into conditions, functions, objects, and much more.

Getting Started with Drupal Rector Development

These steps should serve as a high-level walkthrough to getting started. Development best practices are still being established, and I encourage joining the #rector Slack channel on drupal.slack.com to get help, offer feedback, and coordinate contribution.

Step 1: Getting setup

The Palantir.net team has set up a sandbox project with instructions on how to rapidly set up an environment for Drupal Rector development. The development instructions follow most Github standards, where a fork of the Drupal Rector project is created to observe the standard pull request workflow.

To identify a deprecation to work on, check the consolidated list of deprecation notices in public modules here (filter by category "not covered by rector" and click on the "total occurrences" column for a sense of impact). Mention the deprecation on the rector channel on Drupal slack for discussion and check that someone hasn’t already filed the issue on the drupal.org issue queue. If no one has worked on it, file an issue in the issue queue and assign it to yourself for the unassigned deprecation you wish to work on.

Step 2: Mock up code and analyze

It is important for you to mock up code examples for all variations that demonstrate both the old (has deprecations) and the new (remediated) code. Yes, you should do both. Why? Drupal Rector dependencies ship with an analysis tool that shows you how statements are composed. It breaks down a statement into a hierarchy of parts. Running the analysis tool against the old and new code helps you identify differences.

A specific example file for Drupal::url can be found here, found in a file named DrupalURLStatic.php. To run the analysis tool against that file, as an example:

vendor/bin/php-parse web/modules/custom/rector_examples/src/DrupalURLStatic.php

Step 3: Develop the Rector Rule

3.a. Creating the Rule

Create a new rector rule class within this directory. Check existing base classes for some helpful starting points that will simplify the logic within your rule. When possible, extend a base class, otherwise, extend the AbstractRector class from the Rector utility. Please bear in mind that if a base class exists, this likely automates a lot of the processing. You may not need to follow every step defined below, but I capture the basic steps you need without a base class.

Begin with a getDefinition() method that provides a human readable description and simple before and after code sample.

Next, create a getNodeTypes() method. Running the analysis tool in step 2 against both the old and new code samples offer guidance on how to create the remediated code returned from the processing in the rule. The generated output of the analysis tool will help highlight exactly what development changes are required to get an old code sample to become the new one. For efficiency and best practice, your starting point should be the highest order difference between the old and new code. Are you replacing a whole statement? Are you replacing a function or just an argument? Within this function, return an array of matching class types tied to the highest order class. This will send all instances of this class as an argument to your processing function when the Rector engine executes. Classes are represented as one or more PHPParser node class types. Please note, this is not a Drupal node, but a PHPParser base class representing the breakdown of the PHP language (e.g. functions, objects, statements, etc).

Finally, create the processor by creating a refactor(Node $node) method. It’s common in this method to further refine the conditions of the passed argument. For instance, if you pass a deprecated class method like getEntity, the argument will receive a class of type ClassMethod. A simple condition to check the name attribute of this instance will refine the processing further.

Proceed to create the new desired code hierarchy in the refactor method from the differences between the code samples. As an example, if you only need to change the name of a method, replace the name attribute in the passed argument and return the object at the end of the processing. Processing differences can be drastically more complex, but encouraging the use of base classes and patterns should lower the complexity over time.

An example of a Rector rule

3.b. Register the Rule

Rules should be associated with a deprecation. A deprecation associates to an issue that identifies what major/minor version of core that introduced the deprecation. Any rules that are created from the deprecation need to be registered in the correct config file found in this directory. If there is not an associated config file for the major/minor version, create a new YAML file with the class name for the rule. Add this new YAML file to the project’s YAML file and the new YAML file referenced in the 'all deprecations' YAML file.

The rector.yml configuration file

An example configuration file that registers a Rector rule

Step 4: Running and Validating

Double check your YAML configuration to ensure the rules are properly registered.

To see a dry run of the rector engine processing, use the following to process all of the rector examples:

vendor/bin/rector process web/modules/custom/rector_examples --dry-run

This will help to debug and iterate on your new rector rule. Running this command will display code that matches Rector rules and code that does not. Hopefully the matches are the old code that needs to be updated to the new code. The new code should come back clean.

A partial example of a log output

Remove the --dry-run flag and replace the path to actual code to have Drupal’s rector engine perform code refactoring.

Summary

Automating deprecations can be complex but it is high impact work. It should become our new muscle memory as we continue to deprecate code in the future. Imagine handing developers a tool capable of automatically performing maintenance to code indefinitely. We can get there.

I want to thank the following people for reviewing and contributing to this blog post: Dan Montgomery, Ofer Shaal, Ken Rickard, Dries Buytaert, Gabor Hojtsy.

admin Tue, 05/05/2020 - 12:53

Tags

Drupal 9 Deprecations with SimplyTest.me

admin — Thu, 05 Mar 2020 00:07:23 +0000

Drupal 9 Deprecations with SimplyTest.me

Drupal 9 readiness is as easy as cleaning up deprecations for your Drupal 8 project. SimplyTest is well positioned to help with this.

Deprecation Reporting

You can generate a report of the deprecations by performing the following steps:

Go to simplytest.me
Under “Enter a project name:”, start typing the name of your project and select it
Select the relevant 8.x branch (not release, as this will also scan unreleased commits)
Expand “Additional Options”
Click “Add Additional Project”
Start typing “Upgrade Status” and select it
Click “Launch Sandbox”
Go to “admin/reports/upgrade-status”
Select your project
Click “Scan”

This will generate a list of the deprecations. Do not close the modal or the browser tab, as we will use it later.

Filing an Issue

If you are testing a Drupal.org project, file an issue against that project for the deprecations report. Upgrade Status will report back for an existing issue, but please double check that an existing issue does not exist. I used the following steps to submit the report:

Go to the project page and click “file issue”
Select the appropriate 8.x branch
Issue title: “Drupal 9 Compatibility”
On your reporting tab, click “Export as HTML”
Copy the HTML source containing the table of deprecations
Paste this into the issue description
Click “Save”

Remediation

Currently, the only way to get a completely remediated Drupal 9 project is to manually address each item found in the report. From personal experience, addressing deprecations is a drastically simpler exercise than performing a Drupal 7 to 8 port. This is primarily because Drupal 8 was a big shift off the island and now is concerned about dependencies. Architecturally, Drupal 8 changed a lot of APIs, introduced Composer, changed paradigms (e.g. procedural to object oriented), and leveraged modern design patterns. Drupal 9 leverages basically the same architecture as 8, but changed major versions primarily to remove deprecated functionality.

Again, while the work is manual, the task should be relatively straightforward code refactoring. And, there is good guidance. Many of the items listed in a report also link to the relevant drupal.org issue for the deprecation. This is useful to identify how to properly remediate the deprecation and offer relevant context.

Automated Remediation

I expect deprecations to be the new norm as dependencies end-of-life or changes are made to core with minor releases. We can come to expect more frequent major releases that follow the same paradigm as 8 to 9. But, wouldn’t it be cool if this could be automated?

The Upgrade Rector project built by Gábor Hojtsy, based on the drupal8-rector project created by Dezső Biczó at Pronovix, aims to do just that. It aims to build off of the Upgrade Status module by automatically remediating deprecations. Each deprecation maps to a specific rector rule. These rules map deprecated code to their updated equivalents.

Palantir recently wrote a blog post on the Upgrade Rector project. They are actively looking for contributors to work on rector rules in their Github project. As of now, only one rule for “drupal_set_message” is completed, which is why this process is not yet fully automated. We all need to help get this up to 100%, if possible. Many thanks to Palantir and this blog post discusses the project.

Future State

Why stop there? Move the Upgrade Rector into core. New deprecations in core should mean new rector rule(s) are created and should be a gate in releasing a new deprecation in core. Build infrastructure into drupal.org to periodically scans for these deprecations, files an issue against the project if new deprecations are discovered, post patches to fix them, and fire the automated tests. While I still envision manual testing would be needed to validate the results, this basically would help keep modules maintained automatically. The beginnings of this discussion are happening here.

Resources

admin Wed, 03/04/2020 - 19:07

Tags

SimplyTest.me release welcomes TugboatQA, Centarro, and Linode

admin — Sun, 06 Oct 2019 21:19:32 +0000

SimplyTest.me release welcomes TugboatQA, Centarro, and Linode

On the evening of September 13th, SimplyTest.me launched a major new release of the product. The product release achieved three major goals:

Replacement of the backend with TugboatQA
A new one-click demo framework
A new development environment

All efforts could not have been possible without the generous sponsorship from TugboatQA (a product created by Lullabot), Centarro, and Linode. And, community time offered from Hook 42.

The rest of this post outlines the release and its significance.

TugboatQA Backend

How did we get here?

The backend infrastructure of SimplyTest.me was one server running as a single point of failure. A bulky, monolithic server running a LAMP stack and concurrent virtual hosts/databases/etc. The system had no viable backup, no load balanced failovers (a money problem, not a technical problem), and no affordable replacement.

Community conversations were held to advise on an open source replacement. Many agreed on a panacea that included the use of Docker and Kubernetes running on a cloud-based provider. This gave end users cleanly isolated instances and an infrastructure capable of rapidly provisioning and terminating instances through well-adopted, right fit technologies. This could have been an impactful contribution back to Drupal, too.

While this approach would have been great, who would pay for it? Rough numbers showed that even with the use of tiny spot instances through AWS, we still would have paid at least a few thousand dollars a month for our free-to-use service. Significantly more during major events, like DrupalCon and camps. And, no money or sponsorship to change course. Furthermore, we would have needed to do a ton of technical work across the full stack to achieve this goal. Since the current site still needed to be maintained, this approach would have certainly been a challenge.

Fears had been growing around the sustainability and maintainability of our underlying infrastructure. The backend system repeatedly would go down and over a decade of technical debt had built up. The divergence of Drush 8 and 9 and support for past versions of Drupal also complicated things. The issues were becoming more and more complex. I needed to solicit the help of experts like Elijah Lynn and Greg Boggs, while trying to respect their available time.

Matt Westgate and I connected at DrupalCamp NJ 2019. In 2018, he introduced me to TugboatQA. But, at the time, the lack of an API held back any integration possible with SimplyTest.me. That changed by 2019 as the product matured. This was welcomed news, but I maintained a degree of skepticism that a QA-based DevOps tool could be leveraged for the full set of needs. I maintain the opinion that SimplyTest is more difficult than people initially acknowledge, regardless of people’s good intentions to make suggestions or offer advice.

When we revisited our conversation, SimplyTest.me just opened our OpenCollective but still had very little funding raised. I had a lot of technical questions for the Tugboat team around features and we discussed their high-level approach to achieve my goals. It seemed possible. We exchanged some high level statistics on usage and I finally asked how much it would cost. Their team graciously agreed to pay for our backend. Game changer.

While we had a long road to build out this replacement, the generosity of both Lullabot and TugboatQA to pay for the backend solved a huge hurdle and may have saved the future of SimplyTest.me. It ended up meeting our goals for a new backend. Concerns were raised from the community about not using an open source service/product. This type of situation raises awareness around the harsh realities people like me face when trying to sustain open source communities and run a service with limited fundraising, mostly volunteer led efforts, and minimal corporate backing. I’ll blog about this topic later, but both Dries and Lullabot wrote excellent articles about related topics.

How did we approach this?

The best engineered solutions are ones that are iterated through what is learned. We made two major architectural passes to iron out design decisions.

TugboatQA is based on the concepts of previews. Previews are created from branches on repositories, where new code in the repository could be deployed on a preview for quality assurance. Previews are configured by a set of scripts defined in a YAML file on that branch, similar to a Docker Compose file. SimplyTest.me instances get code from external repositories specific to what a user selects. If SimplyTest were to use a repository, it could only leverage the YAML configuration since the code would be pulled elsewhere.

We started with this approach to create previews from branches. SimplyTest.me, still running Drupal 7, leveraged PHPTemplate to create dynamic preview definitions in YAML. The YAML would change based on what SimplyTest users selected to test. Definitions could clone Drupal, select a version, pull down projects, load patches, and perform installations. This generated YAML was pushed to a unique branch in a repository, where each branch represents the desired submission. All git related details were made configurable and leveraged a Github-based PHP project to properly respect environments and not push code with credentials. Once the branch existed, the Tugboat API was invoked to create the preview through Tugboat’s downloadable CLI tool. I personally ported a Backdrop TugboatQA module into Drupal 7 and posted it on Drupal.org for others to benefit from. This module is a simple utility and abstraction for access tokens, logging, and invoking Tugboat commands uniformly.

After the first pass at the architecture, the process was generally working. We had scaffolding for Drupal 7 and 8, projects, patches, distributions, and installation logic. The primary issue was performance. It took a lot of time to create each preview. Each provision performed similar, preliminary steps: it created the containers for the infrastructure, loaded things like PHP-APC and PHP extensions, cloned Drupal, installed Drush, etc. TugboatQA had already solved this in their platform with a feature called Base Previews. In short, there are one or more branches that held the YAML definition of cached previews. SimplyTest was able to extract common scripts for Drupal 7 and 8, copy base previews, and apply the preview-specific configuration in the preview’s own specific branch. This sped things up significantly.

AmyJune Hineline led testing efforts to make sure we maintained feature parity from the old to the new backend. We iterated and worked through a defined set of feature-based issues that she created until we stabilized the new system. This methodical approach to testing with a set of issues tied to features will help us when we look to launch Drupal 8 in the not-too-distant future.

While we still have some known gaps causing failures, like executing make files for Drupal 7 distributions, we were comfortable launching the new backend and remediating issues as they arise.

Key Outcomes

This new backend has already had significant impact on the project and the service offered to users.

Radically simplified systems

Removal of the old backend server
Removal of HAProxy logic
Removal of backend SSL certificate
Empowered the SimplyTest web application to define and control previews created

Drastically reduced the amount of supporting code that built up over years

Removal of legacy bash scripts that were retrofitted for newer versions of Drupal
Replaced the custom web application to backend interaction with a cleaner, decoupled approach: shared repository and web service API

Offered users drastically better stability

Removal of maintaining highly customized legacy servers that were difficult to maintain/update with updated technologies and changes potentially impacted the entire service, not just specific instances
Isolating failures to either service provider outages and/or gaps in the SimplyTest application, not a bespoke, highly customized backend server

Rapid changes are now possible

New instances are not tightly coupled to other running instances
Simplification opens up new opportunities for contributors to participate without having the concern of bespoke infrastructure setup.

Acknowledgments

I want to thank TugboatQA (Matt, James, Ben), Jonathan Daggerhart (daggerhart), AmyJune Hineline (volkswagenchick), and Hook 42 (giving me contribution time and sending me to camps that I used to do this work and collaborate/sprint on this effort).

One Click Demos

How did we get here?

Drupal distributions have been a significant issue for SimplyTest. Drupal 7 had stronger opinions on distributions, primarily around the use of specifically-named make files. Drush enabled scripting covered most of the use cases to build distributions programmatically. But, loose dependency definition and inconsistent use of standards still caused many issues -- many of which community members looked to me to help triage and support after SimplyTest would fail to create instances due to issues that were introduced.

Drupal 8 still feels like the wild west for consistencies between distributions, primarily caused by how core initially adopted Composer (which is evolving and may be worth revisiting for impact to distributions). A simple comparison between popular distributions like Lightning, Commerce, and Contenta demonstrates my point. Each distribution would likely require custom considerations for SimplyTest to work properly.

Ryan Szrama approached me to see if SimplyTest could be used to serve as the demo framework for Drupal Commerce (specifically, the Belgrade demo). This presented an opportunity to improve SimplyTest, raise funds, and explore paid contributions (which was a goal of mine outlined during the vision/roadmap presented at Drupal Camp Asheville 2018). Ryan agreed to sponsor both the underlying architectural changes to enhance SimplyTest and the specific implementation for Drupal Commerce.

Side note: it is inspiring to see how Ryan has established a sustainable business primarily around an open source product that routinely gives back to Drupal. Centarro continues to find creative ways to give back even more.

How did we approach this?

While our primary goal was to launch a one-click demo feature for Drupal Commerce, SimplyTest is a tool that serves all people and it would have been a wasted opportunity to not build in the capability for additional one click demos. We leveraged a custom hook to allow submodules to register a one click demo. This behaves similarly to the plugin system in Drupal 8. Submodules then invoke the hook, which references a YAML definition. It is expected that the definition supports other SimplyTest features for patches, loading of other projects, and more. Base previews still need to be defined manually and managed through the repository, for now. SimplyTest then loads the registered one click demos and displays each as a button through the web UI.

We shipped this release with support for both Drupal Commerce and the Umami demo. We welcome others interested.

Key Outcomes

Created a new feature that made the creation of Drupal-based demos with one click
Created an extensible framework that solved a long-term difficult issue with distributions and allowed other distributions to be easily rolled in

Governance

I still have not fully established guidelines to help with the governance of this new feature. I believe it’s important for businesses that sponsor SimplyTest to be eligible for their open source distribution to be delivered as a “one click demo”. However, completely open source demos, like Umami, should be eligible too. Fundamentally, I believe it’s important to focus on widely adopted cases around Drupal 8. Maintaining every distribution across every version of Drupal seems like a challenge for this approach. While there will be some discretion, I promise to work with the community in good faith to roll this out for future potential needs.

Development Environment

Given the fact that we were making a significant change, we were able to revisit the creation of a development server from my last attempts over new years (TL;DR - way too difficult). In many ways, the new backend simplified the overall systems. A new dev server was imperative for our testing efforts to launch the new backend. The legacy backend server represented the most complex part of setting up a development environment. And, this was now going away. This made the task a lot more simple to accomplish.

Linode offered SimplyTest sponsorship money to use their infrastructure. We were able to leverage these funds to run a Linode for the web front-end during the entire time we were developing the integration for the new backend. A very basic LAMP stack was created and the Drupal database and files were cloned from the production Drupal application. This unblocked the involvement of others to help test and prepare for the new release.

I thank Linode for giving us this assistance and I continue to support them in other endeavors. We requested and just received additional funding for continued support of the development infrastructure.

Conclusion

When I transitioned to lead the project, I knew I had challenges ahead of me. I knew the backend needed modernization. I knew I wanted more people and companies to help participate in the sustainability of the system. I knew we would need funding to open up future opportunities. I had ideas but no clear path to move it forward. I invested as much time as was needed to get to this day and it wasn’t even possible without the help of others and the new, generous support of TugboatQA, Centarro, and Linode combined with contributed time from Hook 42. I’m glad we got here. The work is not done. But, I feel much better about where this is heading now that we have this major milestone behind us. I also continue to be grateful for the largely volunteer-led efforts from community members that helped in all capacities.

admin Sun, 10/06/2019 - 17:19

Tags

A DevOps Primer

admin — Tue, 30 Apr 2019 02:06:10 +0000

A DevOps Primer

I just closed about 100 browser tabs from an early year activity. While it’s embarrassing I left those tabs open so long (going on five months), I wanted to leave them open to reflect on what I learned. And, what a better way to reflect than a blog post.

The Soapbox

Bear with me for a moment. I subscribe to the philosophy that DevOps represents a line of thinking and a way of working, not a technical product. Like security, when someone says, “I’ll sell you DevOps”, it’s a scam. For those “doing” it right, you need to focus on foundational concepts built into continuous learning. This ideology has been communicated in talks and blog posts by some of our communities’ sharpest minds, including Michelle Krejci, Jeff Geerling, Mark Schwartz, and more. And, in my opinion, there is a direct correlation to Agile frameworks. Concepts like retrospectives to share learnings, sprint planning to prioritize, and backlogs to organize and track work thoughtfully. Most of these ceremonies are open to the whole team to participate, so they naturally advocate for transparency and connection.

Key point: DevOps is ongoing evolution through experience. Technical progress is just one type of outcome, but equally important are solutions involving people and process.

DevOps Philosophy

Change is at the heart of DevOps. Digital products change. People change. Organizations change. DevOps reflects a pragmatic approach to change. Many people discuss it, especially solutions. But, any solution reflects a change that is, hopefully, driven by what is learned and what can deliver better, more impactful results for actual people and real problems encountered.

As a basis, start by evaluating what change occurs. On the surface, change can look like any of the following:

System and software updates, especially security ones
Continuous Integration (deployments of change increments, automated/manual processes)
Changes in the team (people leaving, people being promoted to new roles)
Organizational changes (launching a new product, leadership transitions, exploring new markets)
Incidents (getting hacked, services going down)
Exploring new technology
Sunsetting legacy technology

A mature DevOps practice is capable of adapting to change ongoing and remaining effective. An equally important philosophy is the openness and willingness to change. This is built on trust and the recognition that every team member is capable of bringing value: to allow people to raise their voices, drive change, and have impact.

DevOps can be transformational. It can be really hard and a non-natural fit for the “command and control” models traditionally established with embedded IT departments and IT vendors. Mark Schwartz writes about this concept routinely in blog posts and in his book “A Seat at the Table: IT Leadership in the Age of Agility.” Again, organizations suggesting they can just buy DevOps would be in for a rude awakening or really only purchase an incremental step relevant at that point in time only. Sure, you can purchase a two million dollar Kubernetes, cloud-based infrastructure framed as DevOps. But, what problem are you really trying to solve? Chances are, you’ll be in a position of opening up countless other problems around vendor lock-in, governance, maintenance, and enablement you never anticipated. A better place to start is in the cultural change needed to identify problems, define risk, share ideas and make smarter investments capable of having real impact now and in the future.

Bare Minimum DevOps

Now that I’ve set the context for my thoughts on what DevOps actually is, how do you do it? I’ve long wanted to make a blog post defining some bare minimum DevOps concepts. I’ve seen so many differences when I interact with a client or project. I’ve wanted to point back to something and highlight my thoughts foundationally. The range of differences reflect the varying states of learning organizations are at. And, it’s natural given many of the concepts represent years of learning how to build quality, robust digital solutions that are capable of adapting to change.

Case Study: SimplyTest.me

Remember the browser tabs and the learning I did? I was working on SimplyTest.me. As of right now, it is a perfect example of a system desperately needing some DevOps love. There has been a lot learning and needs are defined. As the project lead, I’ve long wanted to automate routine failures, establish a more easy-to-change system, and open up SimplyTest.me for broader contribution. But, the current system is very hard to do so as it’s developed. The infrastructure is highly customized, the tool is fairly complex, and, subsequently this makes it harder to change. This may make for a future blog post, but the concepts shared reflect this recent exploration for SimplyTest.me and things I’ve encountered over the years.

Environments

Change cannot and should not happen only on production systems. Any change needs prepared, tested, and then deployed to prod. At a minimum, you need both production and non-production environments. Change increments then get vetted through non-production environments before being released to production. A non-production deployment helps vet any changes made on a local system before they hit production.

Parity

Parity is a key concept for evaluating change. As an example, if you are running different versions of PHP across environments, evaluating a change increment on one environment may not match the results of what gets deployed to production.

Evaluation Criteria

If you change things, you need to establish a means for evaluating the change. This could be both manual or automated testing of a change. It could also be part of your development workflow on what needs to be tested when implementing a change. But, the key is you need to have some process defined. And, this process needs to respect both the change increment itself and some means of looking at regressions throughout a system. Because, any change made in one part of the system can unknowingly impact the rest of the system. Establishing some criterion to evaluate change may be just as important as the change itself.

This is not confined to technical change either. When making process or organizational change, there should be some metrics and data used to measure efficacy. After all, if you are making a change, how do you really know if the change worked?

Source of Truth

Also known as the “system of record”, a source of truth is a data integrity concept. Some aspects of environments are more flexible to change than others. Most production systems maintain data (users, content, files, etc) that most likely should not change as change increments are deployed. And, non-production systems leverage production systems as the source of truth, often by synchronizing or replicating a “point in time” snapshot of the data to mirror production data before a change is evaluated. The source of truth is left untouched until the change is ready and often throughout a deployment of a change. Third party systems may be the source of truth for other enterprise functions. For example, an email server may be the source of truth for notifications sent from a system. Recognizing the differences and responsibilities of specific systems and environments help understand source(s) of truth and how they serve a larger change process.

Backups

Even with the best of intentions, changes can break things. Be prepared at any point to restore from a backup, both by rolling back changes and by restoring databases. Test this process (really… it needs to work in the most inopportune of times). Having timely and relevant backups are hugely important for restoring botched deployments, standing up systems after a security incident, and generally gives people piece of mind. Make sure you understand how to restore a backup and bonus points if both the backup and restoration processes are automated.

Keys

Identity management is a really deep and hard problem in IT. On the surface, people think identities apply only to users. But, systems need a proper way to be identified and subsequently verify their authenticity. Keys are a widely adopted approach. Often there is a public and private key, where the public key is referenced by other interacting systems. The private key is used as the verification and is owned strictly by the system. Maintaining keys is a critical way to ensure only specific systems are capable of accessing and associating a system to another (often through users and their specific permissions).

Secured Channels

When systems are interacting, one needs to be concerned with the channels in which systems communicate. Secured channels help encrypt communication. The most common of which is the use of SSL and certificates. Tools like LetsEncrypt have brought SSL to the masses, allowing for a free way to generate (and re-generate) SSL certificates capable of happening in an automated fashion. This helps mitigate the risk of someone intercepting and browsing your traffic between systems.

Scripting

One key way people apply what they have learned is through scripting things that happen routinely. Scripting also can be as simple as a Bash script to perform a routine operation in a consistent fashion or a robust development framework with conditional scripts executing routinely and on-demand. Those intimidated by scripting can start off basically by leveraging tried and true bash scripts that run non-interactive commands across any environment. You can get a long way without introducing complexities found in more robust frameworks.

Code Repositories

Code repositories are the cornerstone of DevOps. They are the source of truth for all system code, any scripts that run, and can be critical in any disaster recovery/restoration process. Code repositories maintain a record of changes and the users who made the change. Code repositories can be cloned to any environment and fetched as needed (the “D” in “DCVS”). Concepts like branches allow for changes to be staged before being merged into a branch associated with production environments. Releases can be implemented as tags that reference specific commits. And, many popular tools (Github, Gitlab) offer features, like pull/merge requests, that are advantageous within a development workflow.

Also, code repositories maintain change events. Creating a new branch may provision a new environment (this has been made popular by CI/CD models) that allow for hands-on testing of changes before release. Merging a branch can destroy an environment and trigger a deployment to a production or non-production environment. And, tools like pull/merge requests allow for code to be reviewed. All of these events are exposed to a broader opportunity that DevOps can leverage. For example, running automated tests can occur when commits are pushed up to a branch or when a deployment to a non-production environment occurs. None of this is possible without a code repository.

SSH

When events occur and scripts need to be run, how do they run on the actual systems? SSH is the answer. SSH is the protocol used to remotely execute commands from one machine to another. I remember the days where every company had huge in-house server closets with expensive terminals to manage the servers. Now, we can purchase affordable infrastructure managed remotely, upload keys, and open terminal sessions that are directly connected to these remote machines. SSH has made much of this possible and is critical for DevOps.

Jenkins

Continuous integration (CI) tools exist to centralize and execute what you have automated. I questioned if this needed to be on the “bare minimum” list. Inevitably, you can do a lot without such a tool, but you won’t get the mileage you could otherwise. CI tools do a great job of logging execution and maintaining a history of execution. Imagine developers having to save their terminal history. Such tools help teams have a central platform where multiple users can log in, access can be control, parameters can be selected to use common scripts, and more. Tools like Jenkins, Travis CI, and CircleCI have native integration with code repositories, can help manage keys and servers, and can manage complex, conditional workflows.

Systems Infrastructure

All of the aforementioned concepts rely on an infrastructure that runs the systems. At a minimum you need at least one server for production and seperate server(s) for non-production infrastructure/environments. Backups should be maintained offline and outside of the servers.

Infrastructure as Code

While it’s not “bare minimum,” container-based, cloud technology can create a flexible infrastructure capable of provisioning and tearing down servers when change events occur. This maintains a durable, extensible infrastructure capable of responding to change. Tools like Docker can help create the definition and images needed for containers. Cloud-based platforms have APIs for creating instances as needed, managed volumes for persistent storage (logs, databases, objects/files), and monitoring needed for auto-scaling. Production infrastructure, or specific aspects of it, should be persistent. Load balancers and proxying become common when implementing this type of infrastructure, including DNS management. Again, while this may not be completely necessary, it’s certainly an appealing option for those building something from the ground up or people looking to level up an existing infrastructure.

Documentation

This may seem commonsensical, but there must be documentation. At a minimum, I look for a README in the code repository, describing how to set up a project, a technical architecture overview, a description of the infrastructure, and a brief rundown of what is found in the code repository. Non-technical users often need Google Docs or something outside of the technical repository. The most common things I’ve seen are flow charts articulating the DevOps workflows, diagrams of the systems, and high-level commentary on insight and motivations. After all, changes in teams or members on teams can be caught completely flat footed without good documentation.

Summary

As I shared, I think DevOps is sometimes a misrepresented topic because I believe it starts with a way of thinking, is often associated with transformation, and is not strictly about technical outcomes. I’ve seen variations in maturity and tried to share what I consider to be a bare minimum set of concepts people need to both practice and apply DevOps. I can’t imagine the number of browser tabs I’ve opened over the years researching this topic. Again, it’s simply not enough to purchase a product and expect the potential outcomes. What do you consider DevOps to be? What set of bare minimum concepts did I miss?

admin Mon, 04/29/2019 - 22:06

Tags

The Season of SimplyTest

admin — Fri, 18 Jan 2019 19:56:53 +0000

The Season of SimplyTest

Last year was spent primarily learning about SimplyTest. We did make some progress, but I think “keeping the lights on” for a system of this complexity was quite a feat after the project transfer. It’s a unique and fairly complex endeavor that bridges all elements of an open source project, a completely free service, and underlying infrastructure. I see all of the good and the bad that comes from each aspect: system maintenance, feedback from community members, customer service (Slack, Twitter, etc), system outages, and more. I recognize how valuable this service is to the community and I strive to offer the best service possible.

I serve as the SimplyTest lead. I do most things and do not have a team. I steer both the current system (primarily for stability) and maintain the vision to help modernize, simplify, and scale SimplyTest with time. I balance my time by prioritizing impact, knowing I work full time and have a family. There are times progress stalls given time constraints and this seems to mirror much of the concerns people raise around a perceived “stagnant” system.

There isn’t vibrant additional contribution to any aspect of SimplyTest. But, I do want to recognize and thank Greg Boggs for his help, especially with system administration. Iris Ibekwe, who developed a new Drupal 8 theme for our future front-end. Several members of my team (Jonathan Daggerhart, Ryan Bateman) at Hook 42 have helped start build out aspects of the Drupal 8 front-end with me. I welcome help and also need to do a better job of asking for it.

There are little to no finances tied to SimplyTest. We are fortunate to have server-level sponsors (shout out to Maloon, Druid, and Linode), but we must make use of free services/tools, pay personally out of pocket, or (very recently) get reimbursed through donations. All labor has been volunteered and some sponsored by Hook 42 (thank you so much!!). I personally have paid for a new design and logo, stickers, domains, and more, long before the Open Collective was established.

At the end of the year, I often take time to work on outside things (e.g. professional development or community work). 2018 was no exception. I had a full backlog of items I wanted to tackle with SimplyTest and wanted to share the last two months of progress.

Development server

Given the underlying complexity of the various distributed systems involved, it is incredibly hard for people to set things up on their own. This hurts contribution.

Thanks to a generous donation from Linode, we received credit on their platform for new infrastructure. This allowed us to set up both a front-end server and a worker server to be used for development. While this is quite close to being finished, this is a huge milestone for SimplyTest to ensure we vet changes on non-production systems and foster an environment that is safe to make changes such that others can learn the system to contribute.

Clean up procedures

The setup of the new development servers required snapshots and images of the underlying infrastructure containers. Just the action of creating these caused the system to go down due to a lack of storage. This uncovered an underlying issue where spawned instances were not being properly cleaned out and the subsequent images were massive.

Processes were developed to manually clean these out. Based on an increase in usage, the system subsequently went down several other times. This was then scripted and moved into daily cron. We are actively monitoring the disk utilization to ensure these processes hold and stabilize the system.

Continuous integration

A brand new Jenkins server was set up to afford opportunities to automate tasks in and between servers. Attempts were made to automate the creation of the development servers. But, there were many complications encountered, like the previous wildcard certificate used for the instances that was not scripted through LetsEncrypt. The storage issues ultimately prevented the automation from moving forward and the amount of technical complexity to make this a reality with the current system is likely not worth pursuing a complete solution. However, it’s nice to have this tool in place to move forward and maintain a platform for learning and rolling in automated improvements.

New SSL for the web frontend

During the creation of the development servers, the scripts leveraged to generate SSL certificates through LetsEncrypt were executed on the development servers multiple times due to both networking issues and updated configuration. Cleanup was required to remove some failed past attempts on the production system tied to certificate renewal. All of this work required the LetsEncrypt service to be invoked multiple times in a development-only capacity.

Sadly, this effort caused us to hit the LetsEncrypt rate limit. Within a few days, the SimplyTest.me SSL certificate expired (at the most inopportune of times). While it required funding, this ended up being an opportunity for us to purchase and place a more permanent SSL certificate to minimize maintenance moving forward (LetsEncrypt has quarterly certificates). This was our first use of the expense system in our Open Collective to reconcile and publicly share the expense.

Monitoring

Given the expansion of infrastructure, it’s imperative to have monitoring in place. As a start, we’re leveraging the free offering of Uptime Robot, which now covers outages for all production, development, and CI infrastructure. There are opportunities over time to expand this for storage thresholds, proactive SSL certificate notifications, SSL expirations, and more.

Expanded donations

We’ve not done a great job of marketing our Open Collective because I believe firmly that people need to be informed of where SimplyTest is heading and can recognize what incentives come with their donations (aside from the service running).

There is a lot of strategy involved and many ideas I wish to explore in the coming and future years (e.g. infrastructure for a modernized system and a paid internship to help those underrepresented in tech). However, several people in the Drupal community and the Drupal Association, has joined as backers. I can’t thank you all enough. I welcome others to join as backers and will work to outline these opportunities and vision moving forward.

Conclusion

I’ll continue to invest my time to make SimplyTest the best it can be. It’s a labor of love but one of the most rewarding and unique contributions we have in our community. I couldn’t be more confident in where this is heading and look forward to many vibrant years ahead for the service.

admin Fri, 01/18/2019 - 14:56

Tags