Drupal

I presented The Drupal 10 readiness initiative - here we go at DrupalCon Europe a month ago. While I published my slides with plenty speaker notes right away, the session videos just became public. While the live presentation was a month ago, most of the content is still up to date.

Drupal 9 is expected to have the shortest Drupal major release lifetime in recent history with Drupal 10 planned to be released in the middle of 2022 (next year!) and Drupal 9 end of life by end of 2023. In this session, we discussed what it takes to get from Drupal 9 to Drupal 10 and how are we going to manage this transition. We also covered what we learned from the Drupal 8 to 9 transition (so far) and how we plan to make it better for 10.

Check out the recording:

Do you own an existing drupal.org project that does not yet have a Drupal 9 compatible release? This week would be a good time to take that step and make a Drupal 9 compatible release! I am paying for two tickets to DrupalCon Europe for new Drupal 9 compatible releases. Read on for exact rules!

DrupalCon Europe is in two weeks already! December 8-11, 2020. It offers 4 keynotes, including the Driesnote and the Drupal Core Initiative Leads Keynote (that I help coordinate), 119 sessions in five different content tracks, 4 workshops, interest group discussions, and networking. These are all included in the 250 EUR (no VAT) ticket. I would love to see you there!

Looking at Drupal 9 compatibility data, while 83% of the top 1000 projects by usage already have a release, the others do not. If we look at all the projects, 43% of them have a Drupal 9 compatible release. This is way better than with Drupal 8 was at the same time, but we can still do better! 22.5% of all projects (2177 projects in total) need an info.yml file change and a release, no other changes required. There is a good chance one of those are yours!

The rules of this giveaway are the following:

1. The participating project must have existed before this week.
2. The project must have its first Drupal 9 compatible release this week, before end of Friday.
3. Selection from eligible projects is random.
4. The lead maintainer on the winning two projects will pick who gets the ticket.
5. In case of a pass, I draw another project.
6. I will be using my existing script from the #DrupalCares campaign to track the newly Drupal 9 compatible projects.
7. The script uses the official drupal.org releases dump and takes dates of releases from there.

I'll keep this list up to date throughout the week with who is in the running:

Update: Congratulations to winners: Marcelo Vani for the first Drupal 9 compatible release of CSV to Config and Daniele Piaggesi for the first Drupal 9 compatible release of Prevnext.

DrupalCons are a great way to learn and connect, but they are especially great to meet various people leading Drupal's future direction. DrupalCon Europe in a couple weeks is no execption.

There is of course the Driesnote to get an update on where Drupal's progress is and get inspired about where its going. There is a dedicated question and answer session with project lead Dries Buytaert where you can inquire about topics not covered in the keynote.

The initiative leads provide a glimpse into their respective areas in the Drupal Initiative leaders keynote. This is a great way to get to know the leaders and learn more about their plans and where you could help.

Various initiatives have dedicated sessions: the Core Automatic Updates Initiative Update and the Configuration Management Initiative 2.0 session provides updates on the progress made and gives you a look forward. There is not one but two sessions about the new experimental Olivero frontend theme in Drupal 9.1.0: Designing for chaos: The design process behind Olivero and The Olivero theme: Genesis and Update on Drupal 9's Newest Theme.

A practically inevitable future milestone is Drupal 10. In The Drupal 10 initiative, here we go I will cover where we are in preparing Drupal 10, what you can expect and where you can be involved. On the way to Drupal 10 is our support for PHP 8. While not specifically about Drupal core's progress, the PHP 8: What's new and changing will showcase the new version of the programming language.

For the future of Drupal, the people leading and the technology being built is at least as interesting as the platform we are collaborating on. You might have heard that merge requests and issue branches became generally available on drupal.org yesterday (yes!). In Drupal.org Update - The latest collaboration tools to help you build Drupal (panel), you can learn more about the current state and future of the platform we use to build Drupal itself.

Also of wide interest is the effect of end of life Drupal versions: Jeremy Andrews and Mike Meyers will cover Drupal 7 and Drupal 8 end of life.

These were just the pre-planned sessions. The real involvement possibilities are at Birds of a Feather discussions where you can be directly involved with core discussions, see how hard problems are deconstructed and solved. Most BoF slots are still open for submission, but there will be various covering core's future. Also contribution events will happen all week where you can learn Drupal development and any other kind of Drupal contribution based on your interests. I expect contribution topic groups forming around Symfony 6 compatibility, removing jQuery UI components, CKEditor 5, the Claro admin theme and Olivero frontend themes and so on.

These are just the core focused sessions out of 119 sessions offered at the event in five tracks. Also there are even 4 in-depth workshops included with each ticket. Hope to see you there! Check out https://events.drupal.org/europe2020 for more information.

Photo credit Boris Baldinger from DrupalCon Amsterdam.

Drupal 8 and 9 core were already made compatible with Composer 2 back in May. Last week the Drupal package repository (packages.drupal.org) rolled out full support for Composer 2 as well. While Michael Anello did Drupal vs. Composer 2-alpha2 benchmarks in July and Malabya also did an even more detailed benchmark in July, various things have been improved in Composer 2 since July and the packages.drupal.org improvement should also show.

Inspired by numbers posted by Konstantinos Skarlatos from a simple scenario to the #composer drupal.org/slack channel today, I went ahead and attempted to produce a reproducible scenario you can also try at home. I included all commands to copy-paste. There are no external dependencies other than a PHP executable on your command line and curl to download files as used. For reference, my PHP is as follows (definitely not the latest):

$ php -v
PHP 7.3.11 (cli) (built: Jun  5 2020 23:50:40) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies

Set up the latest Composer 1 and 2 locally

First let's grab the latest Composer versions locally. This will avoid any confusion as to which version is used when.

$ curl https://getcomposer.org/composer-1.phar --output composer-1.phar
$ php composer-1.phar -V
Composer version 1.10.13 2020-09-09 11:46:34

$ curl https://getcomposer.org/composer-2.phar --output composer-2.phar
$ php composer-2.phar -V
Composer version 2.0.0-RC1 2020-09-10 15:39:45

Install Drupal 9 with create-project

Make sure that we are not using cache and ask composer to profile itself. This got me two Drupal 9.0.6 copies, again to separate our testing environments properly.

$ php composer-1.phar --no-cache --profile create-project drupal/recommended-project composer1site
[...]
[9.3MiB/41.02s] Memory usage: 9.35MiB (peak: 197.23MiB), time: 41.02s

$ php composer-2.phar --no-cache --profile create-project drupal/recommended-project composer2site
[...]
[12.5MiB/18.29s] Memory usage: 12.48MiB (peak: 13.71MiB), time: 18.29s

The time is cut in half but the peak memory usage had an even bigger drop to less than 7% of the Composer 1 memory peak with Composer 2.

Adding and removing a Drupal module

You only do site creation once for a site. Let's add and remove a Drupal module! Konstantinos tested with Metatag which sounds like a good one to do. While Metatag only has Drupal project dependencies, the savings are likely similar for projects with 3rd party dependencies.

$ cd composer1site
$ php ../composer-1.phar --no-cache --profile require drupal/metatag
[...]
[629.4MiB/65.35s] Memory usage: 629.36MiB (peak: 1557.29MiB), time: 65.35s

$ php ../composer-1.phar --no-cache --profile remove drupal/metatag
[...]
[450.9MiB/47.98s] Memory usage: 450.85MiB (peak: 1377.67MiB), time: 47.98s

$ cd ..

$ cd composer2site
$ php ../composer-2.phar --no-cache --profile require drupal/metatag
[...]
[13.7MiB/3.13s] Memory usage: 13.67MiB (peak: 14.23MiB), time: 3.13s

$ php ../composer-2.phar --no-cache --profile remove drupal/metatag
[12.1MiB/0.18s] Memory usage: 12.1MiB (peak: 12.49MiB), time: 0.18s

Ok, that is pretty jaw dropping. Adding Metatag goes from 65 seconds down to 3 seconds. Removing Metatag goes from 48 seconds to 0.18(!) seconds. Memory usage goes down to 1% (one percent!), from a gigabyte and a half to 12-14 megabytes.

What does this mean for Drupal?

Given the explicit support on drupal.org, you should be able to use Composer 2 for your Drupal site builds whenever you are comfortable, maybe already in the RC stage or later on as it gets stable. Assuming any composer plugins you need are compatible. Drupal sites are often using cweagans/composer-patches, which is Composer 2 compatible in the dev version, but did not have a release yet. You can change your composer.json to have "cweagans/composer-patches": "1.x-dev" and do a composer update to get the development version that is already compatible.

Many are not using composer at all to build their sites. However, an increasing number of drupal.org projects require external Composer dependencies to function. Based on last week's data from Ryan Aslett from the Drupal Association, 12% of drupal.org projects directly have a third-party Composer dependency. Another 6% have a Drupal dependency that in turn has a third-party Composer dependency. So even if we only look at direct and one-level indirect dependencies, 18% of Drupal.org projects require Composer to be installed properly. I also looked at various segments of projects and the ones requiring Composer are very well distributed. 15.2% of the top 500 projects (by usage) require Composer while 16.2% of the top 5000 do. So if you are not yet using Composer to build your Drupal sites, it seems to be only a matter of time. The improvements in Composer 2 should help a lot with its adoption I think.

These improvements will also be crucial for the Automated Updates initiative which has Composer 2 compatibility on their roadmap as well. Signature verification is only possible with Composer 2 and the lower memory and time requirements allow it to be run from web requests.

Finally, the downloadable tarballs of Drupal core are built with Composer 1 currently to make it easy to transition from a tarball based site to a Composer site. Allow Drupal 9 (including dev builds) to use Composer 2, part 2 is open to move that to Composer 2. Unfortunately a sizable blocker to that is Composer's "prefer-stable" setting cannot be relied on to produce a stable release. That would need more discussion and a solid resolution to move forward. There is less than 3 weeks before Drupal 9.1 is locked down, so your feedback there would be more than welcome!

Last time I posted an update on Upgrade Status was four months ago. It is fair to say the Drupal contributed project landscape has changed quite a bit in the meantime and Upgrade Status should evolve too. Why? The primary role of Upgrade Status is to help get your site components updated for Drupal 9 compatibility. Most of your site components are contributed modules. In many cases, either your local copies or the remote available updates will already be compatible. 38% of Drupal 8 compatible modules are now Drupal 9 compatible (3535 out of 9258) and most others have a patch waiting to land to either improve or complete compatibility.

The changing role of Upgrade Status

Therefore the role of Upgrade Status for contributed projects is more to make you realize that you are one of (a) already compatible (b) will be compatible with an update or (c) should work with the maintainer on existing issues. As opposed to scanning for problems locally and trying to fix it for yourself. But the current 2.x version of Upgrade Status does not do the job of indicating most of these. Without running the (lengthy) scans on your projects, it does not know which local projects are already compatible. And although it has access to information about whether remote updates are Drupal 9 compatible or not, it will not display them. Look at this screenshot:

We only know the Consumers project update is going to solve your Drupal 9 compatibility issues because the maintainer kindly explained that in their Drupal 9 plan. This requires manual work and is not going to happen universally (eg. as shown below neither Chaos Tools nor Markdown specify a Drupal 9 plan). For projects like Consumers, there is not even a good reason to run our lengthy compatibility checking locally because we know the remote update is available and is going to be compatible. Plus we already know from the local info file that the local version is not yet compatible and requires that update. So if we would provide all this "new" data, the user could understand the situation and decide to not even run a compatibility scan but update the project. So the first step (hopefully) improving the Upgrade Status experience was to add more raw data about local and remote Drupal 9 compatibility. Then it looks like this:

The new "Local 9-ready" column shows if the project is already ready locally (without scanning the project, based on info files in the project). The new "Drupal.org 9-ready" column shows if the remote update is Drupal 9 compatible or not (based on information already provided to core's update module). So what we can see from this example is that Chaos Tools will be entirely resolved with an update, no need to run a problem scan locally. Markdown will not be entirely resolved but is worth updating either way because likely there will be less issues. Not all projects will specify a Drupal 9 plan to go by as discussed above. But if there was no plan specified, we could still link to the issue queue. I don't think applying issue tags consistently is possible to expect from all contributors, so we cannot link to an issue tag search. The link goes to a drupal.org issue search for "Drupal 9" issues in the project. This is the fastest way to get to a place to collaborate with the maintainers of Markdown, given no Drupal 9 plan posted.

Deciding on the next step for each project

So more raw data is great, right? Well, it probably makes the display a lot more confusing. Based on what you read out about the data you would do different things to different projects. For the 2.x version Joe at Drupalize.me published a great post about how to read the data. But what if the module would make suggestions for next steps instead of you needing to carefully examine each part of the data. We could still give users the source data, but save a lot of time to compare data points as to whether a project should be updated or needs more work, etc. Enter the next step suggestion engine I built following extensive discussions with Ofer Shaal. The possible next steps for projects are:

1. Remove: if you have uninstalled projects on your site, there is likely no point in investing effort keeping them up to date.
2. Update: if your project has a remote update, that is the best course of action; even if Drupal 9 compatibility is not entirely done there, it is likely improved.
3. Collaborate with maintainer: contributed projects that are up to date could still have issues; instead of fixing locally, there is high probability that there are issues (including but not limited to from Project Update Bot) in the queue, so better start collaborating instead of fixing locally.
4. Scan: unless your custom project's info files specify Drupal 9 compatibility already, the suggested next step is to scan them
5. Fix with rector: for custom projects there is a 40+% likeliness that there is at least one problem we find that is rector-fixable, we suggest you use rector as your next step to avoid needless manual work (again) in this case
6. Fix manually: if you scanned and there were no rectorable fixes, as a last resort we suggest you fix issues manually

Instead of grouping by custom/contributed and under that installed/uninstalled, the 3.x branch now groups results based on the next suggested step. The raw data is still shown and users can still do whatever they decide if they do not agree with the suggested next step.

This is the state of the module's user interface right after installation. We can already deduct a lot of information from local Drupal 9 readiness, update information and status of projects. For the single custom project in this test setup, we can only move forward with scanning it for problems. The numeric counters of problems found overall were removed, because in the current state of contributed projects rapidly getting ready, counting their individual issues together does not really inform about anything useful.

Overview of status of your site

Usage of Upgrade Status has always been a three-step process. First you need to help the module gather data (run available update checks, run scanning of projects). Second, you would attempt to fix the problems found (update modules, run rector, etc.). Then you scan again and verify compatibility improved. So instead of putting this into documentation, I designed a summary view for the top of the page, that helps you go through the data gathering, the compatibility fixing and the celebratory steps. Here is how that looks like for my test site:

1. The data gathering steps ensure the module works with the best data possible. Having a way to accurately identify project versions is important, so projects are suggested for that (unless already installed). Having up to date available updates information is important, so it is possible to run that directly from here. There are projects where scanning on this UI is the next step to gather the required data (to move to "Fix with rector" or "Fix manually"). On this site I have one of those test modules. And finally, the UI indicates you can scan either project as you see fit.
2. Based on the data available, the second columns shows all the next step suggestions for fixing compatibility issues. Projects to remove because they are uninstalled anyway, projects to update, collaborate on, fix with rector, and so on. If your environment is incompatible that also shows up here with a link to details.
3. Finally there is a column to review what's ready to go. If your environment looks all good, that would show up here, as well as all the projects that are already compatible. A little circle chart shows how far along you are.

Comparing side by side

Check out the current user interface on the left and the new work in progress on the right. A summary of differences:

1. The current UI is almost a blank slate and is not aware of project status locally or update's compatibility remotely.
2. On the current UI to gather data you need to run the full scanner on everything. Then you need to decide on next steps based on your reading of the data.
3. On the current UI the available updates refresh and the suggestion to improve version identification with the deploy modules is sort of hidden under "help text blindess", rather than being an active part of the UI.
4. The new UI is actionable immediately with categories of projects by next step.
5. The count summaries of problems found is gone, the focus is more on the projects rather than the individual problems. (Even though that data is entirely accessible).

What do you think?

I believe that more than two months after Drupal 9's release, the role of Upgrade Status is increasingly not to tell you to do fixes locally but to allow you to obtain the fixes already made and collaborate on fixes being developed for contributed projects. And for custom projects, it should suggest you the most automated way forward (rector preferred) that will result in least work to execute. These were the guiding principles of the new user interface. I built it all in the 3.x branch of Upgrade Status, and you can try it out in 8.x-3.0-alpha1. There are definitely spots for improvement and there are probably things I still did not consider. Feedback would be great either as comments here or support requests on the issue queue. Thanks for your input, it is going to help make this better!