Drupal 9

I am an initiative coordinator on Drupal 9, here is my writing on tools and processes getting there.

Drupal 9 and Drupal 10 readiness at Drupal Global Contribution Weekend 2022

This weekend (Friday to Sunday) is Drupal Global Contribution Weekend 2022! While there are some in person events such as in Ukraine, there are various online events that allow anyone to join. I'll highlight three opportunities to work on your Drupal 9 or Drupal 10 readiness.

  1. Wim Leers just announced an opportunity to try CKEditor 5 with step by step instructions. He will be available for some office hours on all three days to consult with. CKEditor 5 is going to be the content editor available in Drupal 10, replacing CKEditor 4. So getting ready for it sooner than later is a good idea. A beta experimental version of the integration already exists in Drupal 9.3 to take for a test ride.
  2. Joel Pittet with the UBC Computer Science department is hosting a Zoom event to work on Drupal 9 and 10 readiness from 9am to 5pm on all three days Vancouver time (UTC-8). Folks working on other tasks are also welcome.
  3. I'll be available on Friday 9am to 5pm as well CET (UTC+1) in both the #d9readiness and #d10readiness channels on Drupal slack (https://drupal.org/slack).

Some common tasks that I think are good to take now:

  • If your contributed project is not Drupal 9 compatible yet, it is a good time to do that now. You can use Upgrade Status on Drupal 8 and Drupal Rector to find problems and automate fixes.
  • While your project cannot be surely Drupal 10 compatible yet, most deprecated APIs for Drupal 10 were marked in Drupal 8. You can use Upgrade Status on Drupal 9 to identify problems and use Drupal Rector again to fix most of them.
  • Drupal 10 will require PHP 8 at least, so setting up testing for your project on PHP 8 and/or manually testing on PHP 8 are good ways to make sure this part of compatibility is taken care of. All Drupal 9 core releases support PHP 8, so this does not pose a risk for your existing users on supported Drupal versions.

With my and Joel's events, there will be 2*8 hours of Drupal 9/10 readiness contribution on Friday with an hour of break inbetween. That is followed by 8 hours each on the weekend days as well. Join either to get help on your journey and contribute to Drupal!

Major Drupal configuration schema cheat sheet update - 7 years later

I apparently released the Drupal configuration schema cheat sheet 7 years ago (wow!) to help people adopt the then new format to describe configuration structure. I keep getting questions and requests about it, so decided to make a major update to it now and bring it to the present day for Drupal 9.


Updates from 1.5 (April 2015) include:

  • Double the pages, more extensive examples and pointers to core definitions
  • Includes description of config ordering introduced with Drupal 9.3
  • Documents how to pick schema keys for various use cases, such as config entities
  • Documents where to put the schema files and when to use multiple
  • Explains common string subtypes and recent additions to base scalar types
  • Contains a whole section on translatability
  • An entire new section to explain basics of mappings and sequences, the most common request
  • Explains potential of string keyed sequences 4 or so times, so hope it goes through :)
  • Suggests to use common mapping subtypes like mail and text_format
  • Covers ignore and undefined
  • Adds a whole new intro to dynamic typing to make the examples easier to understand
  • Makes all dynamic examples contain all definitions instead of abbreviations for easier understanding
  • Uses the same data to explain the [%parent] and [childkey] dynamic typing, to show the differences better
  • Cites various common core extension points, such as third party settings on themes and config entities
  • Explains advanced properties such as nullable lists
  • Updates test base class names for Drupal 9

I regularly receive feedback about people using the cheat sheet, so I hope this will help make config schema somewhat easier to grok. Improvement suggestions still welcome!

Kudos to Rene Bakx for his most recent questions that tipped me over to update the sheet.

Setting an eye on Drupal 10 compatibility

I presented on the overall status of the Drupal 10 initiative in December at DrupalCon Europe. Then posted an update about the initiative one week ago on the Drupal Core blog.

More recently I worked on making Upgrade Status work meaningfully on Drupal 9 with Andrey Postnikov. Released Upgrade Status 8.x-3.5-alpha1 today to let you test this out. (You may need to use composer require --dev phpspec/prophecy-phpunit to make your phpunit setup complete).

How is this different from prior releases of Upgrade Status? It should run very similar on Drupal 8 as it did before. However prior releases of Upgrade Status explicitly forbid running it on Drupal 9 as the UI was very focused on the transition from 8 to 9. Now the UI elements are adapted and in some cases more general to support running either on Drupal 8 or 9.

Upgrade Status main screen on Drupal 9

Environment requirements are not yet defined for Drupal 10, so that part of the report is intentionally brief. However, we know Drupal 10 will require PHP 8 and the underlying phpstan-drupal library does not work on PHP 8 yet. As a super-important infrastructure piece, I encourage you to sponsor Matt Glaman to make that happen. Even included this link on the module UI.

Should Drupal 9 modules be compatible with Drupal 10 now? Absolutely not! We don't even know the full extent of Drupal 10's API or the final versions of its dependencies. However we do know a lot of deprecated APIs already, some from as far as back as Drupal 8.2.0, and projects can gradually update following those to close the gap with Drupal 10 now.

Part of Upgrade Status results on Drupal 9 for CTools

So running Upgrade Status on Drupal 9 will tell you about .info.yml file incompatibility errors and composer.json incompatibility errors, but I would ignore those for now. The more interesting bits are the already fixable deprecated API uses found that were marked deprecated in Drupal 8 and will be removed in Drupal 10. I installed the top 6 most used modules on a test site, to check the results out and there are plenty you can already help move forward in these modules. Most issues found are related to the legacy deprecated assert methods. It is no problem in Drupal 9 to use them, but those will not be present in Drupal 10 next year. Most have 1-1 replacements, so it would be best to build rector rules for them into drupal-rector and support this transition that way. (Also drupal-rector itself needs to support Drupal 9).

Part of Upgrade Status results on Drupal 9 for Pathauto

I started to work with the Drupal Association to set up a job to figure out the status of Drupal 10 readiness at this time of the 5000+ Drupal 9 compatible projects. (Similar to the deprecation status data we have about Drupal 8 modules tested for Drupal 9 compatibility). It may sound too early as more Drupal 9 APIs will get deprecated for Drupal 10, the reality is we already know a lot of deprecated APIs including those from Symfony and other dependencies. We can use this early data to help prioritise which deprecated APIs to cover with rectors which would allow us to start posting Project Update Bot issues for those that are Drupal 8 to 10 deprecations and are in unsupported Drupal 8 branches. This would allow us to get a headstart on fixing them, automated as much as possible.

At DrupalCon North America 2021 (less than 6 weeks away!) there will be a dedicated Drupal 10 readiness day where you will be able to learn about the state of the new version and will have a chance to work with the lead maintainers for each area to contribute to Drupal core, the tools and readiness of contributed projects.

Make your drupal.org project Drupal 9 compatible this week for a chance at one of two free DrupalCon Europe tickets!

Do you own an existing drupal.org project that does not yet have a Drupal 9 compatible release? This week would be a good time to take that step and make a Drupal 9 compatible release! I am paying for two tickets to DrupalCon Europe for new Drupal 9 compatible releases. Read on for exact rules!

DrupalCon Europe is in two weeks already! December 8-11, 2020. It offers 4 keynotes, including the Driesnote and the Drupal Core Initiative Leads Keynote (that I help coordinate), 119 sessions in five different content tracks, 4 workshops, interest group discussions, and networking. These are all included in the 250 EUR (no VAT) ticket. I would love to see you there!

Looking at Drupal 9 compatibility data, while 83% of the top 1000 projects by usage already have a release, the others do not. If we look at all the projects, 43% of them have a Drupal 9 compatible release. This is way better than with Drupal 8 was at the same time, but we can still do better! 22.5% of all projects (2177 projects in total) need an info.yml file change and a release, no other changes required. There is a good chance one of those are yours!

The rules of this giveaway are the following:

  1. The participating project must have existed before this week.
  2. The project must have its first Drupal 9 compatible release this week, before end of Friday.
  3. Selection from eligible projects is random.
  4. The lead maintainer on the winning two projects will pick who gets the ticket.
  5. In case of a pass, I draw another project.
  6. I will be using my existing script from the #DrupalCares campaign to track the newly Drupal 9 compatible projects.
  7. The script uses the official drupal.org releases dump and takes dates of releases from there.

I'll keep this list up to date throughout the week with who is in the running:

Update: Congratulations to winners: Marcelo Vani for the first Drupal 9 compatible release of CSV to Config and Daniele Piaggesi for the first Drupal 9 compatible release of Prevnext.

Testing Composer 2 RC1 with Drupal 9: huge memory and time savings; and what does it mean for Drupal

Drupal 8 and 9 core were already made compatible with Composer 2 back in May. Last week the Drupal package repository (packages.drupal.org) rolled out full support for Composer 2 as well. While Michael Anello did Drupal vs. Composer 2-alpha2 benchmarks in July and Malabya also did an even more detailed benchmark in July, various things have been improved in Composer 2 since July and the packages.drupal.org improvement should also show.

Inspired by numbers posted by Konstantinos Skarlatos from a simple scenario to the #composer drupal.org/slack channel today, I went ahead and attempted to produce a reproducible scenario you can also try at home. I included all commands to copy-paste. There are no external dependencies other than a PHP executable on your command line and curl to download files as used. For reference, my PHP is as follows (definitely not the latest):

$ php -v
PHP 7.3.11 (cli) (built: Jun  5 2020 23:50:40) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies

Set up the latest Composer 1 and 2 locally

First let's grab the latest Composer versions locally. This will avoid any confusion as to which version is used when.

$ curl https://getcomposer.org/composer-1.phar --output composer-1.phar
$ php composer-1.phar -V
Composer version 1.10.13 2020-09-09 11:46:34

$ curl https://getcomposer.org/composer-2.phar --output composer-2.phar
$ php composer-2.phar -V
Composer version 2.0.0-RC1 2020-09-10 15:39:45

Install Drupal 9 with create-project

Make sure that we are not using cache and ask composer to profile itself. This got me two Drupal 9.0.6 copies, again to separate our testing environments properly.

$ php composer-1.phar --no-cache --profile create-project drupal/recommended-project composer1site
[...]
[9.3MiB/41.02s] Memory usage: 9.35MiB (peak: 197.23MiB), time: 41.02s

$ php composer-2.phar --no-cache --profile create-project drupal/recommended-project composer2site
[...]
[12.5MiB/18.29s] Memory usage: 12.48MiB (peak: 13.71MiB), time: 18.29s

The time is cut in half but the peak memory usage had an even bigger drop to less than 7% of the Composer 1 memory peak with Composer 2.

Adding and removing a Drupal module

You only do site creation once for a site. Let's add and remove a Drupal module! Konstantinos tested with Metatag which sounds like a good one to do. While Metatag only has Drupal project dependencies, the savings are likely similar for projects with 3rd party dependencies.

$ cd composer1site
$ php ../composer-1.phar --no-cache --profile require drupal/metatag
[...]
[629.4MiB/65.35s] Memory usage: 629.36MiB (peak: 1557.29MiB), time: 65.35s

$ php ../composer-1.phar --no-cache --profile remove drupal/metatag
[...]
[450.9MiB/47.98s] Memory usage: 450.85MiB (peak: 1377.67MiB), time: 47.98s

$ cd ..

$ cd composer2site
$ php ../composer-2.phar --no-cache --profile require drupal/metatag
[...]
[13.7MiB/3.13s] Memory usage: 13.67MiB (peak: 14.23MiB), time: 3.13s

$ php ../composer-2.phar --no-cache --profile remove drupal/metatag
[12.1MiB/0.18s] Memory usage: 12.1MiB (peak: 12.49MiB), time: 0.18s

Ok, that is pretty jaw dropping. Adding Metatag goes from 65 seconds down to 3 seconds. Removing Metatag goes from 48 seconds to 0.18(!) seconds. Memory usage goes down to 1% (one percent!), from a gigabyte and a half to 12-14 megabytes.

What does this mean for Drupal?

Given the explicit support on drupal.org, you should be able to use Composer 2 for your Drupal site builds whenever you are comfortable, maybe already in the RC stage or later on as it gets stable. Assuming any composer plugins you need are compatible. Drupal sites are often using cweagans/composer-patches, which is Composer 2 compatible in the dev version, but did not have a release yet. You can change your composer.json to have "cweagans/composer-patches": "1.x-dev" and do a composer update to get the development version that is already compatible.

Many are not using composer at all to build their sites. However, an increasing number of drupal.org projects require external Composer dependencies to function. Based on last week's data from Ryan Aslett from the Drupal Association, 12% of drupal.org projects directly have a third-party Composer dependency. Another 6% have a Drupal dependency that in turn has a third-party Composer dependency. So even if we only look at direct and one-level indirect dependencies, 18% of Drupal.org projects require Composer to be installed properly. I also looked at various segments of projects and the ones requiring Composer are very well distributed. 15.2% of the top 500 projects (by usage) require Composer while 16.2% of the top 5000 do. So if you are not yet using Composer to build your Drupal sites, it seems to be only a matter of time. The improvements in Composer 2 should help a lot with its adoption I think.

These improvements will also be crucial for the Automated Updates initiative which has Composer 2 compatibility on their roadmap as well. Signature verification is only possible with Composer 2 and the lower memory and time requirements allow it to be run from web requests.

Finally, the downloadable tarballs of Drupal core are built with Composer 1 currently to make it easy to transition from a tarball based site to a Composer site. Allow Drupal 9 (including dev builds) to use Composer 2, part 2 is open to move that to Composer 2. Unfortunately a sizable blocker to that is Composer's "prefer-stable" setting cannot be relied on to produce a stable release. That would need more discussion and a solid resolution to move forward. There is less than 3 weeks before Drupal 9.1 is locked down, so your feedback there would be more than welcome!