Drupal 8

With six days to go until Drupal 8's end of life (on November 2, 2021), now is a good time to take stock of your Drupal 8 sites' modules. Use Upgrade Status to check for environment and module compatibility with Drupal 9.

While a lot of Drupal extensions are now compatible with Drupal 9, it can happen that you encounter an incompatible extension. In case adopting the extension fits into your plans, that is the best way forward. However, there may be some delay until you can adopt the project and it may not fit into your plans to adopt in the first place. There are a few solutions to use drupal.org projects that are distributed as incompatible on your Drupal 9 sites.

Obtaining the extension

I do not advise to download the extension as a zip or tar.gz file and add it to your project. As I posted yesterday, it is best to use Composer to ensure all Drupal-specific and third party dependencies are resolved. Manually downloading does not ensure this. But composer will not let you download incompatible extensions!

This is where drupal.org's recently added lenient Composer endpoint comes in. Essentially this is a package repository that advertises only incompatible projects, but it exposes them as compatible instead. Using Composer 2, you can add this additional repository on top of the existing drupal.org endpoint with a higher priority. This way you can transparently add packages that are yet incompatible to your projects. As projects get compatible with Drupal 9, they will disappear from the lenient endpoint and their entry on the standard drupal.org endpoint will transparently take over on your setup.

Making code compatible

Now that you have the codebase on your site, you will still be unable to use the project, as it is indeed still incompatible. If the only patch the extension needs is to its info.yml file where it declares compatibility with Drupal 9, the simplest solution may be to install the Backward Compatibility module. If additional patching is required, or you prefer not to use the Backward Compatibility module, you can install and use the cweagans/composer-patches package for Composer and use it to apply the compatibility fix.

Other methods

The above method is not the only solution to use incompatible projects. Prior to the availability of the lenient Composer endpoint, Damien McKenna wrote a detailed blog post about using the git repository of merge requests to add to your site and Benji Fisher wrote a practical guide about that too. Even before those, James Williams wrote about using the base git repository of projects and applying fixes with composer-patches. These methods require a lot more special case entries in your composer file and would require you to manually track if the project became compatible.

The benefit of the lenient Composer endpoint is that it provides one unified source of projects. For projects you use with Backwards Compatibility, they will seamlessly convert to compatible projects on your site as their drupal.org releases become compatible. For projects you use with composer-patches, Composer will report the inability to apply patches when updates become available, where you have an explicit notification that either the project became compatible or conflicting changes were added.

With seven days to go until Drupal 8's end of life (on November 2, 2021), now is a good time to take stock of your Drupal 8 sites' modules. Use Upgrade Status to check for environment and module compatibility with Drupal 9.

Various people in the Upgrade Status issue queue are concerned about the module requiring Composer though. While Drupal 8 or 9 do not require Composer to work, getting any extension installed on your site that depends on third party components is a sizeable challenge without Composer. With Drupal 8 core dependent on various third party components, it was inevitable that a dependency manager will be needed to build Drupal sites. Upgrade Status itself does require various third party components to check for deprecated API uses and even to locate your Drupal site root. So you cannot avoid using Composer to even check for Drupal 9 compatibility.

If you are not using Composer yet, the best case scenario is you use this opportunity to convert your Drupal site to a Composer based solution. If you started your Drupal 8 site from 8.8.0 or later, the file structure is already ready for Composer. For Drupal sites started on prior versions, the move is more work, as explained in the guide.

In case you are not ready to convert your site to Composer yet, but still want to get a Drupal 9 upgrade readiness check, you need to set up a separate, parallel site with Composer just for this purpose. Your existing site content or configuration does not matter for Upgrade Status checks, you only need to replicate the enabled projects on the site you use.

Install Composer first. It is best to use Composer 2 because it will be much faster and will use significantly less memory, but you should have no problem installing Upgrade Status with Composer 1 either. After that, run these commands to create a new Drupal site from Composer, add core developer dependencies and add this module as a requirement.

Then, copy all of your custom and contributed projects to the web/modules, web/themes, and web/profiles folders. (You could use Composer to add at least the contributed ones, but since you are not planning to use this build for anything else, this will be the quickest way). Finally, visit /admin/reports/upgrade-status, and run the report.

This should help you get a summary of where your site is at. I would suggest you plan for converting to a composer based setup and use composer for dependency management going forward. You will only need composer on your development environment and can still use your usual workflow to push your fully built codebase from there.

With eight days to go until Drupal 8's end of life (on November 2, 2021), now is a good time to take stock of your Drupal 8 sites' modules. Use Upgrade Status to check for environment and module compatibility with Drupal 9. While more than 7 thousand modules are now Drupal 9 compatible, there are still various that are not yet compatible. However, most only need a one line change and a release or even if they need more, an automated fix is posted to their issue queue and is tested by the community.

Many of these incompatible modules have been posted by people who contributed their code but don't have the capacity to maintain it. I don't think that is a problem. That they contributed in the first place, and that their contribution is so useful that it is used on live Drupal sites itself proves the value. So I think we should appreciate and celebrate these folks. On the other hand, we need to acknowledge the lack of maintenance and find new maintainers who can bring these projects forward.

There is an existing process for taking over abandoned projects on drupal.org, however the Drupal Association decided to speed up the first steps of this by automatically opening issues in the affected project's queues two weeks before DrupalCon Europe. You can find the over 3 thousand projects that are under this process at https://bit.ly/drupaladopt (open issues tagged with "Project available for adoption"). If an issue has already been closed, that means the maintainer has declined new maintainer help, so please look at the open issues only.

The steps to request co-maintainership are:

1. Comment on the issue explaining why you would like to maintain the module.

2. If the project is opted in to security coverage, confirm that you have previously received security coverage opt-in permission.

3. If an existing maintainer has not commented, move the issue to the Drupal.org Project Ownership queue by editing the 'Project' field on this issue.

4. From there, a Drupal.org Site Moderator will review the issue and grant maintainership if the requirements are met.

Once you gain co-maintainership, make sure to commit the Drupal 9 compatibility fixes and make a release of the project, so people can rely on a fixed codebase for their sites.

You rock! Thanks for keeping a part of Drupal's ecosystem up to date!

In his State of Drupal keynote at DrupalCon Amsterdam, Dries Buytaert showed once again some tools to use to prepare for Drupal 9 including the Upgrade Status module. To me the process is even more interesting than the tools, because it is entirely different from the last upgrade. As I wrote last week, you now make a lot of incremental improvements on your existing Drupal 8 site that makes the gap to Drupal 9 significantly smaller.

It is a new mindset to look at your Drupal 8 site to improve in preparation for Drupal 9 and we have tools to identify those improvements to make. However Dries also mentioned that we are not quite there to automate those improvements. Back in May Dezső Biczó of Pronovix built out a proof of concept integration with Rector that implements a sample set of refactors, including changes for global constants and best effort solutions to some EntityManager uses, a UrlGeneratorTrait and a drupal_set_message() rector. While the extent of impact of the global constant rectors are not yet known due to limitations in our tools not finding them yet, the rest of the implemented rectors are definitely tapping into the top list of deprecated APIs.

Unfortunately slightly after he posted his blog post about the proof of concept, Dezső did not have time for this project anymore. I think this tool could be of huge help in removing deprecated API use in your projects and thus making them more modern Drupal 8 projects (while being more if not already entirely compatible with Drupal 9). However, we need contributors to cover more of the deprecated APIs, especially around file_*() and db_*() functions. If we can figure out rectors for most of the top 15 errors (and Dezső already did some of them), we could cover over half of all deprecated API use in contributed projects:

To top that off, I also think a simplytest.me style online service would be very useful to run drupal8-rector on your drupal.org project and suggest a patch to apply to remove deprecated APIs. Even better if it allows custom code to be uploaded so people can use it for that too. The more we can automate these tasks the easier the transition of Drupal 8 to 9 will be.

Submit issues and pull requests in the drupal8-rector project to improve API coverage. Look for me on Drupal slack if you get stuck and I'll try to help. I'd also love to talk to you if you want to set up an automated service. Let's do this!

Drupal 8.7.7 expected later this week is the first Drupal release to support extensions compatible with multiple major versions of Drupal. This is huge!

Drupal 9 is planned for June 3, 2020 and we are aiming to make the transition as smooth as possible. In fact we build Drupal 9 in Drupal 8 with API deprecations and optional support for new dependencies (Symfony 4.4 is getting very close, while Drupal 8.7.0 was already Twig 2 compatible). Extensions following these deprecations and making sure they are compatible with the updated dependencies could have codebases that are both compatible with Drupal 8 and 9 at the same time, meaning less support burden and earlier Drupal 9 compatibility. In March 48% of contributed projects on Drupal.org were already "Drupal 9 compatible" (as we could detect at the time), and that increased 54% by July (highfive!). These projects will now have one more thing to do, a one line info.yml file addition, to allow their existing codebase to also run on Drupal 9.

✋Drupal contrib maintainers, thank you! You are already contributing a lot to a successful Drupal 9. Projects that are "compatible with Drupal 9" (as much as we know now) are up to 54% from 48% since March. Deprecated API use is decreasing overall.

You are awesome, keep it up! pic.twitter.com/mHpw4ET0P4

— Gábor Hojtsy (@gaborhojtsy) August 2, 2019

The core: 8.x key did not cut it anymore

Drupal 8 looks at extension .info.yml files and checks for the core: 8.x key to check if the extension can be used with Drupal 8. Unfortunately this was not adequate already due to API additions in Drupal 8 minor releases. You could not specify an extension was only compatible with Drupal 8.4+. People devised clever tricks with dependency on certain versions of the system module, but that was still limiting due to inability to depend on specific patch releases. In case your extension depended on a bugfix being present, you cannot set a dependency on Drupal 8.4.1+ in any way.

Introducing the core_version_requirement key

While solving this issue it became apparent that solving it with the existing core: 8.x key is going to cause backwards compatibility problems with existing Drupal 8 sites. The format of the value could not be changed without causing issues, so a new core_version_requirement key was introduced. This now allows to depend on major and minor versions of core as well as specify multiple version requirements even between multiple minor Drupal versions. For example if you would need a bugfix that is to be included in both Drupal 8.7.23 and 8.8.3 (imaginary future versions) but you wouldn't care for the minor version, then you would use the following, making your module incompatible with versions before the particular bugfix you need on both minor branches, as well as incompatible with all older Drupal 8 branches:

Such precision was never possible before!

The new feature is made available in 8.7.7 to help prepare for Drupal 9, as this makes it possible to also declare Drupal 9 compatibility with the existing codebase you already have. Drupal 8.7.x will get security support up until the planned release of Drupal 9 on June 3, 2020. For extensions that only support Drupal 8.7.7 and later you can now exclusively use the core_version_requirement key. If the extension is also Drupal 9 compatible:

For extensions that also support older versions, you should keep the core: key as well. The core_version_requirement key will be ignored by older Drupal versions, and new versions will validate that you specified non-conflicting requirements in the two keys (with regards to Drupal 8 support) so for extensions supporting older core versions, core_version_requirement also needs to be permissive, but could also allow compatibility with Drupal 9 too:

Read more about this change in the change notice. There is also work ongoing to support dependency metadata from composer.json to drive Drupal's dependency resolution as well. That will coexist with this simpler approach and will provide a more composer-native solution for those who already adopted composer for their extensions. In the meantime at least core_version_requirement will limit what Drupal would install while composer.json requirements will limit what composer will download. Because both Composer and Drupal use the term install, but those don't mean the same thing, here is a comparison chart:

What happens to Drupal.org project version numbers then?

So a 8.x-1.3 version of an extension on drupal.org may only be compatible with a subset of core, but that was already the case earlier. Now an 8.x-1.3 version of an extension on drupal.org may also be compatible with Drupal 9. How will this work? Drupal's composer integration already ignores the 8.x prefix from version numbers and only cares about the project specific version. Drupal core already only installs extensions where the core compatibility requirements are met regardless of which version number was the package you used to download (8.x modules may be only compatible with 8.4+ already). So the ultimate plan is to do away with the 8.x prefix in extension numbers entirely on drupal.org and support semantic versions for contributed extensions instead. In the short term, the Drupal.org project browser also needs to be updated to take the dependency metadata and use that to power compatibility filters, instead of merely using the version number prefix.