Upgrade Status

Drupal 10 readiness will be hugely automated, 95% of deprecated Drupal APIs already covered

With the Drupal 8 end of life in a little over two months and Drupal 10's release next year, this is the time of transitions again at Drupal. However, while Drupal 7 to 8 (or 9) was a big move, the transitions from 8 to 9 and 9 to 10 are much smaller and mostly automated.

Drupal 10 is planned to be released in June, August or December 2022 and the tools are getting ready to support that. The two key tools will be the same as the previous upgrade: Upgrade Status and Drupal Rector.

Matt Glaman has been doing amazing work recently in the underlying components of both tools. Thanks to his work on updating phpstan-drupal for Drupal 10 support, Upgrade Status checks deprecated API uses on Drupal 9 too. Since my last update on that, I added reporting of deprecated modules and new system requirements as well.

Ryan Aslett at the Drupal Association built the project analysis job on top of Upgrade Status and that is run on Drupal 9 projects now as well. So we have an idea of the extent of work that will need to be done for Drupal 10 compatibility. I've updated the dev.acquia.com deprecation dashboard to show Drupal 10 results. While projects should not be expected to be Drupal 10 ready at this time, the dashboard helps us prioritise work on certain parts of the tooling to help the ecosystem upgrade.

To support that, Palantir.net has been sponsoring Matt Glaman to also bring Drupal Rector up to date for Drupal 10 readiness. The results are already outstanding! Today, of the 22204 Drupal deprecated API uses identified in Drupal 9 compatible projects. These are green and purple on the chart below. Drupal Rector has automations to fix 95% of them (green). There are a further 5391 non-Drupal deprecated API uses (yellow) including Symfony and PHPUnit deprecated APIs. Those themselves have third party rectors, so the coverage will further improve by including those. That is in the works.

Image showing the current state of Drupal 10 readiness of Drupal 9 compatible contributed projects

The drupal.org Project Update Bot resolves rectorable deprecated API uses (green) and info/composer issues (blue) when posting patches, so this means that it will be able to resolve most deprecated APIs in its suggested fixes already and we expect it will improve more with third party rectors added.

Drupal 10 itself is a moving target, the branch will be open around October/November so the above does not mean that the tools are complete, but we are significantly further ahead this time compared to the Drupal 8 to 9 transition, making the upgrade to Drupal 10 smoother for everyone.

With the Drupal 8 release, we decided upgrades must be easier going forward and thanks to the fantastic work of contributors and sponsors, we are doing it again.

Setting an eye on Drupal 10 compatibility

I presented on the overall status of the Drupal 10 initiative in December at DrupalCon Europe. Then posted an update about the initiative one week ago on the Drupal Core blog.

More recently I worked on making Upgrade Status work meaningfully on Drupal 9 with Andrey Postnikov. Released Upgrade Status 8.x-3.5-alpha1 today to let you test this out. (You may need to use composer require --dev phpspec/prophecy-phpunit to make your phpunit setup complete).

How is this different from prior releases of Upgrade Status? It should run very similar on Drupal 8 as it did before. However prior releases of Upgrade Status explicitly forbid running it on Drupal 9 as the UI was very focused on the transition from 8 to 9. Now the UI elements are adapted and in some cases more general to support running either on Drupal 8 or 9.

Upgrade Status main screen on Drupal 9

Environment requirements are not yet defined for Drupal 10, so that part of the report is intentionally brief. However, we know Drupal 10 will require PHP 8 and the underlying phpstan-drupal library does not work on PHP 8 yet. As a super-important infrastructure piece, I encourage you to sponsor Matt Glaman to make that happen. Even included this link on the module UI.

Should Drupal 9 modules be compatible with Drupal 10 now? Absolutely not! We don't even know the full extent of Drupal 10's API or the final versions of its dependencies. However we do know a lot of deprecated APIs already, some from as far as back as Drupal 8.2.0, and projects can gradually update following those to close the gap with Drupal 10 now.

Part of Upgrade Status results on Drupal 9 for CTools

So running Upgrade Status on Drupal 9 will tell you about .info.yml file incompatibility errors and composer.json incompatibility errors, but I would ignore those for now. The more interesting bits are the already fixable deprecated API uses found that were marked deprecated in Drupal 8 and will be removed in Drupal 10. I installed the top 6 most used modules on a test site, to check the results out and there are plenty you can already help move forward in these modules. Most issues found are related to the legacy deprecated assert methods. It is no problem in Drupal 9 to use them, but those will not be present in Drupal 10 next year. Most have 1-1 replacements, so it would be best to build rector rules for them into drupal-rector and support this transition that way. (Also drupal-rector itself needs to support Drupal 9).

Part of Upgrade Status results on Drupal 9 for Pathauto

I started to work with the Drupal Association to set up a job to figure out the status of Drupal 10 readiness at this time of the 5000+ Drupal 9 compatible projects. (Similar to the deprecation status data we have about Drupal 8 modules tested for Drupal 9 compatibility). It may sound too early as more Drupal 9 APIs will get deprecated for Drupal 10, the reality is we already know a lot of deprecated APIs including those from Symfony and other dependencies. We can use this early data to help prioritise which deprecated APIs to cover with rectors which would allow us to start posting Project Update Bot issues for those that are Drupal 8 to 10 deprecations and are in unsupported Drupal 8 branches. This would allow us to get a headstart on fixing them, automated as much as possible.

At DrupalCon North America 2021 (less than 6 weeks away!) there will be a dedicated Drupal 10 readiness day where you will be able to learn about the state of the new version and will have a chance to work with the lead maintainers for each area to contribute to Drupal core, the tools and readiness of contributed projects.

Evolving the Upgrade Status user interface - a work in progress you can help influence

Last time I posted an update on Upgrade Status was four months ago. It is fair to say the Drupal contributed project landscape has changed quite a bit in the meantime and Upgrade Status should evolve too. Why? The primary role of Upgrade Status is to help get your site components updated for Drupal 9 compatibility. Most of your site components are contributed modules. In many cases, either your local copies or the remote available updates will already be compatible. 38% of Drupal 8 compatible modules are now Drupal 9 compatible (3535 out of 9258) and most others have a patch waiting to land to either improve or complete compatibility.

The changing role of Upgrade Status

Therefore the role of Upgrade Status for contributed projects is more to make you realize that you are one of (a) already compatible (b) will be compatible with an update or (c) should work with the maintainer on existing issues. As opposed to scanning for problems locally and trying to fix it for yourself. But the current 2.x version of Upgrade Status does not do the job of indicating most of these. Without running the (lengthy) scans on your projects, it does not know which local projects are already compatible. And although it has access to information about whether remote updates are Drupal 9 compatible or not, it will not display them. Look at this screenshot:

We only know the Consumers project update is going to solve your Drupal 9 compatibility issues because the maintainer kindly explained that in their Drupal 9 plan. This requires manual work and is not going to happen universally (eg. as shown below neither Chaos Tools nor Markdown specify a Drupal 9 plan). For projects like Consumers, there is not even a good reason to run our lengthy compatibility checking locally because we know the remote update is available and is going to be compatible. Plus we already know from the local info file that the local version is not yet compatible and requires that update. So if we would provide all this "new" data, the user could understand the situation and decide to not even run a compatibility scan but update the project. So the first step (hopefully) improving the Upgrade Status experience was to add more raw data about local and remote Drupal 9 compatibility. Then it looks like this:

The new "Local 9-ready" column shows if the project is already ready locally (without scanning the project, based on info files in the project). The new "Drupal.org 9-ready" column shows if the remote update is Drupal 9 compatible or not (based on information already provided to core's update module). So what we can see from this example is that Chaos Tools will be entirely resolved with an update, no need to run a problem scan locally. Markdown will not be entirely resolved but is worth updating either way because likely there will be less issues. Not all projects will specify a Drupal 9 plan to go by as discussed above. But if there was no plan specified, we could still link to the issue queue. I don't think applying issue tags consistently is possible to expect from all contributors, so we cannot link to an issue tag search. The link goes to a drupal.org issue search for "Drupal 9" issues in the project. This is the fastest way to get to a place to collaborate with the maintainers of Markdown, given no Drupal 9 plan posted.

Deciding on the next step for each project

So more raw data is great, right? Well, it probably makes the display a lot more confusing. Based on what you read out about the data you would do different things to different projects. For the 2.x version Joe at Drupalize.me published a great post about how to read the data. But what if the module would make suggestions for next steps instead of you needing to carefully examine each part of the data. We could still give users the source data, but save a lot of time to compare data points as to whether a project should be updated or needs more work, etc. Enter the next step suggestion engine I built following extensive discussions with Ofer Shaal. The possible next steps for projects are:

  1. Remove: if you have uninstalled projects on your site, there is likely no point in investing effort keeping them up to date.
  2. Update: if your project has a remote update, that is the best course of action; even if Drupal 9 compatibility is not entirely done there, it is likely improved.
  3. Collaborate with maintainer: contributed projects that are up to date could still have issues; instead of fixing locally, there is high probability that there are issues (including but not limited to from Project Update Bot) in the queue, so better start collaborating instead of fixing locally.
  4. Scan: unless your custom project's info files specify Drupal 9 compatibility already, the suggested next step is to scan them
  5. Fix with rector: for custom projects there is a 40+% likeliness that there is at least one problem we find that is rector-fixable, we suggest you use rector as your next step to avoid needless manual work (again) in this case
  6. Fix manually: if you scanned and there were no rectorable fixes, as a last resort we suggest you fix issues manually

Instead of grouping by custom/contributed and under that installed/uninstalled, the 3.x branch now groups results based on the next suggested step. The raw data is still shown and users can still do whatever they decide if they do not agree with the suggested next step.

This is the state of the module's user interface right after installation. We can already deduct a lot of information from local Drupal 9 readiness, update information and status of projects. For the single custom project in this test setup, we can only move forward with scanning it for problems. The numeric counters of problems found overall were removed, because in the current state of contributed projects rapidly getting ready, counting their individual issues together does not really inform about anything useful.

Overview of status of your site

Usage of Upgrade Status has always been a three-step process. First you need to help the module gather data (run available update checks, run scanning of projects). Second, you would attempt to fix the problems found (update modules, run rector, etc.). Then you scan again and verify compatibility improved. So instead of putting this into documentation, I designed a summary view for the top of the page, that helps you go through the data gathering, the compatibility fixing and the celebratory steps. Here is how that looks like for my test site:

  1. The data gathering steps ensure the module works with the best data possible. Having a way to accurately identify project versions is important, so projects are suggested for that (unless already installed). Having up to date available updates information is important, so it is possible to run that directly from here. There are projects where scanning on this UI is the next step to gather the required data (to move to "Fix with rector" or "Fix manually"). On this site I have one of those test modules. And finally, the UI indicates you can scan either project as you see fit.
  2. Based on the data available, the second columns shows all the next step suggestions for fixing compatibility issues. Projects to remove because they are uninstalled anyway, projects to update, collaborate on, fix with rector, and so on. If your environment is incompatible that also shows up here with a link to details.
  3. Finally there is a column to review what's ready to go. If your environment looks all good, that would show up here, as well as all the projects that are already compatible. A little circle chart shows how far along you are.

Comparing side by side

Check out the current user interface on the left and the new work in progress on the right. A summary of differences:

  1. The current UI is almost a blank slate and is not aware of project status locally or update's compatibility remotely.
  2. On the current UI to gather data you need to run the full scanner on everything. Then you need to decide on next steps based on your reading of the data.
  3. On the current UI the available updates refresh and the suggestion to improve version identification with the deploy modules is sort of hidden under "help text blindess", rather than being an active part of the UI.
  4. The new UI is actionable immediately with categories of projects by next step.
  5. The count summaries of problems found is gone, the focus is more on the projects rather than the individual problems. (Even though that data is entirely accessible).

What do you think?

I believe that more than two months after Drupal 9's release, the role of Upgrade Status is increasingly not to tell you to do fixes locally but to allow you to obtain the fixes already made and collaborate on fixes being developed for contributed projects. And for custom projects, it should suggest you the most automated way forward (rector preferred) that will result in least work to execute. These were the guiding principles of the new user interface. I built it all in the 3.x branch of Upgrade Status, and you can try it out in 8.x-3.0-alpha1. There are definitely spots for improvement and there are probably things I still did not consider. Feedback would be great either as comments here or support requests on the issue queue. Thanks for your input, it is going to help make this better!

Upgrade Status April 2020 update and introducing Upgrade Rector

Upgrade Status is growing to be an almost (if not entirely) indispensable tool for Drupal 8 sites to asses and prepare for their upgrade to Drupal 9. With the latest additions of the Drupal core version and environment checklist as well as additional deprecation checking for Twig, deprecated libraries, deprecated theme functions, info.yml files and composer.json files, the results are much more complete. Unfortunately not all of those are possible in drupal-check due to Drupal not being bootstrapped there. With the addition of two drush commands though, this expanded set of results is available for the command line as well, including for CI system integration.

A relatively new development is Palantir's drupal-rector which is amazing at providing automated fixes for an increasing number of deprecated APIs. I also built a UI on top of that called Upgrade Rector to make it easier to propose fixes for your team's custom modules as well as your contributed modules. You should always keep the Drupal 9 plan of projects in mind, and contributed in accordance with their indicated Drupal 9 plan, which is well facilitated if you use Upgrade Rector within the Upgrade Status UI. Check out this demo video where I walk through the latest improvements:

I hope you find the tools useful and continue to contribute to both projects to help make the upgrade path to Drupal 9 even easier.