Drupal

Dwayne McDaniel did some thorough reporting of deprecated code use in all Drupal 8 contributed modules in March. Ultimately this kind of reporting would be best to have on drupal.org but while that is figured out, Dwayne's data set provides a very nice way to mine data about Drupal 9 readiness of contributed modules and to inform our tooling to improve the process. His original numbers showed that almost 44% of contributed modules had no deprecated code use at the time. What I was interested in was how to help the rest of the 56%.

Dwayne created an updated process and a new repository this week with fresh data. I was still curious so I delved right into the data and started mining it. A key question I was interested in is how much of the most widespread deprecations are actionable right now. An actionable deprecation is something core deprecated in a previous version that is not supported anymore, so you can update your code to remove the use of that API. Currently Drupal 8.6 and 8.7 are supported, so deprecations there should only be acted on for your custom code. However deprecations in and before 8.5 are entirely fine to act on.

First I counted the top list of deprecated APIs used from Dwayne's data across all of contributed projects. Then I wrote a script to collate api.drupal.org documentation to the deprecation notices. Ideally phpstan itself would report these messages directly and Matt Glaman is working on that. However since that is still blocked on the phpstan side, one needs a different data source to find the deprecation documentation for each occurrence, so I took to api.drupal.org to get that for now. Once that is found, we can categorize the deprecations into actionable, not actionable and actionable for custom code only. For the later case you know which core version you are using, and that should be an up to date minor version. So you don't need to deal with what core branches the community supports otherwise.

The results look really promising so far in terms of how much contributed modules can make progress on even today. If all already actionable deprecations get resolved, there will be very little left at least of the deprecations we already know.

A month ago Dezső Biczó created a set of proof of concept Rector fixes to automate some of these deprecation fixes, so I opened an issue with this new data set to try and cover the top ones that are not just actionable but approachable to automate.

Image

As with all interesting data sets, this summary is just the tip of the iceberg. There is huge potential to mine this data set for other uses, such as finding modules that potential contributors at an event could contribute fixes to. I don't know yet how much I can continue to work with this data myself, and of course others doing analysis of their own would be more than welcome.

Are you a drupal.org project maintainer? Now would be a good time to fill in your Drupal 9 porting information in your project, so you can let contributors know how to best engage with your in the process towards Drupal 9.

Disclaimer: The data is based on the state of contributed projects on May 20, 2019 based on Drupal core's 8.8.x development branch on May 20, 2019. The lookup on api.drupal.org was not perfect and I found some bugs there that are being resolved as well so the data gets more accurate. Also, as contributed modules will get updated, there will be less uses of deprecated APIs. As core will introduce more deprecations, the data could get worse. There may also be phpstan Drupal integration bugs or missing features, such as not finding uses of deprecated global constants yet. This is a snapshot as the tools and state of code on all sides evolve, the resulting data will be different.

Update as of March 16, 2020: an entirely new version of these slides are now available, see http://hojtsy.hu/blog/2020-mar-16/whole-new-version-open-source-state-d…

I am about to present about Drupal 9 at DrupalCamp Belarus in May and then at Drupal Developer Days Transylvania in June . I already presented an Acquia webinar with Dries Buytaert on the topic, and was on the Lullabot Podcast discussing Drupal 9 with Angie Byron and Nathaniel Catchpole. I am a firm believer that this know-how should spread as far and wide as possible. I should not be needed to travel around the globe to present the topic and people should not spend the same time again to redo slides for their local presentations. There is no intellectual property here to hide, as many people should be aware and excited and participating as possible. The topic should be presented at Drupal Meetups, Camps, and inside your own companies. So the natural next step for me was to create an open source slideshow.

I took all that we learned from the webinar and Dries' keynote at DrupalCon Seattle as well as new technology that emerged since then. I also used a free slide template and Google Slides so you can make a copy for yourself and add your own contact information as well as edit the slides down to shorter or longer timeslots. The 51 slides in my test run for about 35 minutes, leaving 10 minutes for discussion in a 45 minute slot. You would likely need to cut content for shorter sessions. There are only basic buildup animations, so if you need to present offline that is also an option. Edit in your contact/introduction info and export and present as PDF.

The 1.0 version of the slides have been presented by Christian Fritsch at DrupalCamp Munich last week and I updated some content to the current 1.1 version as it is available now. I'll keep updating slides based on all your feedback. I shared the slides with public comments allowed, so keep the feedback coming there, comments here or some other way you can get ahold of me.

Resources to watch/listen to learn more include:

1. Dries' State of Drupal presentation from DrupalCon Seattle
2. Lullabot Podcast on Drupal 9
3. Acquia Webinar on Drupal 9

Thanks to Acquia for funding me to create this slideshow and thank you for presenting it!

Dries Buytaert recently published a great post on how to prepare for Drupal 9. He explains how we build Drupal 9 in Drupal 8 using deprecations and the tools to use to detect use of deprecated code. Acquia funded me and Zoltán Herczog to work on one of the tools in the past few weeks. Zoltán just released the second alpha of Upgrade Status. It is definitely worth a try!

Here is how it works:

Traditionally Drupal was developed in two parallel branches. Take the example of Drupal 7 and 8. There was the supported stable branch of Drupal 7 that received bugfixes only and there was a development branch of Drupal 8 where developers went free to change and improve things the way they have seen best. As the development of Drupal 8 was nearing release, contributed module and theme authors were asked to take a look and release new versions as well in preparation. While the new branch was built with great care, it was not proven yet in various production environments. It also often took quite some time for contributed modules and themes to update.

Image

Traditional Drupal code vs. module compatibility

Drupal 9 is planned to be only 18 months away now, wow! It is already being built in Drupal 8 by marking APIs to be removed in Drupal 9 as deprecated and eventually upgrading some dependency version requirements where needed. Once the Drupal 9 git branch will be open, you will be able to test directly against Drupal 9. That should not stop you from assessing the compatibility of your module with Drupal 9 now. To prepare for compatibility with Drupal 9, you need to keep up with deprecated functionality and watch out for upgraded dependencies (when we know which are those exactly). Of these two, automation can go a long way to help you keep up with deprecated APIs.

(Disclaimer: Dries did not ask/order/suggest/request me to post this neither to make any changes whatsoever.)

I was reading Drupal Confessions with great interest, because my primary job for quite a while now is to help enable the best efforts in the Drupal community to work together to reach new heights. I am really privileged to be able to do this and make a living out of it, it is the job I always dreamed of. I also hope that I am being useful to the Drupal community in doing so. I kept nodding in agreement with the open letter's statements on diversity so I could not get rid of my cognitive dissonance on their connection with reality for several days now. The open letter says "We ask you to fight for us, Dries, to protect us from intolerance, harassment, smearing, bullying, and discrimination, no matter why or where it originates from."

As someone who is getting a paycheck at least indirectly from Dries in the past 10 years, and working as directly with him as possible in Acquia's Office of the CTO for the second half of that period, I have the opportunity to look at his practices in terms of diversity. Which would be the most observable group to learn more from if not his personal department where he directly appoints people?

In the past 5 years I've been closely working with Dries as a remote employee in the Office of the CTO at Acquia, which consisted of 15 people at most at any given time. In this time I worked with at least the following types of people directly in this group (in random order): blind, stutterer, person with ADHD, person with serious sleep problems, person with even more serious sleep problems, divorced, gay, bi, lesbian, asexual, polyamorous, gospel singer, BDSM, religiously raised, atheist, clinically infertile, cosplayer, very tall, short, people from Eastern Europe, Western Europe, US and India, native-borns, immigrants, people in offices, people always working remotely, capitalist, socialist, pacifist, people from a military family, adopting parent, transgender, people with tics, people who don't drink alcohol, etc. The teams I worked on usually had relatively high percentage of women in technical roles. My current team is 50% female, and 1/3rd of the department overall is female. This could not happen by accident.

While none of this represent the whole rainbow of humanity, and it could definitely be further improved, our limited group of 15 people already covered outstanding diversity in my view. Also this is a group that cares, we discuss and live through our struggles and support each other on every step where we can. I would challenge many of those signing the open letter to practice this kind of diversity in their teams. I for one really enjoy the varying values that people brought in and am sad that some of those great people have left to pursue other technical challenges.

In this five years, I never experienced that when hiring people either of these things were considered as a negative or that any person was treated or felt treated badly for any personal life matter. Neither that any of the amazing people who unfortunately left and are covered in the list left because of any of those.

Given that I just cannot get over my cognitive dissonance. Who are being convinced of what?

We started regular Drupal usability meetings twice a week almost a year ago in March 2016. That is a long time and we succeeded in supporting many key initiatives in this time, including reviews on new media handling and library functionality, feedback on workflow user experience, outside-in editing and place block functionality. We helped set scope for the changes required to inline form errors on its way to stability. Those are all supporting existing teams working on their respective features where user interfaces are involved.

However, we also started to look at some Drupal components and whether we can gradually improve them. One of the biggest tasks we took on was redesigning the status page, where Drupal's system information is presented and errors and warnings are printed for site owners to resolve. While that looks like a huge monster issue, Roy Scholten in fact posted a breakdown of how the process itself went. If we were to start a fresh issue (which we should have), the process would be much easier to follow and would be more visible. The result is quite remarkable:

Image

Starting with Drupal 8, we decided to make more rapid innovation possible by releasing minor versions every 6 months that may come with new features and backwards compatible changes. Now that we released Drupal 8.1.0 and almost 8.2.0 as well, how did we do? Also what else is possible and what is blocking us to make those moves? What do all the changes mean for how might Drupal 9 unfold?

Dries Buytaert posted last Wednesday The transformation of Drupal 8 for continuous innovation and on the same day I presented Checking on Drupal 8's rapid innovation promises at DrupalCon Dublin. Here is a video recording of my session, which should be good for those looking to get to know Drupal's release process and schedule, as well as how we made it possible to experiment within Drupal core directly with Drupal 8. While I did hope for more discussion on the possibilities within Drupal 8 with the participants, somehow the discussion pretty much ended up focusing on Drupal 9, when it should be released and how much change should it come with.

In my previous post I explained why there will be a Drupal 9 even though we have previously unseen possibilities to add new things within Drupal 8.x.y. Now I'd like to dispel another myth, that initiatives are only there to add those new things.

Drupal 8 introduced initiatives to the core development process with the intention that even core development became too big to follow, understand or really get involved with in general. However because there are key areas that people want to work in, it makes sense to set up focused groups to organize work in those areas and support each other in those smaller groups. So initiatives like Configuration Management, Views in Core, Web Services, Multilingual, etc. were set up and mostly worked well, not in small part because it is easier to devote yourself to improving web services capabilities or multilingual support as opposed to "make Drupal better". Too abstract goals are harder to sign up for, a team with a thousand people is harder to feel a member of.

Given the success of this approach, even after the release of Drupal 8.0.0, we continued using this model and there are now several groups of people working on making things happen in Drupal 8.x. Ongoing initiatives include API-first, Media, Migrate, Content Workflows and so on. Several of these are primarily working on fixing bugs and plugging holes. A significant part of Migrate and API-first work to date was about fixing bugs and implementing originally intended functionality for example.

The wonder of these initiatives is they are all groups of dedicated people who are really passionate about that topic. They not only have plan or meta issues linked in the roadmap but also have issue tags and have regular meeting times. The Drupal 8 core calendar is full of meetings happening almost every single workday (that said, somehow people prefer Wednesdays and avoid Fridays).

If you have an issue involving usability, a bug with a Drupal web service API, a missing migration feature and so on, your best choice is to bring it to the teams already focused on the topics. The number and diverse areas of teams already in place gives you a very good chance that whatever you are intending to work on is somehow related to one or more of them. And since no issue will get done by one person (you need a reviewer and a committer at minimum), your only way to get something resolved is to seek interested parties as soon as possible. Does it sound like you are demanding time from these folks unfairly? I don't think so. As long as you are genuinely interested to solve the problem at hand, you are in fact contributing to the team which is for the benefit of everyone. And who knows, maybe you quickly become an integral team member as well.

Thanks for contributing and happy team-match finding!

Ps. If your issue is no match for an existing team, the friendly folks at #drupal-contribute in IRC are also there to help.

Earlier this week Steve Burge posted the intriguingly titled There Will Never be a Drupal 9. While that sure makes you click on the article, it is not quite true.

Drupal 8.0.0 made several big changes but among the biggest is the adoption of semantic versioning with scheduled releases.

Scheduled releases were decided to happen around twice a year. And indeed, Drupal 8.1.0 was released on time, Drupal 8.2.0 is in beta and Drupal 8.3.x is already open for development and got some changes committed that Drupal 8.2.x will never have. So this works pretty well so far.

As for semantic versioning, that is not a Drupalism either, see http://semver.org/. It basically means that we have three levels of version numbers now with clearly defined roles. We increment the last number when we make backwards compatible bug fixes. We increment the middle number when we add new functionality in a backwards compatible way. We did that with 8.1.0 and are about to do it with 8.2.0 later this year. And we would increment the first number (go from 8.x.x to 9.0.0) when we make backwards incompatible changes.

So long as you are on some version of Drupal 8, things need to be backwards compatible, so we can just add new things. This still allows us to modernize APIs by extending an old one in a backwards compatible way or introducing a new modern API alongside an old one and deprecate (but not remove!) the old one. This means that after a while there may be multiple parallel APIs to send emails, create routes, migrate content, expose web services and so on, and it will be an increasingly bigger mess.

There must be a balance between increasing that mess in the interest of backwards compatibility and cleaning it up to make developer's lives easier, software faster, tests easier to write and faster to run and so on. Given that the new APIs deprecate the old ones, developers are informed about upcoming changes ahead of time, and should have plenty of time to adapt their modules, themes, distributions. There may even be changes that are not possible in Drupal 8 with parallel APIs, but we don't yet have an example of that.

After that Drupal 9 could just be about removing the bad old ways and keeping the good new ways of doing things and the first Drupal 9 release could be the same as the last Drupal 8 release with the cruft removed. What would make you move to Drupal 9 then? Well, new Drupal 8 improvements would stop happening and Drupal 9.1 will have new features again.

While this is not a policy set in stone, Dries Buytaert had this to say about the topic right after his DrupalCon Barcelona keynote in the Q&A almost a year ago:

Read more about and discuss when Drupal 9 may be open at https://www.drupal.org/node/2608062