Using Drupal as a collaborative software translation tool

I've intended to announce this development at Drupalcon San Francisco but unfortunately the session on this was merged with a more general i18n session which was coupled by the ash cloud above Europe, so I could not go. Evidently, collaborative software translation is not a mainstream topic. On the other hand, I keep receiving requests of the general applicability of the tools uses about every two weeks, and this interest always amazes me. While the localization server tool used on grew out of needs of the Drupal community, the solutions were architected to be useful in a general purpose software translation environment. While the architecture was there, it was lacking useful UI controls to just run it as a generic software translation tool.

Existing non-Drupal users like the Gallery 2/3 project and the Musescore desktop app utilize custom data connector modules, which the localization server nicely supports, allowing for custom code to gather data for translation. Gallery even uses a custom Localization client port for clients to submit translations to the server, even though their software is not Drupal based. However, translating arbitrary software without writing your custom connector code was not possible earlier.

Thanks to recent changes and additions to the 6.x-2.x version of the Localization server however, people can now set up ready to use translation servers for arbitrary software easily. The built-in functionality supports using Gettext Portable Object files as source files to fill up the list of translatable strings and provides a user interface to set up projects and releases as well as upload the source files. Gettext Portable Objects is a well known de-facto standard and many tools exist to cross-convert to this file format. Here is a quick screenshot tour of the settings involved. I've used the neat Curie theme that I've recently released for these screenshots.

  1. Grab Drupal 6 and the following modules: Localization server (latest Drupal 6.x-2.x development version6.x-2.x-beta2 or later); the Plural formula configurator which will let you easily configure languages and the jQuery Update module's 6.x-2.x latest development release (alpha 1 might also work).
  2. Enable the Localization server and the Localization community UI modules. These will enable Locale and jQuery Update as dependencies. To make it easier to set up plural formulas for your languages, I suggest to enable the plural formula configurator as well.
  3. Add any number of languages you'd like to support via Administration » Site configuration » Languages. With the plural configurator, you'll be able to set up plural formulas for languages (the module gives you examples of most common languages and sets up defaults for languages it knows already).
  4. There are two connector modules included with the localization server. Enable the Localization server for Gettext files connector. This will let you work with gettext .po(t) files as sources.

Ok, now that we have all the modules set up, configure them. A module can provide multiple connectors with possible configuration, so l10n_server requires you to select which connector formats you'd like enabled. In this case, it is just one connector without configuration, so just mark it enabled. This will let you add projects which use this connector.

To start off translating a project, the server needs to know of that project. since we are using a "by file upload" connector, we can/need to create the project manually. Go to the localization server projects administration page and click on the "Add project" tab. Fill in the short name for the project (which will appear in URLs), the name, and possibly the home link. You'll have the gettext file handler preselected already.

Now that you have a project to work with, you can start adding releases. Projects can later be removed or disabled (temporarily or indefinitely). It is also possible to start over (which will remove all data except the main identifiers of the project that we just entered above).

Click the releases link and you'll see we have no releases for this project yet. Well, we need to add one at least. So go to adding a new release, add the version number of your release you are about to provide translation for and possibly a download link for this release. Like the project home link, this will only be used to show to users, not to actually download anything. Upload a .po or .pot formatted source file with text you have translatable in your software. I've made up a very quick three-piece mysoftware.po for this example. Note that the server will not deal with the .po headers in the file, so I just omitted them for this example:

msgid "My software"
msgstr ""
msgid "Welcome to My software"
msgstr ""
msgid "Download translations"
msgstr ""

How you generate .po files from your software depends totally on your API. PHP itself supports the gettext standard function naming conventions, so you can use the standard gettext tools. You can of course translate software written in whatever language with this tool, since .po files are just plain text transporters for string pairs. If you have a list of translatable strings in any format, generating files in the above demonstrated .po format should not be hard.

The server should inform you that all strings are now saved as sources for translation. Similarly to projects, you are able to start over releases (which will delete all related source strings and let you reimport an updated .po file) or delete the release completely.

But instead of destructing what we've already achieved, let's see the translation UI finally. If you go the Translate menu item from the navigation menu, you'll see a quick overview of our one project with one release with 3 translatable strings. I've set up Hungarian as the only language here, but if you have more, you'll see a selection of languages as well. This screen can be spiced up with an instant status report of the translation status, if you go to your general localization server settings and pick mysoftware as the highlighted project. A nice status report with progress bars for each language will show up on the main page.

Once you actualy go to the translation tab for a language, you'll see the list of strings to translate with fields to add new translations. This user interface will allow you to copy source strings and edit translations. The filter on the top of the page can be used to focus on certain sets of translatable strings and is very useful to track down contributions by individuals.

From here, you can use the user interface to translate and collaborate on your software UI. As experience showed this can even be used to take suggestions for changes in original English user interface text. I'd only like to take one last chance to highlight a neat feature of the server which lets you see and compare all outstanding translation suggestions for strings, quickly assessing the difference between them and deciding which one fits with your translation guidelines or style. This fancy live diff works as you type in a new suggestion for a string and also works for all existing suggestions.

There is a lot more to this user interface and the localization server has lots of other cool features. You can connect the server to the Drupal Organic groups module with the Localization Groups sub-module and create separate translation groups for languages with controlled permissions on the group level. You can enable the remote submission API module to allow for remote translation submissions (but as with the Gallery project, you'll need to work on a custom client for this if your software is not Drupal based, because we only have the localization client implementing this API on the client side now). You can track top contributors in the language teams and export the translations as .po files for distribution is whatever form.

These tools proved very useful on, and making them useful for the general audience should hopefully help get more of you on board in using and improving it. We have some exciting things on the horizon including actions integration (where stuff happening in the translation flow will inject short status reports to a status feed) and integration with automated translation tools (eg. Google translate) as well as more sophisticated translation memory (TM) tools. There are still missing pieces in our tools, the Drupal-sepcific export options will still show up when you use the gettext connector only for example, but we are getting pretty close.

Your feedback welcome!


Thomas Zahreddin's picture

Hi Gabor,

i like your plans :-)

Are you aware of ? This project collects translations of some open source projects, but lacks a nice UI.

I also have in plan a „translation training“ module - not a real module, but a big array of often used terms and a small introduction.

Thanks for all your effort in the translation process.

Thomas Zahreddin

Gábor Hojtsy's picture

There are lots of interesting projects in this space from Launchpad's Rosetta through Transifex to all kinds of tools people use. Given the interest in our solutions and the modular and open source nature with possibility to couple with discussions, forums, activity streams, news, releases, poll, etc. Whatever a translation team needs is probably somewhere in Drupal, so if self-hosting is not an issue for a project, they can go to great length if using this system.

zephyrus's picture

Hi Gábor,
first of all, big thanks for this project. It's going to save me a lot of work :-). I was looking for a crowdsourced translation tool for our company's website and Drupal Localization Server fulfills almost all our needs. There's just one thing -- we want our users to be able to vote on translation suggestions. It should simplify and speed up the approval process.

There apparently isn't a module for that, is it? After a quick look at the code, I suppose I'll have to modify the server itself. Is there any work in progress on this?

Gábor Hojtsy's picture

As with other Drupal modules, this one also has its issue queue with things work in progress and merely requested. See - you'll probably notice at least one related issue: Your contributions are welcome so we can make this work best for all not just for your project :)

Bryan Klein's picture

"You can connect the server to the Drupal Organic groups module with the Localization Groups sub-module and create separate translation groups for languages with controlled permissions on the group level. "

Can you provide some details for how this should be done?

I would like each language to have a group, and they can only see and contribute to translations in that language group. Within the group I want two roles, those who can suggest translations and those who can administer translations within the language group.

Guidance on how to connect the dots would be hugely appreciated.

Thank you for your time and this excellent project.

Gábor Hojtsy's picture

What have you set up so far and where are you blocked?

Adam Gerthel's picture

Is it possible to use this for custom modules (both Features and custom site-specific modules) as well?

This is what I'm contemplating:

Extracting po-files from a module (a feature for example) using potx, uploading the .po file to our own i18n_server, translate all strings, and have l10n_update get the .po file from our own i18n_server. Basically the exact same works, but for our custom (and site/project specific) modules.

Gábor Hojtsy's picture

Yes it should work that way. l10n_update already supports using unique servers per module.

Adam Gerthel's picture

Thank you for the quick reply. I'm trying out the server module right now. How are the text groups assigned in .po-files or does it not matter? Since you can now import to, and export from, a certain text group. There doesn't seem to be any relation to this in .po-files?

For example:

Let's say that a module creates a content type with a field upon installation, and the module has been translated to other languages. How does Drupal know in which text group to put the translation? Field labels (for example) aren't translatable unless you install the Field Translation module.

Gábor Hojtsy's picture

Yes, .po files are textgroup independent. Drupal stores translatable strings in their right textgroups, and you can export .po file from them. When you import .po files, you need to manually specify the textgroup to import to on the UI.

Adam Gerthel's picture

But a single module only uses a single .po file. Is the textgroup information stored in the comment for each string?

If so, what happens if you manually import a module's .po file through the UI? You have to specify a single textgroup. If a .po file contains translations for several textgroups wouldn't that clash?

Gábor Hojtsy's picture

I think I already made this clear above. .po files have no information on textgroups. There needs to be at least one .po file per textgroup if you think of working with .po files.

Adam Gerthel's picture

Fair enough. But doesn't POTX extract one single .po file?

Gábor Hojtsy's picture

Yes, potx is only concerned with strings from the source code.

Tim's picture

Thank you for this module ! I'm really interested in using it. I made some tests with it and it seems that when I create a new project release, I can only upload one unique .po file with "Gettext files connector". Is this correct ? Because I have translations split in different .po files which can be used by different modules. Thanks.

Gábor Hojtsy's picture

The unit of translation for the localization server is the string. It collects strings into projects and releases. So the idea of the .po file upload for a project is to allow the server to be used for any kind of software that has releases to be translated. .po files can be generated with a wide range of tools. Internally, the localization server does not care about .po files, it is just used as a transport mechanism. Then after you translate your text, you can get your translations out again as .po files (one .po file for a release).

If you want to work with multiple .po files for a project/release, you can use command line gettext tools to merge them before you import and to merge your output with the individual translation templates to put the translations to their right places. Some scripting would be required to make this work easier.

Eugene's picture

Thank you for your hard work and excellent module!
Have a question about using own connectors to proccessing non-Drupal and non-Gettext files. Could not find anywhere information, how to implement in own connector not only the import feature, but also the export functions (for upload files in the original format)? Or export of data is possible only in the Gettext format , no matter what connector is used for project?

Gábor Hojtsy's picture

There is no special support for writing exports in different formats, but all data is stored in fairly easily understandable database structures (I hope). So you can just use the full power of the Drupal/PHP API to implement such code. It is likely some of the gettext export code can be refactored to support this better.

Eugene's picture

Gábor, thank you for supporting information! Make custom export of course not a problem, so I will do. Thanks again for the useful module!

Add new comment