Chromium release management explained

Some people seem to be confused about the Chromium release management, the weird x.y.z.t versions, the channels, the PPAs… I often receive questions about those subjects from end-users, but also from fellow Ubuntu developers. In this post, I will try to explain and demystify a few things. In order to do that, I also need to cover Google Chrome.

It may be a shock to you but the first thing you need to realize is that there is no such thing as an upstream release of Chromium. Give me some time to explain, it is not as bad as it sounds.

Upstream release management only deals with Google Chrome. As you probably already know, Google Chrome is based on Chromium. So a bit of understanding of both projects is necessary.

Chromium and Google Chrome

Chromium is an open source project, where everyone is welcome to contribute something (code, tests, ideas, bugs..). Of course, a significant fraction of the core developers are Googlers, yet, most of the code they produce lands in Chromium Trunk, and is then usable by everyone. As I will explain later, the Chromium trunk is branched off at a very regular interval to produce various builds.

Now, what about the heavily marketed Google Chrome?

(the new campaign, in the Métro, in Paris)

(or last year, also in Paris, Gare de l’Est)

Chrome is definitely closed source. It is composed of Chromium, plus a colorful branding and a quite obscure list of features added on top. I say obscure, not because it is unknown but because it is often not verifiable. You have to trust what Google has announced at some point in the past, and hope they stick to it. But remember it is a very active project, so the target is moving quite fast and there may be more stuff than you think in it. This article is not about this particular point though, decide for yourself what suits you best. I just would like to say that without Chrome, Chromium wouldn’t be where it is today. Google invests a lot on Chromium because it is the base of both Chrome and Chrome OS. It makes perfect sense for them to do so and as a user of Chromium, I’m grateful for the existence of Chrome and that Google open-sourced it early (as Chromium).

Google Chrome release management

In the ads above, it is obviously the Stable release of Chrome that is targeted.
You probably already heard of some other Chrome builds, living in what Google calls Channels (as in distribution channels). There are a few of those channels available: Stable, Beta, Dev (there is also Canary on Windows).

They look something like this:

They provide different levels of “freshness” depending on how long ago they were branched off of trunk. If you decide to follow the dev channel, you will see new features more often than with the more mature beta and stable channels.
So if you are looking for stability, you should look one way (greener), but if you prefer the coolest/newest/hottest but potentially less stable feature-set, you will look the other (redder) way.
For Linux users, those channels take the form of deb or rpm repositories, hosted by Google. As much as Google would like to auto-update like on other platforms, it’s still a pull model triggered by the user along with the system upgrades.

The versions used for those releases is considered by some as completely random. Is it, really?
In the sources, it is defined in a file called src/chrome/VERSION. For the last stable release, 8.0.552.237, we can read:

MAJOR=8
MINOR=0
BUILD=552
PATCH=237


MAJOR is what users usually refer to, Chromium 5, 6, here Chromium 8
MINOR is almost always 0
BUILD is important. It indicates the branch, derived from trunk. It defines the feature set for the major release.
PATCH is more cryptic, almost totally opaque for outsiders. It represents minor releases within the branch. It keeps increasing

Now, let’s see what the orange dots on the previous diagram really look like by watching what happened in the last few months.

On the left, the table is only a subset of all the versions that trunk received (in fact, simple daily commits with just a BUILD bump in src/chrome/VERSION). The full table has more than a hundred lines for the same period.

On the right, it shows every single version that landed in the 3 channels.

To read this, let’s take as example the last stable release, in green, branch 552 (the third number).

552 was branched off of trunk on Oct 12. The next day, the branch landed in the dev channel, received 4 minor updates in the next two weeks before being replaced by 570.

A few days later, the beta channel received 552 (where it was left off in the dev channel). It cooked in there for 4 weeks (with 3 minor updates published) before jumping to the stable channel (as .215).

Once there, 552 received two more minor updates (security/regressions fixes). As long as 552 was still in the beta channel, it also received those updates (here, just .224).

552 may receive more minor updates in the stable channel but it is already obvious that it will soon jump to 9.0.597.x (x >= 67) as 597 has already been cooking for about a month in the beta channel.

So as a whole, it took about 51 days for Chromium v8 (following branch 552) to first appear in the stable channel after it left trunk (with 5 dev and 4 beta “pre” versions published). That’s a good enough exposure to catch most of the problems pro-actively. So far, the “1 major release every 6 weeks” rule has been respected for Chrome 7 (Oct 19-Dec 2) and is slightly overdue for Chrome 8 (winter holidays?).

Chromium release management

I said in the introduction that there is no such thing upstream. They have enough in their plate with just Chrome. So it is up to each downstream distribution to decide what is best for its users.
In case of Ubuntu, as the packager of Chromium since day 1 (*), I decided to align the Chromium builds with the Chrome releases, i.e. same release date, same version number, same VCS revision (sort of..). The problem was that while Windows already had dev/beta/stable releases, Linux and Mac had nothing. Not to mention that the concept of channels itself was unusual. It took a while to see the first Linux channel appear and it’s only when Linux got its beta channel that I started to submit the aligned releases to Ubuntu/Universe (that was version 5.0.307.x in Lucid in Feb 2010, which took longer than expected to turn stable as it happened after Lucid was released).

(*) I started working on this back in September 2008 when Chrome was first announced, despite the fact that there was still nothing but a testsuite for Linux. I started producing daily builds of whatever was buildable for Linux in trunk in early November 2008. Those builds were in a PPA that quickly got some traction from curious users.

Aligning versions was a challenge in itself, because Chrome releases were popping up randomly without any indication of what VCS branch and revision was used to produce them. It was not a trivial problem because as we have seen before, Chrome is Chromium plus some stuff, and as that stuff is not public, it is not available in the branches, and as a consequence, the final VCS revision is not the same. After a long debate, Google finally offered a (non trivial) way to locate what branch/tag of Chromium was used to produce Chrome. It changed form a few times but eventually, it stabilized to its current form: the omaha proxy. This thing gives the full version of each channel per platform. Once passed to a (google) script called gclient, it is possible to fetch the proper sources from the VCS.

Some packagers prefer source tarballs, which were not available by then. For various reasons, I personally don’t mind using the method mentioned above. It is flexible enough to have exceptions for pulling out only the parts we need, so the result is way smaller than the full tarballs (almost 1GB compressed now), which appeared at a later point in time.

Enough history. Where are we now?

It is now possible to monitor the Chrome releases using that omaha proxy, and produce the proper source tarballs.

Given enough reactivity, it is possible to match the release date too, assuming an efficient enough workflow is in place and capitalizing on the intensive tests performed upstream on the almost identical Chrome builds. The former point, reactivity, could be a problem for security updates, preferring proactivity to reactivity. It improved a lot in the last 6 months, but it’s still on the reactive side so more work is needed here. The latter point assumes the build flags are identical, or at least similar enough, which is the case for Ubuntu, but definitely not for Debian and Fedora as they diverged, pruning features, changing build flags, using system libs, etc, so they have to redo way more QA.

For Ubuntu, the Chrome channels have their Chromium clones, taking the form of PPAs (*). Since I’ve been tracking everything happening upstream closely, I was able to gradually put a system in place allowing me to produce every single build aligned on the 3 channels almost always “0 day”, for all the (still) supported Ubuntu distributions and architectures. That means a lot of builds. Nowadays, it is working so well that most of it is automatic. I still have to do a lot of packaging work but since I also do daily builds of trunk, I encounter and fix the problems there, before they appear in the channels/PPA. Afterwards, it’s just a matter of merging changes downward whenever they are needed. Of course, the non-PPA builds that land in the official Ubuntu repositories are manual, and receive some QA from different people, which I’m really grateful for.
(*) Stable, Beta, Dev and Trunk

That’s probably more than what you wanted to know about this topic. Thank you for reading this far.. I hope you liked it. Please let me know in the comments (or using the Like or Rate features below).