For the first time, Google provides detailed insights into the management of the internally used Linux distribution gLinux. This is based on Debian’s “testing” branch and is the successor to the previously deployed Ubuntu LTS-based Goobuntu. Above all, Google is still convinced of the rolling release cycle that was established when the change was made, says Google in a blog post.
The company does not reveal how many Linux users are among the Google employees – but it does say that “over 100,000 devices” had to be updated with the new version of Goobuntu for each of the two-year releases of Ubuntu LTS. The two-year release cycles of Ubuntu LTS have led to a number of disadvantages, as Google describes in the blog post. The diverse range of tasks of the devices in the company meant that the “complete adaptation of the computers could be difficult and time-consuming”.
Even if the process was automated with a self-developed tool, there were regularly hundreds of bug reports and support requests from “special cases” (corner cases). A complete update of all systems in the group often took a whole year – before the next update cycle was due shortly afterwards.
Increments instead of big bang updates
When designing the Goobuntu successor, gLinux Rodete (Rolling Debian Testing), the aim was to distribute the workload over time using rolling releases. The main reason for choosing Debian was the relatively large similarity to Ubuntu and the large overlap of the packages used.
While Debian stable also follows a two-year cycle, Debian testing relies on the desired continuous releases; Upstream updates are therefore available within a few days. Google has now settled on weekly updates for gLinux – initially for a test fleet of 1 percent of all devices, and then shortly afterwards for all internal Linux users.
Do you already know the free iX newsletter? Register now and don’t miss anything on the monthly release date: heise.de/s/NY1E The next issue will be about the main topic of August-iX: professional hardware from Refurbisher.
Own workflow system Sieve
The specially developed workflow system Sieve is used for this. For each new version of a Debian package, start a new build. The packages are then brought together in groups and tested together on a complete system installation. If errors occur, check the Debian Bug Tracker and, if in doubt, report the problems yourself. At the same time, the Sieve can also use a number of tricks with a toolbox to identify the source of possible dependency problems using “educated guesses”.
There were a number of advantages with the switch to gLinux: The time and energy that the engineering team had to spend on distribution updates has dropped significantly, or is much less concentrated. The new timing of the updates of the Google Linux fleet to upstream releases also increases the security of the systems because known security gaps are automatically closed much more quickly. Under Goobuntu, much more manual work was previously required for this.
The blog post also recommends others to equip their own distributions with rolling releases: They are still convinced of the advantages of the changeover, which began in 2015 and was only completed in mid-2020 when catching up with the then current Debian variant.