glib2.0: Uninitialised memory is written to gschema.compiled, failure to parse this file leads to gdm, gnome-shell failing to start

I built a test package based on mutter 3.36.9-0ubuntu0.20.04.1, and reverted the three commits introduced by LP #1905825, namely:

commit: 92834d8feceeac538299a47a8c742e155de4e6e8
From: Kai-Heng Feng <email address hidden>
Date: Mon, 21 Dec 2020 14:34:43 +0800
Subject: renderer/native: Refactor modeset boilerplate into new helpers
Link: https://gitlab.gnome.org/GNOME/mutter/-/commit/92834d8feceeac538299a47a8c742e155de4e6e8

commit: 097af7ddb381606da74c737859cc92fff72ed417
From: Kai-Heng Feng <email address hidden>
Date: Mon, 21 Dec 2020 14:59:32 +0800
Subject: monitor-manager-kms: Disable CRTCs if there is no monitor
Link: https://gitlab.gnome.org/GNOME/mutter/-/commit/097af7ddb381606da74c737859cc92fff72ed417

commit: 93f3ce3c305571bfc39f6d9e5d221e1b60a920a4 Mon Sep 17 00:00:00 2001
From: Kai-Heng Feng <email address hidden>
Date: Fri, 13 Nov 2020 14:19:26 +0800
Subject: monitor-manager-xrandr: Disable CRTCs if there is no monitor
Link: https://gitlab.gnome.org/GNOME/mutter/-/commit/93f3ce3c305571bfc39f6d9e5d221e1b60a920a4

The testpackage is available in [1]:

[1] https://launchpad.net/~mruffell/+archive/ubuntu/sf311791-test

I provided the testpackage to the affected user, and they installed it in a test VDI on their VMware Horizon environment, and wrote back that it works.

So, it seems to be the CRTc changes from bug 1905825.

The affected user's environment uses Nvidia GRID GPUs, passed into the VDI. They are using the Nvidia GRID 450.51.05 driver.

I have been trying to reproduce this for a few days now. Google Cloud has an option to pass a Nvidia P4 GPU into the instance, and turn on Nvidia GRID, so I have been using that platform.

I have tried with gpu, without gpu, multiple versions of mutter, with and without VMware Horizon Viewagent 7.13, but in each of my runs, I see gdm3 running, and I see the usual processes that get spawned off of it. I have tried multiple versions of the Nvidia GRID driver, available from [2], but they appear to act the same.

[2] https://cloud.google.com/compute/docs/gpus/grid-drivers-table

I think I am missing something, but I'm not sure what. I have tried using the "virtual display" feature with and without, and when it is enabled and I view the screenshot, I just see a black screen with a blinking cursor, no matter what mutter package I have installed.

It could be a virtual hotplug might be needed, but I'm not sure how that can be achieved on Google Cloud, or how it would happen on the affected user's VMware Horizon on VMware ESXi environment.

I will ask the affected user for some more logs, so we can better see what is going on.

I spent some time attempting to reproduce on AWS today, using a g4dn.xlarge instance, which has a Nvidia Tesla T4 GPU, which supports GRID.

I installed ubuntu-desktop-minimal and rebooted, and gdm started fine with mutter 3.36.9-0ubuntu0.20.04.1. I confirmed this by looking at the instance screenshot.

I then downloaded and installed the Nvidia GRID 450.51.05 driver, and rebooted. The system came up, I checked lsmod, the nvidia kernel modules were loaded, and the instance screenshot shows gdm running. I then installed VMware Horizon Agent 7.13.0-17123958 and rebooted, and again, gdm started just fine.

Attached status of gdm3.service, nvidia-smi and instance screenshot.

I wonder if this is GPU specific? or hypervisor specific, since the affected user runs on VMware esxi. I will again ask for more information.

But for the moment, this seems very specific to the affected user's environment, and not a widespread regression.

I wonder if the affected system uses `xrandr` on GDM start? The settings will be lost after a CTRC disable/enable cycle.

I had a discussion with maintainer here: https://gitlab.gnome.org/GNOME/mutter/-/issues/1619

The affected user cannot reproduce the issue anymore. The new mutter packages install fine in their environment, and gdm starts every time. They have since rolled out the new mutter packages to their fleet without any issues.

I'm going to mark this bug as invalid, as we can't reproduce the issue anymore.

Maybe it was a temporary issue with restarting services? I'm not sure.

If the affected user ever experiences the issue again, I will re-open.

Okay, the affected user has now started experiencing the issue again, and was forced to roll mutter back on their fleet.

We managed to get some logs this time, and we have determined what is happening.

When the user upgrades the mutter packages, particularly mutter-common, the libglib2.0-0 hook is called post-install to re-compile the gsettings gschemas into the binary file /usr/share/glib-2.0/schemas/gschemas.compiled

Processing triggers for libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.3) ...

For some reason or another, this fails or errors out, leaving the system with no /usr/share/glib-2.0/schemas/gschemas.compiled file present.

If the VDI was running, and you attempt to start a program, like nautilus, it fails to open, and you see the following in syslog:

Jul 1 09:28:05 hostname kernel: [510232.497360] traps: nautilus[523090] trap int3 ip:7f9118995295 sp:7ffe78b4de90 error:0 in libglib-2.0.so.0.6400.6[7f9118959000+84000]
Jul 1 09:28:05 hostname nautilus[523090]: No GSettings schemas are installed on the system

If you then reboot, gdm and gnome-session both fail to start, and you see a blank screen with a blinking cursor.

If you ssh in and check the logs, it is the same issue:

Jul 1 09:29:06 hostname kernel: [ 13.311371] traps: gnome-session-b[2304] trap int3 ip:7febfac7c295 sp:7ffe75338c60 error:0 in libglib-2.0.so.0.6400.6[7febfac40000+84000]
Jul 1 09:29:06 hostname gnome-session[2304]: gnome-session-binary[2304]: GLib-GIO-ERROR: No GSettings schemas are installed on the system
Jul 1 09:29:06 hostname gnome-session[2304]: aborting...
Jul 1 09:29:06 hostname gnome-session-binary[2304]: GLib-GIO-ERROR: No GSettings schemas are installed on the system#012aborting...
Jul 1 09:29:06 hostname gdm3: GdmDisplay: Session never registered, failing
Jul 1 09:29:06 hostname systemd[1]: session-c6.scope: Succeeded.
Jul 1 09:29:06 hostname gdm3: GdmLocalDisplayFactory: maximum number of X display failures reached: check X server log for errors
Jul 1 09:29:06 hostname gdm3: Child process -2282 was already dead.

You can reproduce the problem pretty easily. Take a started Ubuntu 20.04 Desktop VM and run:

$ sudo rm /usr/share/glib-2.0/schemas/gschemas.compiled

Then try open gedit or nautiuls and see them fail. Then try reboot, and you will meet the black screen with blinking cursor.

I looked at all the changes in mutter 3.36.9-0ubuntu0.20.04.1 and there doesn't seem to be any related to gsettings.

Maybe it has something to do with the gsettings in the affected user's environment? I will try ask for a tarball.

But what we do know, if that when we upgrade mutter, the gschemas.compiled is removed and never replaced, or is corrupted.

Are you sure that's the same bug as the original issue here? The original issue sounded plausibly related to the fix for bug 1905825 as you say.

The diff between:

  ubuntu/3.36.7+git20201123-0.20.04.1
  ubuntu/3.36.9-0ubuntu0.20.04.1

indeed shows no changes mentioning "schema" or "gsettings".

The diff between:

  ubuntu/3.36.1-3ubuntu3
  ubuntu/3.36.7+git20201123-0.20.04.1

does show some changes mentioning "schema" and "gsettings. So are you sure about the statement "the issue does not occur with 3.36.7+git20201123-0.20.04.1."?

It's curious other people are not reporting this problem so I wonder what is unique about the customer's machines. Do they have any PPAs installed?

I fear we might be onto a new separate issue now so reusing this bug might be confusing the situation.

Hi Daniel,

Yes, I am sure this is the same issue that they are experiencing there, and I now believe the issue lies in glib, and not mutter.

When we install mutter-common, it calls the libglib2.0-0 hook to recompile the gsettings schemas.

The customer provided me with a tarball of their /usr/share/glib-2.0/schemas directory, and I have spent the day looking at it.

I deleted all the schemas from a test 20.04 VM, and extracted the tarball of their schemas in place, and rebooted the VM.

From there, the same exact problems occurred. Each program could not load the compiled gschema file, and hit a breakpoint in the glib library.

Jul 2 13:41:04 ubuntu tracker-miner-f[1235]: No GSettings schemas are installed on the system
Jul 2 13:41:04 ubuntu tracker-extract[1234]: No GSettings schemas are installed on the system
Jul 2 13:41:04 ubuntu kernel: [ 13.280095] show_signal: 7 callbacks suppressed
Jul 2 13:41:04 ubuntu kernel: [ 13.280097] traps: tracker-miner-f[1235] trap int3 ip:7fb6202ac295 sp:7fff0d5c7cd0 error:0 in libglib-2.0.so.0.6400.6[7fb620270000+84000]
Jul 2 13:41:04 ubuntu kernel: [ 13.281163] traps: tracker-extract[1234] trap int3 ip:7f8718ac3295 sp:7ffe774d1c40 error:0 in libglib-2.0.so.0.6400.6[7f8718a87000+84000]

Jul 2 13:41:00 ubuntu gnome-session[1175]: gnome-session-binary[1175]: GLib-GIO-ERROR: No GSettings schemas are installed on the system
Jul 2 13:41:00 ubuntu gnome-session[1175]: aborting...
Jul 2 13:41:00 ubuntu gnome-session-binary[1175]: GLib-GIO-ERROR: No GSettings schemas are installed on the system#012aborting...
Jul 2 13:41:00 ubuntu gdm3: GdmDisplay: Session never registered, failing
Jul 2 13:41:00 ubuntu gdm3: GdmLocalDisplayFactory: maximum number of X display failures reached: check X server log for errors
Jul 2 13:41:00 ubuntu gdm3: Child process -1157 was already dead.

Now, looking closer, we see their gschema.compiled file exists. This means that we aren't dealing with a missing file and it not being re-created, but instead a corrupted gschema.compiled file.

I rebuilt the file with:

$ sudo glib-compile-schemas /usr/share/glib-2.0/schemas/

and rebooted, and the system came up normally. Very interesting.

From there, I rebuilt the file several times, each time checking the sha256 value. Each time it was exactly the same, so the compile process appears to be deterministic.

I then did a binary diff of the corrupted gschema.compiled file, and a freshly rebuilt one.

I found two bytes were different:

$ cmp -l ~/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}'
0000376F E3 25
00003771 A4 65

$ xxd ~/schemas/gschemas.compiled > ~/corrupt.bin
$ xxd /usr/share/glib-2.0/schemas/gschemas.compiled > ~/working.bin
$ diff ~/corrupt.bin ~/working.bin
887,888c887,888
< 00003760: 0515 0000 ffff ffff 7837 0000 0000 e300 ........x7......
< 00003770: a455 0000 0000 0000 6f72 672e 676e 6f6d .U......org.gnom
---
> 00003760: 0515 0000 ffff ffff 7837 0000 0000 2500 ........x7....%.
> 00003770: 6555 0000 0000 0000 6f72 672e 676e 6f6d eU......org.gnom

I need to determine exactly how these two bytes ended up differen...

Attached is a debdiff for glib2.0 on Focal which fixes this problem.

Thanks, sponsored to the queue

Thanks Iain. Looks like both of us sponsored it for the queue. I'd appreciate rejecting either one of them.

Thanks!

> Looks like both of us sponsored it for the queue. I'd appreciate rejecting either one of them.

I pushed to the VCS[0] at the same time as sponsoring, so I'll reject the other one which doesn't quite match up with it.

[0] https://salsa.debian.org/gnome-team/glib/-/commits/ubuntu/2.64.6-1_ubuntu20.04.4/

Hello Matthew, or anyone else affected,

Accepted glib2.0 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glib2.0/2.64.6-1~ubuntu20.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Performing verification for Focal.

I will first reproduce the problem with glib2.0 2.64.6-1~ubuntu20.04.3 from -security with the libglib2.0-0 libglib2.0-bin libglib2.0-data packages.

I deleted all existing schemas from /usr/share/glib-2.0/schemas and replaced them with a set of schemas which reproduce the problem easily from my customer.

$ cd /usr/share/glib-2.0/schemas/
$ sudo rm *
$ sudo cp ~/schemas/* .

The gsettings.compiled from the customer has been corrupted, and when I reboot, gdm fails to start and I get a blank screen with a blinking insertion pointer.

The sha256 of the customers corrupted gsettings.compiled is:

$ sudo openssl sha256 /usr/share/glib-2.0/schemas/gschemas.compiled
SHA256(/usr/share/glib-2.0/schemas/gschemas.compiled)= 2c98dc9a7fdbac858a8d5ca7e4dd813f16058a46dba2c54b5239cd8cdba5bb3e

When I ssh back in, and recompile the file:

$ sudo glib-compile-schemas /usr/share/glib-2.0/schemas
Error parsing key “logout” in schema “org.gnome.settings-daemon.plugins.media-keys” as specified in override file “/usr/share/glib-2.0/schemas/50_vmware_viewagent.gschema.override”: 0-22:can not parse as value of type 'as'. Ignoring override for this key.
$ sudo openssl sha256 /usr/share/glib-2.0/schemas/gschemas.compiled
SHA256(/usr/share/glib-2.0/schemas/gschemas.compiled)= 78163b5f0000efbd6320ce0d355c9531bf657a4f4dc15f057d95ef144323cd56

The sha256 has changed. Doing a bindiff, I see:

$ sudo cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}'
0000376F E3 4F
00003771 A4 C2

We see two bytes different. These bytes are the uninitialised memory this bug is about. When I reboot, gdm starts fine, but that is because this time I got lucky and the parser for the gschema.compiled file thinks 4F and C2 are okay. But there are combinations which aren't okay, and will end up with a corrupted gschema.compiled file.

Re-compiling the file again:

$ sudo glib-compile-schemas /usr/share/glib-2.0/schemas
Error parsing key “logout” in schema “org.gnome.settings-daemon.plugins.media-keys” as specified in override file “/usr/share/glib-2.0/schemas/50_vmware_viewagent.gschema.override”: 0-22:can not parse as value of type 'as'. Ignoring override for this key.
$ sudo openssl sha256 /usr/share/glib-2.0/schemas/gschemas.compiled
SHA256(/usr/share/glib-2.0/schemas/gschemas.compiled)= 460c70faca7afc26fa88a0e5918d312478e15f20ad84f4afaa5d17627a823e01

The sha256 changed, and if we bindiff, the bytes have changed:

$ sudo cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}'
0000376F E3 A6
00003771 A4 A1

If we run glib-compile-schemas through valgrind, it reports that we are writing to uninitialised memory:

https://paste.ubuntu.com/p/sxrQtbswpw/

I then enabled -proposed and installed libglib2.0-0 libglib2.0-bin libglib2.0-data version 2.64.6-1~ubuntu20.04.4.

Now, when I re-compile the gschemas.compiled file, the sha256 matches every time, meaning no more non-deterministic behaviour caused by writing unitialised memory to disk:

$ sudo gli...

All autopkgtests for the newly accepted glib2.0 (2.64.6-1~ubuntu20.04.4) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

libssh/0.9.3-2ubuntu2.1 (armhf)
apport/2.20.11-0ubuntu27.18 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#glib2.0

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The autopkgtests worked after a retry

The verification of the Stable Release Update for glib2.0 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package glib2.0 - 2.64.6-1~ubuntu20.04.4

---------------
glib2.0 (2.64.6-1~ubuntu20.04.4) focal; urgency=medium

  * Initialise memory used for file builder buffers to zero, since memory
    artifacts found themseleves into gschema.compiled files, leading to glib
    being unable to parse the gschema.compiled files, causing gdm, gnome-shell
    and various gnome applications to fail to
    start. (LP: #1930359)
      - d/p/gvdb-builder-Initialise-some-memory-to-zero-in-the-bloom-.patch

 -- Matthew Ruffell <email address hidden> Mon, 12 Jul 2021 15:56:16 +0100