Please backport fix for & in attributes

Status changed to 'Confirmed' because the bug affects multiple users.

Another upstream report of this was here:
https://github.com/hughsie/fwupd/issues/590

From https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1780520/comments/2:

"I have uploaded a fixed package to my PPA: https://launchpad.net/~fmarier/+archive/ubuntu/ppa?field.series_filter=xenial

Here's how I fixed the problem:

1. upgrade the appstream-glib package to the one in my PPA
2. delete /var/cache/app-info/xmls/fwupd.xml
3. run `fwupdmgr refresh` as root to re-download the XML file"

This is the patch I added to the package: https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1780520/+attachment/5161135/+files/06_ampersand_in_attribute_values.patch

Is the step to delete old metadata actually required? That will complicate
the SRU if so..

On Wed, Jul 11, 2018, 13:25 François Marier <email address hidden> wrote:

> >From https://bugs.launchpad.net/ubuntu/+source/appstream-
> glib/+bug/1780520/comments/2:
>
> "I have uploaded a fixed package to my PPA:
>
> https://launchpad.net/~fmarier/+archive/ubuntu/ppa?field.series_filter=xenial
>
> Here's how I fixed the problem:
>
> 1. upgrade the appstream-glib package to the one in my PPA
> 2. delete /var/cache/app-info/xmls/fwupd.xml
> 3. run `fwupdmgr refresh` as root to re-download the XML file"
>
> This is the patch I added to the package:
> https://bugs.launchpad.net/ubuntu/+source/appstream-
>
> glib/+bug/1780520/+attachment/5161135/+files/06_ampersand_in_attribute_values.patch
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1780442
>
> Title:
> Please backport fix for & in attributes
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fwupd/+bug/1780442/+subscriptions
>

I was still getting the error messages after upgrading the package because the cached file still had the unescaped ampersand in it.

It seems like `fwupdmgr refresh` doesn't re-generate the file if it hasn't changed on the server.

It appears --force does not do it after all, I had to remove the file.

Uploaded fix in appstream-glib 0.5.13-1ubuntu6

When will this fix be released?

It needs to go through the SRU process (https://wiki.ubuntu.com/StableReleaseUpdates) which can take a few weeks.

I want to make sure it's clear this is a high-impact regression. Right now I'm unable to use Software at all because of this, Software is empty and says "no software data found". Software is useless. Manually editing fwpd.xml or removing it and refreshing appstream restores functionality. For me personally this ain't such a big deal as I know how to workaround it, but normal users will be clueless and should not be expected having to wait weeks for restored functionality because of something as trivial as a unescaped ampersand.

Sorry it should say "fwupd.xml". Weird that you can't edit recent comments at Launchpad, maybe Ubuntu should consider switching to GitLab ;)

Hello Mario, or anyone else affected,

Accepted appstream-glib into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/appstream-glib/0.5.13-1ubuntu6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Accepted since the fix will help anyone new impacted by this bug. But what happened to the upgrade path as mentioned in comments 4 and 5? Should the cache be deleted in postinst if upgrading up to a fixed version?

Please add this to Regression Potential and test for this during SRU verification.

Hello,

The package name here is libappstream-glib8 (no package found by the name appstream-glib). I installed the new version:

$ apt-cache policy libappstream-glib8 | grep -i installed
  Installed: 0.5.13-1ubuntu6

... and I confirm it fixes the bug here, *under the following circumstances*:

1. I did reboot just after installing the new version. I am not sure whether this step is necessary, but I wanted to force a reload of the lib in memory, as it was in use by a couple of processes:

$ dpkg -L libappstream-glib8 | grep .so
/usr/lib/x86_64-linux-gnu/libappstream-glib.so.8.0.6
/usr/lib/x86_64-linux-gnu/libappstream-glib.so.8
$ lsof /usr/lib/x86_64-linux-gnu/libappstream-glib.so*
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
compiz 2096 **** mem REG 8,17 279256 1837580 /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8.0.6
gnome-sof 2249 **** mem REG 8,17 279256 1837580 /usr/lib/x86_64-linux-gnu/libappstream-glib.so.8.0.6

2. It did not work until I removed the offending cache file first. Running `sudo fwupdmgr refresh --force` and `sudo appstreamcli refresh-cache --force --verbose` wouldn't update the file otherwise.

(I am not adding the verification-done tag as updating the package without removing the cache was not enough to fix the issue)

gumbeto - I'd recommend opening a new bug about the issue for clients with the bad cache. This fix is still useful as it will stop new users hitting the problem. I've asked on https://github.com/hughsie/fwupd/issues/565 if upstream has a solution for that issue.

If you open a new bug please mark this one as verified.

Hello, I agree this is very helpful already, but it seems it is still not a full fix for this issue yet. There is no other reason for removing and refreshing the cache than this bug, so I think opening a new one would only confuse things.

Per discussion upstream there is a commit needed for fwupd as well to make it not abort on the bad data. So the appstream-glib tasks should get a verification-done tag and flow through, but we also need to SRU fwupd with this commit: https://github.com/hughsie/fwupd/commit/fe1c4de5a4b178cae3e4e9325ea84ff85c4b1be3

@Robert
I've uploaded a fwupd package with the designated above fix.

@gumbeto
Watch this bug for when an archive admin releases the fix and you can try it out by putting in some corrupted data back into the cache again (you can manually modify it to "corrupt" it).

Sure, will try to do that tomorrow.

Hello Mario, or anyone else affected,

Accepted fwupd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupd/0.8.3-0ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@Mario et all, as a simple user I am not very familiar with bug tracking procedures for Ubuntu, but I understood the verification tag to be associated with the bug, not a package. It did not make much sense to me to mark the bug verified until there was a full fix. I suppose you can't be sure something will be part of a final fix until the whole is verified (e.g. a newer version of the same pkg might be needed).

Anyway, I just tested `fwupdmgr refresh` with both new packages and an old cache file with the unescaped ampersand. It worked perfectly here, so updated tag to verification-done-xenial

Thanks gumbeto for the verification! For the future: it is also important for the verification to include package version numbers of what -proposed packages have been tested. Otherwise yes, waiting with performing the validation until all required packages are in -proposed is a good choice.

This bug was fixed in the package appstream-glib - 0.5.13-1ubuntu6

---------------
appstream-glib (0.5.13-1ubuntu6) xenial; urgency=medium

  * debian/patches/0001-Never-include-in-attribute-values.patch:
    - Handle '&' in attribute values (LP: #1780442)

 -- Robert Ancell <email address hidden> Mon, 30 Jul 2018 11:43:02 +1200

The verification of the Stable Release Update for appstream-glib has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package fwupd - 0.8.3-0ubuntu4

---------------
fwupd (0.8.3-0ubuntu4) xenial; urgency=medium

  * debian/patches 0001-Do-not-abort-startup-if-the-XML-metadata-file-is-inv.patch:
    - Fix a crash from bad metadata (LP: #1780442)

 -- Mario Limonciello <email address hidden> Thu, 16 Aug 2018 13:41:15 -0500