need uc20 grade=dangerous channel=beta images built

grade dangerous images allow override of channels... but they must be on per-snap basis.

Thus alternative to this request is to somehow allow passing a channel map from cdimage to livecdrootfs get stuff back and publish them in unique directories. It has to be on per-snap basis, because the set of snaps in a model use different tracks. latest/<risk> for snapd & core20, yet 20/<risk for gadgets and kernels.

Signing a new "dangerous-beta" model assertion, and publishing it again as channel "dangerous-beta" is what proposed above. For consistency maybe the /dangerous/ should then be renamed /dangerous-edge/.

You say that you need this, but didn't specify the why.

As far as I understand, due to snapd deficiencies cert team cannot automatically test production grade:signed images. They can only test grade:dangerous images at the moment due to snapd enforced constraints.

If and when, snapd improves, we can stop building dangerous-edge and dangerous-beta images. Or like do the snapd improvements instead of fulfilling this request.

Where is it tracked to improve snapd, to unblock cert testing automatically grade:signed images?

Yes this is a stated snapd deficiency that will be solved by the 1.0 release, but there is a possibility that we do not have it solved in the next week or so, so we have this request. If you would like to file a snapd bug to track cloud-init support on grade > dangerous images, feel free to do so.

> If and when, snapd improves, we can stop building dangerous-edge and dangerous-beta images

This is correct, when snapd improves we can stop doing this but since snapd doesn't do this currently we want this bug report instead.

The whole point of grade=dangerous is that it changes the behavior of snapd. So testing images built with grade=dangerous is not a reliable proxy for assuring the correctness of the grade=signed images. How are these going to be tested, since they can't be tested automatically, and in what way is automated testing of grade=dangerous images valuable given the behavior is deliberately different from the production images?

> So testing images built with grade=dangerous is not a reliable proxy for assuring the correctness of the grade=signed images.

This is partially correct, insofar as yes there are some non-trivial behavior changes between the grade==dangerous and grade==signed images, however much of the behavior difference is not noticeable in the kinds of tests that we want the cert team to be running automatically. For instance here is as complete a list as I could find of the current list of behavior changes:

* You cannot specify unasserted snaps in a grade > dangerous image when building the image
* You cannot specify extra snaps not in the model assertion when building the image
* Snapd will not automatically import cloud-init configuration files placed in the ubuntu-seed partition at runtime (which is how cert team is currently using cloud-init with grade==dangerous channel==edge image testing)

Over time, this list will grow (and indeed we have plans for it to change between UC20 beta release and 1.0 release) and so you are correct in that we should not in general be substituting test results for grade==dangerous channel==beta for grade==signed channel==beta test results, but considering that the difference right now is so minimal I don't think there is a significant difference in behavior to justify not testing this considering the alternatives detailed below.

> How are these going to be tested

Well right now AFAIK they are not being automatically tested at all, which is the whole problem. We, snapd team, are working on getting cloud-init support for grade > dangerous images but it is not here yet as the problem is non-trivial. With this support in place the cert team can automatically test grade==signed channel==beta images, but until then they either test nothing automatically or they only test grade==dangerous channel==beta images.

> in what way is automated testing of grade=dangerous images valuable given the behavior is deliberately different from the production images?

Currently since the cert team is unable to test the grade==dangerous images due to issues with snapd not supporting cloud-init yet in that environment, they are testing grade==dangerous channel==edge images, and this is unfortunate because there is a non-trivial diff between the set of functionality on edge versus what's on beta, and the edge images change daily and so it's been really unclear to them if they need to keep re-running their tests every single day or which images are really considered "release candidates" which doesn't make sense for images produced from the edge channel. We really want them to be testing what's on beta, not on edge, as they have never been tasked with testing anything on edge up to this point, and honestly edge channels for all snaps is likely too unstable for test results to be considered reliable anyways.

Thanks,
Ian

So I was personally a bit going back and forth about this bug. Originally, at the very beginning, I had the wrong assumption that the dangerous builds were beta based anyway - but that was just my personal imagination!

Looking at it now, seeing Ian's comment about current actual differences between grade signed and dangerous images (in practice), I think we should just do it and build the beta dangerous images in cdimage. It is unfortunate that we can't test grade signed images, which are the actual images that we want to release, and from the release team perspective it's not good. But the certification team anyway performs manual validation on the images as part of the process, so the final grade signed images will be going through a final set of testing anyway (besides the automated runs on the dangerous ones). So I think from the acceptance criteria we should be good.
Generating grade dangerous beta images will certainly have its merits.

When asking the snapd team if maybe we could drop the dangerous edge image builds, they said they still think those are useful for quick edge testing.

As Dimitri mentioned, we will need to create a new model assertion for this + add a new CHANNEL handler to livecd-rootfs. I also agree that it would look better if we then renamed the dangerous one to dangerous-edge. This will require additional work though, since besides the steps outlined by xnox, we'd also have to re-create the dangerous edge ones to something like dangerous-edge, sign them, upload to the store and (possibly) request the removal of the 'dangerous' models (to not have cruft/leftovers).

Ok, so actually this is much easier. After talking with mvo it seems that grade dangerous models can have their channels freely overridden, which means no new model assertions need to be created. What I did is I prepared an upload for livecd-rootfs which adds the support for dangerous-<CHANNEL> channel. Once such a channel is given, we will use the dangerous model (which defaults to edge) and then override it with the channel name that's supplied after the -. I did a test build with this change and it seemed to work:

https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/focal/ubuntu-core/+build/218342

Pushed it to the ubuntu/focal branch and uploaded to snappy-dev/image. I also switched cdimage to build the new images.

I geuss we still need to do livecd-rootfs SRU to ensure we don't loose this code.

This bug was fixed in the package livecd-rootfs - 2.668

---------------
livecd-rootfs (2.668) groovy; urgency=medium

  [ Łukasz 'sil2100' Zemczak ]
  * Enable overrides of UC20 grade dangerous channels - as this is possible.
    (LP: #1879350)

  [ Patrick Wu ]
  * live-build/ubuntu/hooks/040-hyperv-desktop-images.binary: fix issues
    with Hyper-V xrdp support.

  [ Balint Reczey ]
  * Remove fstab from squashfs images (LP: #1877078)

 -- Dimitri John Ledkov <email address hidden> Tue, 23 Jun 2020 15:05:11 +0100

Hello Ian, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I have just fired up a livefs build of ubuntu-core on focal (UC20), without the snappy PPA and with -proposed enabled, with CHANNEL=dangerous-beta:

https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/focal/ubuntu-core/+build/229423

The build succeeded and the model-assertion used is the dangerous-edge one, which is what was expected (as there is no dangerous-beta assertion). By looking at the image manifest I see that the core20 snap is indeed taken from beta, so the channel override happened successfully. In other words: the build ended up generating a dangerous beta-based image, as expected.

The build used livecd-rootfs 2.664.3. Considering this verified.

This bug was fixed in the package livecd-rootfs - 2.664.3

---------------
livecd-rootfs (2.664.3) focal; urgency=medium

  [ Łukasz 'sil2100' Zemczak ]
  * Enable overrides of UC20 grade dangerous channels - as this is possible.
    (LP: #1879350)

  [ Iain Lane ]
  * Hack seeding of linux kernel in ubuntustudio/focal
    ubuntustudio-default-settings in focal release has a Recommends to this
    kernel, which makes it impossible to update the kernel later on, since we
    would install the -updates and release kernel, which isn't allowed and
    causes FTBFS. Hack out the focal-release kernel and let the rest of the
    build process pull in the right one. (LP: #1884915)

 -- Iain Lane <email address hidden> Tue, 21 Jul 2020 16:25:18 +0100

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.