Comment 5 for bug 1879350

> So testing images built with grade=dangerous is not a reliable proxy for assuring the correctness of the grade=signed images.

This is partially correct, insofar as yes there are some non-trivial behavior changes between the grade==dangerous and grade==signed images, however much of the behavior difference is not noticeable in the kinds of tests that we want the cert team to be running automatically. For instance here is as complete a list as I could find of the current list of behavior changes:

* You cannot specify unasserted snaps in a grade > dangerous image when building the image
* You cannot specify extra snaps not in the model assertion when building the image
* Snapd will not automatically import cloud-init configuration files placed in the ubuntu-seed partition at runtime (which is how cert team is currently using cloud-init with grade==dangerous channel==edge image testing)

Over time, this list will grow (and indeed we have plans for it to change between UC20 beta release and 1.0 release) and so you are correct in that we should not in general be substituting test results for grade==dangerous channel==beta for grade==signed channel==beta test results, but considering that the difference right now is so minimal I don't think there is a significant difference in behavior to justify not testing this considering the alternatives detailed below.

> How are these going to be tested

Well right now AFAIK they are not being automatically tested at all, which is the whole problem. We, snapd team, are working on getting cloud-init support for grade > dangerous images but it is not here yet as the problem is non-trivial. With this support in place the cert team can automatically test grade==signed channel==beta images, but until then they either test nothing automatically or they only test grade==dangerous channel==beta images.

> in what way is automated testing of grade=dangerous images valuable given the behavior is deliberately different from the production images?

Currently since the cert team is unable to test the grade==dangerous images due to issues with snapd not supporting cloud-init yet in that environment, they are testing grade==dangerous channel==edge images, and this is unfortunate because there is a non-trivial diff between the set of functionality on edge versus what's on beta, and the edge images change daily and so it's been really unclear to them if they need to keep re-running their tests every single day or which images are really considered "release candidates" which doesn't make sense for images produced from the edge channel. We really want them to be testing what's on beta, not on edge, as they have never been tasked with testing anything on edge up to this point, and honestly edge channels for all snaps is likely too unstable for test results to be considered reliable anyways.

Thanks,
Ian