Results indicate failure but log does not indicate why. Manual tests should attach command output the same way automated tests do

Bug #769144 reported by Jeff Lane 
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Checkbox (Legacy)
Won't Fix
Medium
Unassigned

Bug Description

Another one that appears to have failed, according to the results, but the logs do not give any clue as to why. Just opened this as a reminder that we should fix the test to provide better output.

https://certification.canonical.com/hardware/201011-6697/submission/cVNKVs4iJ4iMfta/results?form.status=FAIL&form.status=UNRESOLVED

Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

Wouldn't this be the case if the tester marked the test as failed but didn't provide an notes? As the test itself says, "If your system fails to wake at all and must be rebooted, restart System Testing after reboot and mark this test as Failed."

Changed in checkbox:
status: New → Incomplete
Revision history for this message
Jeff Lane  (bladernr) wrote :

Possibly... we may just need a process change in that case. Something along the lines of "If a test fails, add a note explaining the failure and the bug number opened"

IN the case of automated tests though, some sort of data should be logged for any result (pass or fail, IMO)

Changed in checkbox:
status: Incomplete → New
Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

What's happening here is that the way things stand, manual tests don't attach the command output, only the content of the text field. My suggestion would be to change the behavior so that if the text field is empty then the command output is attached. Let me know if you agree with this, because the bug description will have to be changed (or maybe raise a new one). There's nothing wrong with the sleep state test specifically at the moment.

Changed in checkbox:
status: New → Incomplete
Revision history for this message
Jeff Lane  (bladernr) wrote :

+1 as that removes the relying on the tester to remember/care enough to actually add some commentary to the failure.

Changed in checkbox:
status: Incomplete → Confirmed
summary: - [sleep_state_test] Results indicate failure but log does not indicate
- why
+ Results indicate failure but log does not indicate why. Manual tests
+ should attach command output the same way automated tests do
Revision history for this message
Jeff Lane  (bladernr) wrote :

Title changed accordingly.

Changed in checkbox:
importance: Undecided → Low
Revision history for this message
Jeff Lane  (bladernr) wrote :

Discussed at the Montreal sprint. We need a way to capture both the script output and any comments and then eventually munge those together for results submission.

Changed in checkbox:
status: Confirmed → Triaged
Revision history for this message
Daniel Manrique (roadmr) wrote :

Bumping to medium due to the dupe.

tags: added: checkbox-core
Changed in checkbox:
importance: Low → Medium
Revision history for this message
Zygmunt Krynicki (zyga) wrote :

Since plainbox already does this I don't think we want to ever change checkbox-legacy anymore

affects: checkbox → checkbox-legacy
Changed in checkbox-legacy:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.