Comment 14 for bug 499095

Here's the QA on dogfood, so it's ready to be CP'd:

Setup
=====
1. from codelines/current, bzr pull -r8840 (db-devel rev. before the fix landed)
2. Emptied the current queue of builds (after issues with ferraz and new concordia builders).
3. Applied http://pastebin.ubuntu.com/353343/ to force GIVENBACK processing.
4. Uploaded and processed:
    https://dogfood.launchpad.net/~michael.nelson/+archive/pocketsphinx/+build/1344510

Reproduce problem
=================
1. Started buildd-manager and confirmed that..
  a) initial build_a correctly set date started etc.:
      http://people.canonical.com/~michaeln/tmp/499095-build_a_1.png
  b) After going through GIVENBACK it was rescheduled but got stuck as expected (no start date, no build log etc.)
      http://people.canonical.com/~michaeln/tmp/499095-build_a_2.png

2. Cleanup:
  a) Manually set the build date_started and job.status
  b) Reverted the forec GIVEBACK change, restarted buildd-manager and let the buildmanager clear the job normally.

Demo fix
========
1) from codelines/current, bzr pull (which pulled 8842 - including the fix).
2) Re-applied http://pastebin.ubuntu.com/353343/ to force GIVENBACK processing.
3) Created, uploaded and processed:
     https://dogfood.launchpad.net/~michael.nelson/+archive/pocketsphinx/+build/1344517
4) Started buildd-manager and confirmed that initial build_b correctly set date_started etc.
    http://people.canonical.com/~michaeln/tmp/499095-build_b_1.png
5) After going through GIVENBACK it was rescheduled correctly (has a new date started, and the log is being updated as the build progresses:
    http://people.canonical.com/~michaeln/tmp/499095-build_b_2.png

    Note: we still need some UI changes here. We're displaying the current builder being used to build (build_queue.builder) as well as the historical builder that was used to do the previous build (build.builder), as well as the preserved log/start/finish times. Heh, there's a note without a bug in the code about this:

        # XXX cprov 2006-05-30: Currently this information is not
        # properly presented in the Web UI. We will discuss it in
        # the next Paris Summit, infinity has some ideas about how
        # to use this content. For now we just ensure it's stored.

so I just created bug 504660.

6) Reverted the force GIVEBACK patch and restarted the buildd_manager and the build completed as per normal without any manual intervention:
http://people.canonical.com/~michaeln/tmp/499095-build_b_3.png

So, I'll be requesting a CP of stable 10114. Until that is landed, we can run https://pastebin.canonical.com/26358/ to clear the stuck builders (and may need to re-run it once after the CP lands in case any more crept in at the last minute.)