Enable watching Red Hat Bugzilla bugs

I think this must have been done administratively, so we need to trawl through history and figure out why.

A small batch on dogfood does seem to exhibit some unhappiness:

launchpad@labbu:~$ http_proxy=http://squid.internal:3128/ ftp_proxy=http://squid.internal:3128/ https_proxy=http://squid.internal:3128/ /srv/launchpad.net/codelines/current/cronscripts/checkwatches.py -t redhat-bugs -v --batch-size=10
2017-10-19 18:33:54 DEBUG Cronscript control file not found at file:cronscripts.ini
2017-10-19 18:33:54 INFO Creating lockfile: /var/lock/launchpad-checkwatches.lock
2017-10-19 18:34:02 DEBUG Using a global batch size of 10
2017-10-19 18:34:23 INFO Updating 10 watches for 10 bugs on https://bugzilla.redhat.com
2017-10-19 18:34:25 INFO Unknown remote importance 'unspecified unspecified' for bug u'1000208' on https://bugzilla.redhat.com.
2017-10-19 18:34:27 DEBUG no rules relevant to 'app.mainsite_only.canonical_url'
2017-10-19 18:34:33 INFO Failure updating bug u'1000208' on https://bugzilla.redhat.com (local bug: 1215656) (OOPS-12ed1702f41f9563fb6bcfdf49491093)
2017-10-19 18:34:33 INFO Unknown remote importance 'unspecified unspecified' for bug u'1001934' on https://bugzilla.redhat.com.
2017-10-19 18:34:34 INFO Unknown remote status 'CLOSED DEFERRED' for bug u'1004673' on https://bugzilla.redhat.com.
2017-10-19 18:34:34 INFO Unknown remote importance 'unspecified unspecified' for bug u'1004673' on https://bugzilla.redhat.com.
2017-10-19 18:34:34 INFO Unknown remote status 'CLOSED EOL' for bug u'1006289' on https://bugzilla.redhat.com.
2017-10-19 18:34:34 INFO Unknown remote importance 'unspecified unspecified' for bug u'1006289' on https://bugzilla.redhat.com.
2017-10-19 18:34:38 INFO Failure updating bug u'1006289' on https://bugzilla.redhat.com (local bug: 1274320) (OOPS-95670b7f6c634dfdbee67561262785d7)
2017-10-19 18:34:40 INFO Unknown remote importance 'unspecified unspecified' for bug u'1010941' on https://bugzilla.redhat.com.
2017-10-19 18:34:41 INFO Unknown remote status 'CLOSED EOL' for bug u'1013005' on https://bugzilla.redhat.com.
2017-10-19 18:34:41 INFO Unknown remote importance 'unspecified unspecified' for bug u'1013110' on https://bugzilla.redhat.com.
2017-10-19 18:34:46 INFO Failure updating bug u'1013110' on https://bugzilla.redhat.com (local bug: 1234612) (OOPS-cbbef56c8ad93e230f52e6d9cf0373e2)
2017-10-19 18:34:46 INFO Unknown remote importance 'unspecified unspecified' for bug u'1014976' on https://bugzilla.redhat.com.
2017-10-19 18:34:46 INFO Time for this run: 44.674 seconds.
2017-10-19 18:34:47 DEBUG checkwatches ran in 44.673754s (excl. load & lock)
2017-10-19 18:34:47 DEBUG Removing lock file: /var/lock/launchpad-checkwatches.lock

This may just be a matter of tweaking our code for the current set of RH statuses and importances.

Also, the OOPSes are things like this:

Traceback (most recent call last):
  Module lp.bugs.scripts.checkwatches.bugwatchupdater, line 85, in updateBugWatch
    self.importBugComments()
  Module lp.bugs.scripts.checkwatches.base, line 261, in wrapper
    return func(self, *args, **kwargs)
  Module lp.bugs.scripts.checkwatches.bugwatchupdater, line 155, in importBugComments
    "when importing comments for %s." % self.bug_watch.title)
  Module lp.bugs.model.bugtracker, line 627, in ensurePersonForSelf
    email, display_name, rationale, creation_comment)
  Module lp.registry.model.person, line 3652, in ensurePerson
    hide_email_addresses=True)
  Module lp.registry.model.person, line 3574, in createPersonAndEmail
    "%s is not a valid email address." % email)
InvalidEmailAddress: triniton is not a valid email address.

I don't know if you're using a python module for interfacing with Bugzilla or trying to talk to the API directly, but most of us interfacing with Bugzilla systems use python-bugzilla[1], which is adapted to RH/SUSE BZ and works for generic Bugzilla systems, too.

[1]: https://github.com/python-bugzilla/python-bugzilla

I've grepped 6.5 years of IRC logs and I can't find any indication of why this bug tracker was disabled, so I'm going to guess that it was due to high volume of OOPSes, see if I can get things generally working on dogfood, and then re-enable it.

It was disabled due to Red Hat Bugzilla's API crashing whenever asked for anything interesting (bug #598575). It looks like the Bugzilla has been upgraded since and is no longer so terribly broken.

Neal: We're using our own implementation, although it's talking to the same XML-RPC backend. I don't know whether the code would get very much shorter if we refactored it to use python-bugzilla; it would still need to fit into our framework for external bug trackers, so it wouldn't surprise me if there ended up being a bit of an impedance mismatch.

Anyway, all of the things that I've identified as going wrong so far (and fixed in my attached merge proposal) are specifically to do with translating Bugzilla's view of the world into something compatible with the Launchpad data model, so an external library wouldn't be able to help here.

Fixed in stable r18487 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/18487>.

This looks fine as far as it goes, but further testing reveals a couple of other things I'd like to fix before enabling this again on production:

 * There are some comments whose author is just a string of all-caps initials. Trying to import these OOPSes because sanitize_name results in the empty string, we then append "-redhat-bugs", and the result of that isn't a valid LP username.
 * There's a lot of noise because we can't push comments to bugzilla.redhat.com; this is because we don't have credentials set up for doing so. IMO if we don't have credentials configured for a Bugzilla instance then we shouldn't consider it eligible for comment pushing (or back-linking).

r18490 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/18490) is part of this bug's fix.

Fixed in stable r18491 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/18491>.

There are some remaining OOPSes due to comments with 'author': '' in the XML-RPC response, but otherwise this seems to be behaving reasonably now.

It's importing again now. It will take some time to catch up with six years of backlog, of course!