Milestone:+index timeouts with many bugs

Newer timeout: OOPS-1766ED73

Hi Edwin, I've had a look as requested.

I see lots of python time, *and* lots of requests.
The ValidPersonCache and person lookups are a pretty sure sign of death-by-queries, and I'm sure you'll see a benefit by changing things to use the existing off-by-default eager loading for bugs / milestones on that page.

If that doesn't improve things enough, the next step would be to get a python profile from [qa]staging, which requires asking a losa to enable profiling and then hitting the page with ++profile++ inserted after the // in the url.

Fixed in stable r12038 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12038>.

Marking QA-OK since it's not breaking anything as far as I can see, and it's blocking a critical rollout elsewhere.

Fixed in stable r12090 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12090>.

This milestone's +index page has gone from 219 queries to 51 queries.
https://launchpad.net/ubuntu/+milestone/ubuntu-8.10/+index

However, the milestone originally reported is still timing out on qastaging.
https://edge.launchpad.net/ubuntu/+milestone/ubuntu-9.10/+index

I have marked this bug qa-ok in order to not block any other fixes since this revision is safe to land in production.

According to OOPS-1812QS3, this query is taking 3.5 seconds, but it's the next 8 seconds spent in python where the timeout occurs.

SELECT BugTask.assignee, BugTask.bug,
BugTask.bugwatch, BugTask.date_assigned, BugTask.date_closed,
BugTask.date_confirmed, BugTask.date_fix_committed, BugTask.date_fix_released,
BugTask.date_incomplete, BugTask.date_inprogress, BugTask.date_left_closed,
BugTask.date_left_new, BugTask.date_triaged, BugTask.datecreated,
BugTask.distribution, BugTask.distroseries, BugTask.id, BugTask.importance,
BugTask.milestone, BugTask.owner, BugTask.product, BugTask.productseries,
BugTask.sourcepackagename, BugTask.status, BugTask.statusexplanation,
BugTask.targetnamecache, Bug.date_last_message, Bug.date_last_updated,
Bug.date_made_private, Bug.datecreated, Bug.description, Bug.duplicateof,
Bug.heat, Bug.heat_last_updated, Bug.id, Bug.latest_patch_uploaded,
Bug.message_count, Bug.name, Bug.number_of_duplicates, Bug.owner, Bug.private,
Bug.security_related, Bug.title, Bug.users_affected_count,
Bug.users_unaffected_count, Bug.who_made_private, Product.answers_usage,
Product.blueprints_usage, Product.owner, Product.translations_usage,
Product.active, Product.autoupdate, Product.bug_reported_acknowledgement,
Product.bug_reporting_guidelines, Product.bug_supervisor, Product.bugtracker,
Product.date_next_suggest_packaging, Product.datecreated, Product.description,
Product.development_focus, Product.displayname, Product.downloadurl,
Product.driver, Product.enable_bug_expiration,
Product.enable_bugfiling_duplicate_search, Product.freshmeatproject,
Product.homepage_content, Product.homepageurl, Product.icon, Product.id,
Product.lastdoap, Product.license_approved, Product.license_info,
Product.reviewed, Product.logo, Product.max_bug_heat, Product.mugshot,
Product.name, Product.official_answers, Product.official_blueprints,
Product.official_malone, Product.official_rosetta, Product.private_bugs,
Product.programminglang, Product.project, Product.registrant,
Product.remote_product, Product.reviewer_whiteboard, Product.screenshotsurl,
Product.security_contact, Product.sourceforgeproject, Product.summary,
Product.title, Product.translation_focus, Product.translationgroup,
Product.translationpermission, Product.wikiurl, SourcePackageName.id,
SourcePackageName.name
FROM BugTask
    LEFT JOIN BugTask AS ConjoinedMaster
        ON ConjoinedMaster.bug = BugTask.bug AND BugTask.distribution = %s AND
            ConjoinedMaster.distroseries = %s AND NOT (ConjoinedMaster.status IN (%s))
    LEFT JOIN Bug ON BugTask.bug = Bug.id
    LEFT JOIN Product ON BugTask.product = Product.id
    LEFT JOIN SourcePackageName ON BugTask.sourcepackagename = SourcePackageName.id
WHERE Bug.id = BugTask.bug
    AND BugTask.milestone = 12698
    AND ConjoinedMaster.id IS NULL
    AND Bug.duplicateof is NULL
    AND (Bug.private = FALSE
         OR EXISTS (
            SELECT BugSubscription.bug
            FROM BugSubscription, TeamParticipation
            WHERE TeamParticipation.person = 1363376
                AND BugSubscription.person = TeamParticipation.team
                AND BugSubscription.bug = Bug.id
                ))
ORDER BY BugTask.status, BugTask.i...

That suggests :
 - GIL/thread load (storm seems to interact badly with high load - I
observed this on staging)
 - or something in our python code - e.g. canonical url

I suggest getting a profile of the page from qastaging.

You get a very similar error by saying, through the api

person = launchpad.people['mbp']
tasks = person.searchTasks()

fails quite consistently, with eg https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1854EA3

On Fri, Jan 28, 2011 at 1:27 PM, Martin Pool <email address hidden> wrote:
> You get a very similar error by saying, through the api
>
> person = launchpad.people['mbp']
> tasks = person.searchTasks()

Most timeouts will be similar; please ignore whatever the lp-oops tool
suggests may be the same bug; its totally incorrect all of the time.