Launchpad itself

Stuck at updating diff/branch--merge proposal and branch don't show commits but branch code view does?

Asked by Richard Wilbur

Two recent bzr branches[1] uploaded(pushed) to Launchpad with merge requests registered seem to exhibit this same problem (even several days after branch push and merge request):
1. The merge proposal shows[2]:
"""
Updating diff...

An updated diff will be available in a few minutes. Reload to see the changes.
"""
Expected to see a list of "Unmerged Revisions" and a "Preview Diff" showing context of changes.

2. The branch summary page shows[3]:
"""
Updating branch...

Launchpad is processing new changes to this branch which will be available in a few minutes. Reload to see the changes.

Recent revisions

This branch is empty.
"""
Expected to see a list of "Recent Revisions".

3. The branch code view shows[4]: All the recent changes on the branch preceded by the previous changes on the trunk--as expected. It looks like the branch push to launchpad happened correctly and loggerhead is working fine!

I'm interested to know what, if anything, I could do better to avoid this situation. It certainly gets in the way of code reviews.

When I created the branch 'branches-uncommitted' I used what I consider to be a standard recipe:
bzr branch lp:bzr trunk
cd trunk
bzr merge lp:<feature branch>
<edit files>
bzr selftest
bzr commit -m 'Description of changes.'
bzr push lp:~richard-wilbur/bzr/branches-uncommitted
#Then I visited:

https://code.launchpad.net/~richard-wilbur/bzr/branches-uncommitted/+register-merge

#Filled out the form and submitted.

Several days later I revisited the merge request and branch summary pages and it still says "Updating diff ...", etc. What's going on here? Should I file a bug?

Sincerely,
Richard Wilbur

References:
[1] lp:~shawn111/bzr/lp_propose_message
     lp:~richard-wilbur/bzr/branches-uncommitted
[2] https://code.launchpad.net/~shawn111/bzr/lp_propose_message/+merge/258742
     https://code.launchpad.net/~richard-wilbur/bzr/branches-uncommitted/+merge/259295
[3] https://code.launchpad.net/~shawn111/bzr/lp_propose_message
     https://code.launchpad.net/~richard-wilbur/bzr/branches-uncommitted
[4] http://bazaar.launchpad.net/~shawn111/bzr/lp_propose_message/changes/6604?start_revid=6604
     http://bazaar.launchpad.net/~richard-wilbur/bzr/branches-uncommitted/changes/6607?start_revid=6607

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Solved by:
Colin Watson
Solved:
Last query:
Last reply:

This question was reopened

Revision history for this message
Best Colin Watson (cjwatson) said : 		#1

What's going on here is that there are some quite fundamental problems with the way that Bazaar branches are modelled in the Launchpad database: these make branch scanning very slow, and for branches with deep history the scanning process can often time out, which leads to the symptoms you're seeing. I've got our sysadmins to force a rescan of these branches for you, and your merge proposals are working now.

In future, you can in fact fix this yourself. If Launchpad's view (as opposed to loggerhead's) of a branch isn't up to date within about 10 minutes, then you can safely assume that it has timed out. Then, run "lp-shell production devel" (from the lptools package) and run something like this:

  lp.branches.getByUrl(url="lp:~shawn111/bzr/lp_propose_message").unscan(rescan=True)

(Obviously replace the URL with the real one.) This will force Launchpad to try again.

Revision history for this message
Richard Wilbur (richard-wilbur) said : 		#2

Colin,

Symptom Resolved: Thank you for your quick turn-around time on the answer. The pages are indeed looking much better.

Underlying Problem: You mention "some quite fundamental problems with the way Bazaar branches are modelled in the Launchpad database..." Is there something we can do on the Bazaar end of this problem to help speed up the slow branch scan?

Sincerely,
Richard
(Bazaar Project Maintainer)

Revision history for this message
Colin Watson (cjwatson) said : 		#3

It's really a Launchpad problem rather than a Bazaar problem: it models the full DAG in the database, such that every revision in every branch, including duplicates, has its own row in the BranchRevision table: so pushing a new branch means inserting a number of rows equal to the total number of revisions in that branch's history. For deep histories this is almost prohibitively slow. We've known about this for some time, but it's quite a lot of work to fix, and that kind of effort is currently going into Git support instead. (I'm not ruling out us fixing it in future, though.) William Grant has looked into this in the past and knows more of the details here.

One thing that may help is that when Launchpad itself moves to git, that will greatly reduce the size of the BranchRevision table (Launchpad itself has many branches and a very deep history), which may well ease database pressure for other users.

Revision history for this message
Richard Wilbur (richard-wilbur) said : 		#4

Thanks Colin Watson, that solved my question.

Revision history for this message
Richard Wilbur (richard-wilbur) said : 		#5

Unfortunately the symptoms have returned but I am unable to use the solution Colin Watson showed me. It worked when he gave it and for a number months subsequently.

Now, when I invoke lp-shell with that command for a branch I get an exception because the object seems to no longer have an 'unscan' method.
-----------------------------------------------
Traceback (most recent call last):
  File "/usr/bin/lp-shell", line 122, in <module>
    main()
  File "/usr/bin/lp-shell", line 78, in main
    run_command(options.command, launchpad)
  File "/usr/bin/lp-shell", line 27, in run_command
    exec(command)
  File "<string>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'unscan'
-----------------------------------------------
I tried poking around in lp-shell interactive mode and there is indeed no unscan. I tried calling lp_refresh() on the branch object but it doesn't seem to have the desired effect.

Revision history for this message
Colin Watson (cjwatson) said : 		#6

There's no unscan method because whatever you did to retrieve the branch returned None, not a branch object, and None doesn't have any webservice methods! Can you give us more context that shows what you did to retrieve the branch?

Revision history for this message
Richard Wilbur (richard-wilbur) said : 		#7

The command I was trying to run was:

lp-shell -c 'lp.branches.getByUrl(url="lp:~friedrich-v/bzr/russian-localization-fix").unscan(rescan=True)' production devel

which, now that I try it, works perfectly! Evidently when I tried to turn it into a shell script I messed up the quoting of the shell argument substitution to get the branch URI. And when I was looking at the object under lp-shell interactive mode, I forgot to specify 'production devel'.

Anyway, thanks again for your help. Sorry for the false alarm.

Revision history for this message
Richard Wilbur (richard-wilbur) said : 		#8

Thanks Colin Watson, that solved my question.