Please import FEniCS mbox archives into Launchpad mailing lists

Asked by Johannes Ring on 2010-04-20

Hi,

Last year we moved all the mailing lists for the FEniCS project from www.fenics.org to Launchpad. Now we would like to import the old mbox archives into the mailing lists on Launchpad to keep the history. I have seen from other similar requests that this is technically possible, so here are the links to the mbox archives and where they should be imported:

http://www.fenicsproject.org/pub/mailinglists/dolfin-dev.mbox --> https://launchpad.net/~dolfin
http://www.fenicsproject.org/pub/mailinglists/fenics-dev.mbox --> https://launchpad.net/~fenics
http://www.fenicsproject.org/pub/mailinglists/ferari-dev.mbox --> https://launchpad.net/~ferari
http://www.fenicsproject.org/pub/mailinglists/ffc-dev.mbox --> https://launchpad.net/~ffc
http://www.fenicsproject.org/pub/mailinglists/fiat-dev.mbox --> https://launchpad.net/~fiat
http://www.fenicsproject.org/pub/mailinglists/instant-dev.mbox --> https://launchpad.net/~instant
http://www.fenicsproject.org/pub/mailinglists/meshbuilder-dev.mbox --> https://launchpad.net/~meshbuilder
http://www.fenicsproject.org/pub/mailinglists/syfi-dev.mbox --> https://launchpad.net/~syfi
http://www.fenicsproject.org/pub/mailinglists/ufc-dev.mbox --> https://launchpad.net/~fenics-ufc
http://www.fenicsproject.org/pub/mailinglists/ufl-dev.mbox --> https://launchpad.net/~ufl
http://www.fenicsproject.org/pub/mailinglists/viper-dev.mbox --> https://launchpad.net/~fenics-viper

Most (if not all) of the mailing lists on Launchpad are already in use so I guess a merge is needed in each case. It would be highly appreciated if you could do this for us.

Thanks a lot for your help.

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
[LEGACY] Canonical WebOps Edit question
Last query:
2011-05-17
Last reply:
2011-05-17
Whiteboard:
Hi Administrator: I have updated the mailman mhonarc manager utility to support mbox merging. I have included it with the historic mboxes in a tarball: scp devpad:/home/curtis/fenics.tar.gz There is a readme in the tarball that includes the steps to simplify doing multiple merges. mmm-archive-manager is now packaged in ~launchpad's ppa.
Barry Warsaw (barry) said : #1

It's technically possible to import mbox files, though a little more difficult to do the merge. However if the old mbox files end before the new mbox files begin, we can simply cat the new ones to the end of the old ones. I'm not positive whether MHonArc can faithfully regenerate the combined archives maintaining existing urls, but I do think that's the case.

If the merge is more complicated, we'll have to experiment to find out. The issue is assigned to the LOSAs, and I am willing to help out in the process.

Anders Logg (logg) said : #2

On Wed, Apr 21, 2010 at 06:25:19PM -0000, Barry Warsaw wrote:
> Question #107943 on Launchpad Registry changed:
> https://answers.edge.launchpad.net/launchpad-registry/+question/107943
>
> Barry Warsaw posted a new comment:
> It's technically possible to import mbox files, though a little more
> difficult to do the merge. However if the old mbox files end before the
> new mbox files begin, we can simply cat the new ones to the end of the
> old ones. I'm not positive whether MHonArc can faithfully regenerate
> the combined archives maintaining existing urls, but I do think that's
> the case.
>
> If the merge is more complicated, we'll have to experiment to find out.
> The issue is assigned to the LOSAs, and I am willing to help out in the
> process.

We closed down the old mailing lists at the same at as we set up the
new so there shouldn't be any overlap (except for one or two test
mails).

The last message in the old archive is dated

  Tue, 24 Nov 2009 15:10:17 +0100

This is also the first mail to the new lists (cross-posted):

  https://lists.launchpad.net/fenics/msg00000.html

Thanks for your efforts!
--
Anders

Johannes Ring (johannr) said : #3

Hi again,

Did someone ever attempt to merge the lists? Would it still be possible to do so? It is something that would be very nice for us to have.

Curtis Hovey (sinzui) said : #4

I do not think anyone has attempted this. We have just regenerated all the Lp mailing lists with a new tool that can help in this situation. I think the next step is to extend the regeneration script to combine the mboxes you list and regenerate the messages and indexes. This is actually a destructive process. Were any messages ever deleted from the Lp archives, because they will be restore from with the newly build mbox?

Curtis Hovey (sinzui) said : #5

I do not think anyone has attempted this. We have just regenerated all the Lp mailing lists with a new tool that can help in this situation. I think the next step is to extend the regeneration script to combine the mboxes you list and regenerate the messages and indexes. This is actually a destructive process. Were any messages ever deleted from the Lp archives, because they will be restore from with the newly build mbox?

Curtis Hovey (sinzui) said : #6

Sorry, those last messages were an effort to ask for more information, not a claim that the issue is resolved.

Anders Logg (logg) said : #7

On Tue, Apr 19, 2011 at 01:08:52PM -0000, Curtis Hovey wrote:
> Question #107943 on Launchpad itself changed:
> https://answers.launchpad.net/launchpad/+question/107943
>
> Status: Open => Answered
>
> Curtis Hovey proposed the following answer:
> I do not think anyone has attempted this. We have just regenerated all
> the Lp mailing lists with a new tool that can help in this situation. I
> think the next step is to extend the regeneration script to combine the
> mboxes you list and regenerate the messages and indexes. This is
> actually a destructive process. Were any messages ever deleted from the
> Lp archives, because they will be restore from with the newly build
> mbox?

We haven't touched the Launchpad archives. What we have is the mbox
files from the time before Launchpad when we ran our own email list
on our own server. We'd just like to add these to the Launchpad
archives so we have the entire history searchable.

Thanks for the effort!
--
Anders

Johannes Ring (johannr) said : #8

As Anders said, we haven't touched the Launchpad archives.

(Change status back to "Open".)

Curtis Hovey (sinzui) said : #9

I have assigned the question to an admin. We have a utility now that can merge history mboxes with the Lp mailing list mboxes.

Johannes Ring (johannr) said : #10

That sounds good. Thanks a lot for your work so far!

Haw Loeung (hloeung) said : #11

I have imported as per Curtis' instructions. Could you please let me know if there's any problems?

Thanks,

Haw

Curtis Hovey (sinzui) said : #12

I do not see any of the old messages. I expect to see message https://lists.launchpad.net/fenics-viper/msg00000.html to be:
    Date: Tue, 13 Feb 2007 00:33:41 +0100
    Subject: [Viper-dev] vtk file reader

There is were about 494 messages in viper, I see only 13.

/0\ I am incompetent There is a missing switch in each command. Without the switch all we did was regenerate the existing archive...no merge.

Here is a repeat of the instructions, with step four fixed:

First
Install/copy the latest version of mmm-archive-manager included in this
directory so that other merges can be done in the future.

Second
copy this directory and its contents to /srv/lists.launchpad.net/var/mailman
so that we can make these many merges easy.

Third
mmm-archive-manager must alway be run in the mailman directory.
cd /srv/lists.launchpad.net/var/mailman

Fourth
./fenics/mmm-archive-manager -m dolfin ./fenics/dolfin-dev.mbox
./fenics/mmm-archive-manager -m fenics ./fenics/fenics-dev.mbox
./fenics/mmm-archive-manager -m ferari ./fenics/ferari-dev.mbox
./fenics/mmm-archive-manager -m ffc ./fenics/ffc-dev.mbox
./fenics/mmm-archive-manager -m fiat ./fenics/fiat-dev.mbox
./fenics/mmm-archive-manager -m instant ./fenics/instant-dev.mbox
./fenics/mmm-archive-manager -m meshbuilder ./fenics/meshbuilder-dev.mbox
./fenics/mmm-archive-manager -m syfi ./fenics/syfi-dev.mbox
./fenics/mmm-archive-manager -m fenics-ufc ./fenics/ufc-dev.mbox
./fenics/mmm-archive-manager -m ufl ./fenics/ufl-dev.mbox
./fenics/mmm-archive-manager -m fenics-viper ./fenics/viper-dev.mbox

Fifth
Remove the fenics directory.

Haw Loeung (hloeung) said : #13

Okay, check now.

Curtis Hovey (sinzui) said : #14

I think is this complete. I see messages going back to 2004 is some lists.

Johannes Ring (johannr) said : #15

Thanks! This looks very nice except for DOLFIN where it seems like no messages from the old mbox archive have been included in the list on Launchpad. Can you take a look?

Curtis Hovey (sinzui) said : #16

Dophin failed to update because I provided a zero-length inbox :(

First
Download http://www.fenicsproject.org/pub/mailinglists/dolfin-dev.mbox and
copy it to /srv/lists.launchpad.net/var/mailman so that we can make these many merges easy.

Second
mmm-archive-manager must alway be run in the mailman directory.
cd /srv/lists.launchpad.net/var/mailman

Third
./fenics/mmm-archive-manager -m dolfin ./dolfin-dev.mbox

Haw Loeung (hloeung) said : #17

That looks better:

    Writing database ...
    23278 new messages
    23278 total messages

Haw Loeung (hloeung) said : #18

I believe it's all done now. Please let me know if it hasn't all been imported.

Johannes Ring (johannr) said : #19

Everything seems to be in place now. Thanks a lot for your efforts!

Anders Logg (logg) said : #20

On Wed, May 18, 2011 at 08:25:59AM -0000, Johannes Ring wrote:
> Question #107943 on Launchpad itself changed:
> https://answers.launchpad.net/launchpad/+question/107943
>
> Johannes Ring posted a new comment:
> Everything seems to be in place now. Thanks a lot for your efforts!

Great! Thanks!

--
Anders