Automate private mailing lists

From Elliot:

There is an apache module written in python that handles the openID
authentication. For each protected directory, the plugin reads from two
files: one to get a list of teams to query membership in, and one to get
a list of teams that are authorized to access the directory.

Conceptually, this should be a single list, but it is not due to some
awkwardness in how apache configs are specified and how openID endpoints
work, they are separate - it might be possible to collapse them but I've
not thought very hard about it yet.

I imagine changing this apache module to be able to work in an automated
way would involve changing it to do lookups of each directory it serves
against a database rather than reading from a text file, and getting the
list of team names from the DB. Then, launchpad would write to the DB
table in order to specify the teams which were allowed to access a
particular archive. Probably 1 day of work or less.

I'd volunteer to do a preimplementation call with someone working on
this module, as I've helped Tom Haddon debug it in the past and have it
running on my home apache server right now.

More from Elliot:

Barry Warsaw wrote:
Thanks Elliot, that was very helpful. Is the code available anywhere?

Yep, https://code.edge.launchpad.net/~rowan/launchpad/apache-openid

There are a few other interesting private branches at
https://code.launchpad.net/launchpad also...

Whenever someone goes to hacking on this module, it should be fixed not
to go into an infinite openID loop when denying someone access to the
resource :)

Please keep in mind that it might end up being simpler to stuff all the emails in the Message/MessageChunk tables and make the archives accesible via Launchpad - I think we already have nearly all the infrastructure we need to put together the web interface and won't have to reimplement stuff like Bug #NNN autolinking in the third party tools.

We have a general understanding of what must happen, and know what files must be generated. We need to decide how to transfer the configuration when a team's visibility changes.

I'm thinking we extend the mailing list actions in the XMLRPC interface to handle changes in visibility.

Just to follow up to Stuart's comment: MHonArc doesn't really buy us that much in terms of sophisticated web ui to the archives. I think we'll still start by looking at the openid stuff, but let's keep in mind implementing an archiver for Launchpad (which could potentially have lots of other benefits).

Elliot, ~rowan doesn't exist. Also, where are those magic private branches of which you speak?

* https://code.edge.launchpad.net/~rowan/canonical-bis-openid/apache-openid

* Tom Haddon has apache conf examples; it's basically used everywhere
  (directory, moin, pastebin)

* special directory called openid

* special protect line -> config file

* htaccess? on ml creation or update

* pass private flag to mailman and let it do the work

* code above adds team membership support underneath openid package

* add one config & extend text file per resource

https://pastebin.canonical.com/11800/

The core code landed in 2.1.12, but the apache mod is not ready. This feature may be releasable /before/ the 2.2.1 release date if all the stakeholders agree test the apache mod as soon as it is available.

This branch contains the reviewed, approved, and ready to be tested code.

bzr+ssh://bazaar.launchpad.net/%7Ebarry/canonical-bis-openid/apache-openid/

RT 32546

A test on staging failed. Logged in users were blocked, but annonymous users were not. There appears to be a conflict in code or config that prevents user from logging in, and the openid_mod lets the user see the directory. The team that was tested is ubunet-discuss.

Config: https://pastebin.canonical.com/13047/
Errors: https://pastebin.canonical.com/13048/

Barry can you make this your top priority on Tuesday of your return?

this looks like a configuration problem because from the error log i can't see where the team name is being extracted from the directory path. you can even see a message such that the regexp pattern match failed.

i can't tell what's going on without dumping out the environment that the plugin is seeing. the directory path may not be getting passed into the environ the same way it is on a dev machine.

my net connection is not so good right now, but i will make this my top priority when i return.

With Tom and Lamont's help, we solved the configuration problems. We also needed to add lists.staging to staging's trustroots. I've confirmed that access on staging appears to work as expected and will follow up with details on the mailing list.

This has now been rolled out to production.