Glance

Is glance-api not multithreaded?

Asked by Paul Bourke

I am testing a Glance setup with the api/registry running on a single machine, with mysql and swift backends.

If I issue a long running command to glance, such as a large upload, further glance commands seem to queue up behind it, not returning until or very soon before the long running command completes.

Is this expected behavior? Or do I have something misconfigured?

Question information

Language:
English Edit question
Status:
Solved
For:
Glance Edit question
Assignee:
No assignee Edit question
Solved by:
Paul Bourke
Solved:
Last query:
Last reply:
Revision history for this message
Jay Pipes (jaypipes) said : 		#1

Glance is indeed "multi-threaded". Well, not multi-threaded, but it is multi-processing. It uses the eventlet library for its WSGI server to handle multiple incoming requests on separate coroutines.

Could you elaborate on specifically what you are doing?

Also, the Glance client (both the programmatic Python client and the CLI tool) are not multi-threaded, so you may just be experiencing the fact that those clients only perform one command at a time?

-jay

Revision history for this message
Paul Bourke (pauldbourke) said : 		#2

After looking at the code a little I understand (at least at a high level), what you mean by 'multi-processing'.

The reason I ask this is that I am experiencing very poor performance from glance-api.

To expand on the example in the original question, consider the following test:

$ dd if=/dev/zero of=file1G bs=1M count=1024
$ glance add name=file1G is_public=true
..

Then in a second terminal:
$ dd if=/dev/zero of=file10M bs=1M count=10
$ glance add name=file10M is_public=true
..

One would expect the 10MB upload to complete long before the 1G. However what I'm observing is the second upload hanging completely, only returning a second or two before the first one.

Note that I am issuing two separate client commands. (This also happens with alternate clients, for example using curl to download two separate images from the glance server).

Have you ever experienced anything similar to this?

Revision history for this message
Jay Pipes (jaypipes) said : 		#3

Could you please pastebin your conf files? Remember to remove any
security creds... Thanks!

-jay

On Wed, Aug 24, 2011 at 12:15 PM, Paul Bourke
<email address hidden> wrote:
> Question #168937 on Glance changed:
> https://answers.launchpad.net/glance/+question/168937
>
>    Status: Answered => Open
>
> Paul Bourke is still having a problem:
> After looking at the code a little I understand (at least at a high
> level), what you mean by 'multi-processing'.
>
> The reason I ask this is that I am experiencing very poor performance
> from glance-api.
>
> To expand on the example in the original question, consider the
> following test:
>
> $ dd if=/dev/zero of=file1G bs=1M count=1024
> $ glance add name=file1G is_public=true
> ..
>
> Then in a second terminal:
> $ dd if=/dev/zero of=file10M bs=1M count=10
> $ glance add name=file10M is_public=true
> ..
>
> One would expect the 10MB upload to complete long before the 1G.
> However what I'm observing is the second upload hanging completely, only
> returning a second or two before the first one.
>
> Note that I am issuing two separate client commands.  (This also happens
> with alternate clients, for example using curl to download two separate
> images from the glance server).
>
> Have you ever experienced anything similar to this?
>
> --
> You received this question notification because you are a member of
> Glance Core, which is an answer contact for Glance.
>

Revision history for this message
Paul Bourke (pauldbourke) said : 		#4

Jay,

/etc/glance/glance-registry.conf: http://pastebin.com/9nxF3UGf
/etc/glance/glance-api.conf: http://pastebin.com/zWRYwjaM

Thanks for your help with this.

Revision history for this message
Jay Pipes (jaypipes) said : 		#5

Thanks Tim,

Would you mind emailing me your API log files, please? Configs look just fine. My email is jaypipes at REVERSE('moc.liamg').

Cheers,
jay

p.s. Sorry about setting this to answered... Launchpad does that when I respond via email automatically it seems...

Revision history for this message
Paul Bourke (pauldbourke) said : 		#6

I've been testing Glance over SSL connections which I think is what's causing the extreme latency in responses.

Large uploads with SSL enabled seems to completely max out the cpu which I'm guessing is whats giving the api trouble responding to further requests.

If you have any input on running Glance over SSL I'd be interested to hear about it, other than that I'm happy to consider this problem solved!

Thanks again.

Revision history for this message
Jay Pipes (jaypipes) said : 		#7

Hey again Paul :)

Glance over SSL isn't really something that has been tested (so I'm surprised it even works, slow as it is!)

I've been working on adding SSL support in this blueprint:

https://blueprints.launchpad.net/glance/+spec/support-ssl

With code in review here:

https://review.openstack.org/#change,190

I've been a bit stuck adding a functional test case, though, and would love to get your thoughts on ways to generate a self-signed cert for the server and validate a client connection in a test...

Lemme know your thoughts :)

Cheers,
jay