auth_token middleware fails for concurrent requests

Bug #843073 reported by klmitch
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
High
Unassigned

Bug Description

We were trying to run some performance tests on [melange](https://code.launchpad.net/~raxnetworking/nova/melange) using [keystone](https://github.com/openstack/keystone) auth_token middleware. The auth_token middleware fails with "500 Internal Server Error" for concurrent requests. I have pasted the server log below.

```log
2011-08-10 14:36:56 DEBUG [eventlet.wsgi.server] Traceback (most recent call last):
  File "/home/rackspace/openstack/nova_melange/.nova-venv/lib/python2.6/site-packages/eventlet/wsgi.py", line 345, in handle_one_response
    'Content-Length' not in [h for h, _v in headers_set[1]]:
IndexError: list index out of range
```

The log just gives an indication that heders_set array is empty. Digging deep in to the error we found that this was due to the auth_token middleware.

This happens because *start_response* callback is stored as a instance level attribute in auth_token and being used in serveral private methods. When concurrent request are made, same instance of start_response callback is being reused across the multiple request.

The issue can be fixed by passing start_response as arguments to these private_methods such as _forward_request etc.

The same applies to request *env* and *proxy_headers* variables, which are being used as instance attributes should be passed as method arguments.

If you need any help in reproducing issue, please let me know. I have pasted the command that was used for benchmarking.

```
autobench --single_host --host1=10.10.253.25 --port1=9898 --uri1=/v0.1/ipam/ip_blocks --low_rate=50 --rate_step=10 --high_rate=60 --num_call=1 --num_conn=100 --file=results.tsv --timeout=180
```
Response:

```
Connection rate: 0.9 conn/s (1080.7 ms/conn, <=100 concurrent connections)
.
.
Reply status: 1xx=0 2xx=1 3xx=0 4xx=0 5xx=99
```

Tags: bug
Revision history for this message
klmitch (q-noreply) wrote :

Iam working on this issue.

Revision history for this message
klmitch (q-noreply) wrote :

@endeepak Can you test yogi's fix? https://review.openstack.org/#change,214

    git fetch https://review.openstack.org/p/openstack/keystone refs/changes/14/214/1 && git checkout FETCH_HEAD

Revision history for this message
klmitch (q-noreply) wrote :

The auth middleware is working fine in the latest code. Thanks.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.