Dispersion Report

what version of Swift are you using ? is this a SAIO setup ? what OS version? what version of eventlet ?

I haven't tried it with python 2.7 but I'l see if I can replicate this issue using 2.7

Swift, version 1.4.1. SAIO setup, Ubuntu 11.04.

Python 2.7.1+, eventlet 0.9.14-0ubuntu2

-Regards

On Mon, Jul 18, 2011 at 2:11 AM, Marcelo Martins <
<email address hidden>> wrote:

> Your question #165052 on OpenStack Object Storage (swift) changed:
> https://answers.launchpad.net/swift/+question/165052
>
> Status: Open => Needs information
>
> Marcelo Martins requested more information:
> what version of Swift are you using ? is this a SAIO setup ? what OS
> version? what version of eventlet ?
>
> I haven't tried it with python 2.7 but I'l see if I can replicate this
> issue using 2.7
>
> --
> To answer this request for more information, you can either reply to
> this email or enter your reply at the following page:
> https://answers.launchpad.net/swift/+question/165052
>
> You received this question notification because you asked the question.
>

Hi Kevin,

Ok, I gave it a try on a natty slice and although I did not encounter the same traceback that you posted I did see some Tracebacks in my logs.

- On my first step, the stats-reporter completed but throwing some errors on stdout which I believe to be unrelated.
Error sample:
ERROR: 184.106.119.94:6041/sdb4: 15 seconds
ERROR: 184.106.119.94:6011/sdb1: 15 seconds
ERROR: 184.106.119.94:6021/sdb2: 15 seconds
ERROR: 184.106.119.94:6021/sdb2: 15 seconds

The log files though showed some actual errors.
Error Sample:

contatiner-server.error log
==========================
Jul 19 00:12:39 saio-natty account-server STDOUT: hub.switch()
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/pymodules/python2.7/eventlet/hubs/hub.py", line 177, in switch
Jul 19 00:12:39 saio-natty account-server STDOUT: return self.greenlet.switch()
Jul 19 00:12:39 saio-natty account-server STDOUT: LockTimeout: 10 seconds: /srv/node/sdb4/accounts/110358/d5e/6bc5ab18a40b60c23020fc56b52abd5e
Jul 19 00:12:39 saio-natty account-server STDOUT: Traceback (most recent call last):
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/pymodules/python2.7/eventlet/greenpool.py", line 80, in _spawn_n_impl
Jul 19 00:12:39 saio-natty account-server STDOUT: func(*args, **kwargs)
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 508, in process_request
Jul 19 00:12:39 saio-natty account-server STDOUT: proto = self.protocol(socket, address, self)
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/python2.7/SocketServer.py", line 639, in __init__
Jul 19 00:12:39 saio-natty account-server STDOUT: self.handle()
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/python2.7/BaseHTTPServer.py", line 343, in handle
Jul 19 00:12:39 saio-natty account-server STDOUT: self.handle_one_request()
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 232, in handle_one_request
Jul 19 00:12:39 saio-natty account-server STDOUT: self.handle_one_response()
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 336, in handle_one_response
Jul 19 00:12:39 saio-natty account-server STDOUT: result = self.application(self.environ, start_response)
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/common/middleware/healthcheck.py", line 38, in __call__
Jul 19 00:12:39 saio-natty account-server STDOUT: return self.app(env, start_response)
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/account/server.py", line 305, in __call__
Jul 19 00:12:39 saio-natty account-server STDOUT: res = getattr(self, req.method)(req)
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/account/server.py", line 97, in PUT
Jul 19 00:12:39 saio-natty account-server STDOUT: req.headers['x-bytes-used'])
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/common/db.py", line 1348, in put_container
Jul 19 00:12:39 saio-natty account-server STDOUT: with lock_parent_directory(self.pending_file, self.pending_timeout):
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
Jul 19 00:12:39 saio-natty account-server STDOUT: return self.gen.next()
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/common/utils.py", line 615, in lock_path
Jul 19 00:12:39 saio-natty account-server STDOUT: sleep(0.01)
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/pymodules/python2.7/eventlet/greenthread.py", line 30, in sleep
Jul 19 00:12:39 saio-natty account-server STDOUT: hub.switch()
Jul 19 00:12:39 saio-natty account-server STDOUT: File "/usr/lib/pymodules/python2.7/eventlet/hubs/hub.py", line 177, in switch
Jul 19 00:12:39 saio-natty account-server STDOUT: return self.greenlet.switch()
Jul 19 00:12:39 saio-natty account-server STDOUT: LockTimeout: 10 seconds: /srv/node/sdb4/accounts/110358/d5e/6bc5ab18a40b60c23020fc56b52abd5e

- On my second step, I decided to update python-eventlet with the actual version that we use at the moment which is eventlet-0.9.15 . So I removed the package that came with natty and installed the 0.9.15 from source. Once the new version was installed no more tracebacks showed up in my logs. The errors "ERROR: 184.106.119.94:6041/sdb4: 15 seconds" persisted though but I think that may be related to something else, but I did not look too deep into it.

Unless, you are trying to find out how swift works with python 2.7 and perhaps report bugs, provide patches or etc ... , I would suggest working with 10.04 which is the LTS version of ubuntu and the one that is recommended for a production environment.

-
Marcelo
https://github.com/btorch/swift-saio.sh

Interesting...SAIO version running 1.4.0 reports were happy...I will rebuild
on 10.

Thanks,

On Mon, Jul 18, 2011 at 7:21 PM, Marcelo Martins <
<email address hidden>> wrote:

> Your question #165052 on OpenStack Object Storage (swift) changed:
> https://answers.launchpad.net/swift/+question/165052
>
> Status: Open => Answered
>
> Marcelo Martins proposed the following answer:
> Hi Kevin,
>
> Ok, I gave it a try on a natty slice and although I did not encounter
> the same traceback that you posted I did see some Tracebacks in my logs.
>
> - On my first step, the stats-reporter completed but throwing some errors
> on stdout which I believe to be unrelated.
> Error sample:
> ERROR: 184.106.119.94:6041/sdb4: 15 seconds
> ERROR: 184.106.119.94:6011/sdb1: 15 seconds
> ERROR: 184.106.119.94:6021/sdb2: 15 seconds
> ERROR: 184.106.119.94:6021/sdb2: 15 seconds
>
> The log files though showed some actual errors.
> Error Sample:
>
> contatiner-server.error log
> ==========================
> Jul 19 00:12:39 saio-natty account-server STDOUT: hub.switch()
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/pymodules/python2.7/eventlet/hubs/hub.py", line 177, in switch
> Jul 19 00:12:39 saio-natty account-server STDOUT: return
> self.greenlet.switch()
> Jul 19 00:12:39 saio-natty account-server STDOUT: LockTimeout: 10 seconds:
> /srv/node/sdb4/accounts/110358/d5e/6bc5ab18a40b60c23020fc56b52abd5e
> Jul 19 00:12:39 saio-natty account-server STDOUT: Traceback (most recent
> call last):
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/pymodules/python2.7/eventlet/greenpool.py", line 80, in
> _spawn_n_impl
> Jul 19 00:12:39 saio-natty account-server STDOUT: func(*args, **kwargs)
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 508, in
> process_request
> Jul 19 00:12:39 saio-natty account-server STDOUT: proto =
> self.protocol(socket, address, self)
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/python2.7/SocketServer.py", line 639, in __init__
> Jul 19 00:12:39 saio-natty account-server STDOUT: self.handle()
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/python2.7/BaseHTTPServer.py", line 343, in handle
> Jul 19 00:12:39 saio-natty account-server STDOUT: self.handle_one_request()
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 232, in
> handle_one_request
> Jul 19 00:12:39 saio-natty account-server STDOUT:
> self.handle_one_response()
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 336, in
> handle_one_response
> Jul 19 00:12:39 saio-natty account-server STDOUT: result =
> self.application(self.environ, start_response)
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/common/middleware/healthcheck.py",
> line 38, in __call__
> Jul 19 00:12:39 saio-natty account-server STDOUT: return self.app(env,
> start_response)
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/account/server.py",
> line 305, in __call__
> Jul 19 00:12:39 saio-natty account-server STDOUT: res = getattr(self,
> req.method)(req)
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/account/server.py",
> line 97, in PUT
> Jul 19 00:12:39 saio-natty account-server STDOUT:
> req.headers['x-bytes-used'])
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/common/db.py",
> line 1348, in put_container
> Jul 19 00:12:39 saio-natty account-server STDOUT: with
> lock_parent_directory(self.pending_file, self.pending_timeout):
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
> Jul 19 00:12:39 saio-natty account-server STDOUT: return self.gen.next()
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/local/lib/python2.7/dist-packages/swift-1.4.1-py2.7.egg/swift/common/utils.py",
> line 615, in lock_path
> Jul 19 00:12:39 saio-natty account-server STDOUT: sleep(0.01)
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/pymodules/python2.7/eventlet/greenthread.py", line 30, in sleep
> Jul 19 00:12:39 saio-natty account-server STDOUT: hub.switch()
> Jul 19 00:12:39 saio-natty account-server STDOUT: File
> "/usr/lib/pymodules/python2.7/eventlet/hubs/hub.py", line 177, in switch
> Jul 19 00:12:39 saio-natty account-server STDOUT: return
> self.greenlet.switch()
> Jul 19 00:12:39 saio-natty account-server STDOUT: LockTimeout: 10 seconds:
> /srv/node/sdb4/accounts/110358/d5e/6bc5ab18a40b60c23020fc56b52abd5e
>
>
> - On my second step, I decided to update python-eventlet with the actual
> version that we use at the moment which is eventlet-0.9.15 . So I
> removed the package that came with natty and installed the 0.9.15 from
> source. Once the new version was installed no more tracebacks showed up
> in my logs. The errors "ERROR: 184.106.119.94:6041/sdb4: 15 seconds"
> persisted though but I think that may be related to something else, but
> I did not look too deep into it.
>
>
> Unless, you are trying to find out how swift works with python 2.7 and
> perhaps report bugs, provide patches or etc ... , I would suggest working
> with 10.04 which is the LTS version of ubuntu and the one that is
> recommended for a production environment.
>
> -
> Marcelo
> https://github.com/btorch/swift-saio.sh
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/swift/+question/165052/+confirm?answer_id=2
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/swift/+question/165052
>
> You received this question notification because you asked the question.
>

Did it work ok for you on 10.04 ? I haven't used 1.4.0

Moved to 10.04, working without issue...

Thanks,
-Kevin
On Wed, Jul 20, 2011 at 8:41 AM, Marcelo Martins <
<email address hidden>> wrote:

> Your question #165052 on OpenStack Object Storage (swift) changed:
> https://answers.launchpad.net/swift/+question/165052
>
> Marcelo Martins posted a new comment:
> Did it work ok for you on 10.04 ? I haven't used 1.4.0
>
> --
> You received this question notification because you asked the question.
>

cool, glad it works now

Ubuntu 11.10 with swift packages from ubuntu and SAIO install

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/usr/bin/swift-dispersion-report", line 183, in direct
    object_copies_found[found_count] += 1
IndexError: list index out of range
Removing descriptor: 6
Queried 2621 objects for dispersion reporting, 29s, 0 retries
0.00% of object copies found (0 of 10484)