swift incoming and outgoing measurements

Asked by Tomasz Wanczyk

I have installed latest version of ceilometer from github on ubuntu 12.10 (swift 1.7.4) and there are any incoming and outgoing results of measurements in mongo db:

> db.resource.find()
{ "_id" : "8853b675830d49c39ac9fa30f82ccf0b", "metadata" : null, "meter" : [

 { "counter_name" : "storage.objects", "counter_unit" : "object", "counter_type" : "gauge" },

 { "counter_name" : "storage.objects.size", "counter_unit" : "B", "counter_type" : "gauge" },

 { "counter_name" : "storage.objects.containers", "counter_unit" : "container", "counter_type" : "gauge" } ], "project_id" : "8853b675830d49c39ac9fa30f82ccf0b", "received_timestamp" : ISODate("2013-02-10T21:24:59.189Z"), "source" : "openstack", "timestamp" : ISODate("2013-02-10T21:24:59Z"), "user_id" : null
}

{ "_id" : "964edc27774e432297fb7ebfe362c6a4", "metadata" : null, "meter" : [

{ "counter_name" : "storage.objects", "counter_unit" : "object", "counter_type" : "gauge" },

 { "counter_name" : "storage.objects.containers", "counter_unit" : "container", "counter_type" : "gauge" },

 { "counter_name" : "storage.objects.size", "counter_unit" : "B", "counter_type" : "gauge" } ], "project_id" : "964edc27774e432297fb7ebfe362c6a4", "received_timestamp" : ISODate("2013-02-10T21:24:59.460Z"), "source" : "openstack", "timestamp" : ISODate("2013-02-10T21:24:59Z"), "user_id" : null
}

Are those measurements implemented or something went wrong ?

Question information

Language:
English Edit question
Status:
Solved
For:
Ceilometer Edit question
Assignee:
No assignee Edit question
Solved by:
Julien Danjou
Solved:
Last query:
Last reply:
Revision history for this message
Julien Danjou (jdanjou) said :
#1

Yes, but you have to configure and enable the Ceilometer middleware in Swift. I'm not sure this is documented for now.

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#3

/etc/swift/proxy-server.conf

[pipeline:main]
pipeline = catch_errors healthcheck cache tempurl authtoken keystone ceilometer proxy-server

[filter:ceilometer]
use = egg:ceilometer#swift

Revision history for this message
Julien Danjou (jdanjou) said :
#4

Did you try to upload a file or something? That should be counted as far as I can see.

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#7

Yes I uploaded few small and big files

Revision history for this message
Julien Danjou (jdanjou) said :
#8

Also, is your ceilometer.conf correct? It's still read by the middleware.

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#9

I think so, I am mostly using default config, I have seen when rabbitmq is not running swift doesn't work OK, but when it is running swift works correctly but there are any incoming and outgoing traffic results in mongo. Are in ceilometer.conf some options responsible for that functionality?

Revision history for this message
Julien Danjou (jdanjou) said :
#10

No, there's no way to disable it via ceilometer.conf anyway. I don't see what could be wrong, I'd suggest to try to debug the middleware to see what's going on. :(

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#11

Maybe the reason is I am not using ceilometer-agent-compute in this setup? I have the same problem on debian 7 (swift 1.7.4 from github) and can't find solution :(

Revision history for this message
Julien Danjou (jdanjou) said :
#12

The agent compute has nothing to do with that.

Revision history for this message
Julien Danjou (jdanjou) said :
#13

I have investigated and this is actually bug #1124121

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#14

After applying patch, proxy-server is not starting, Any idea?

Revision history for this message
Best Julien Danjou (jdanjou) said :
#15

I think you may need this patch too: https://review.openstack.org/#/c/21873/

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#16

I have applied these 3 patches (agent, pipeline, swift_middleware) and result is the same. Swift-proxy is not starting. I have checked and install ceilometer from github (no patch applied) and swift-proxy is not starting.

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#17

I have upgraded swift to 1.7.6, next installed ceilometer (github) and swift-proxy is starting ( there are some debug messages like:

2013-02-19 15:57:34.229 1264 DEBUG ceilometer.pipeline [-] Pipeline config file: /etc/ceilometer/pipeline.yaml setup_pipeline /usr/local/lib/python2.7/dist-packages/c│
eilometer-2013.1.a318.g93e27b9-py2.7.egg/ceilometer/pipeline.py:372 │
2013-02-19 15:57:34.232 1264 INFO ceilometer.pipeline [-] Pipeline config: [{'publishers': ['meter_publisher'], 'interval': 60, 'transformers': None, 'name': 'meter_p│
ipeline', 'counters': ['*']}] │
2013-02-19 15:57:34.236 1264 INFO keystone.middleware.auth_token [-] Starting keystone auth_token middleware │
2013-02-19 15:57:34.237 1264 INFO keystone.middleware.auth_token [-] Using /var/lib/swift/keystone-signing as cache directory for signing certificate │
                                                                                                                                                                      │
# swift-init proxy status │
proxy-server running (1264 - /etc/swift/proxy-server.conf)

But something goes wrong becasue load average is growing and:

..
Feb 19 15:58:24 proxy-1659-9 swiftproxy Started child 2825
Feb 19 15:58:24 proxy-1659-9 swiftproxy Removing dead child 2817
Feb 19 15:58:24 proxy-1659-9 swiftproxy Started child 2826
Feb 19 15:58:24 proxy-1659-9 swiftproxy Removing dead child 2819
Feb 19 15:58:24 proxy-1659-9 swiftproxy Started child 2827
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2820
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2828
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2821
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2829
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2823
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2830
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2822
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2831
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2824
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2832
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2825
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2833
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2826
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2834
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2828
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2835
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2827
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2836
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2829
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2837
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2830
Feb 19 15:58:25 proxy-1659-9 swiftproxy Started child 2838
Feb 19 15:58:25 proxy-1659-9 swiftproxy Removing dead child 2831
...

Any idea? :)

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#18

Above was related to log_dir and log_file. Without these options (ceilometer.conf) problem doesn't exist.
Moreover, after applying patch set 7 most looking OK. There is one problem with ceilometer-agent-central which can't communicate with keystone. I am sure keystone is configured correctly. Using keystone client and passing the same options from ceilometer.conf I get token.

-------------
2013-02-19 23:03:13.096 13135 INFO ceilometer.openstack.common.rpc.common [-] Connected to AMQP server on localhost:5672
2013-02-19 23:03:13.097 13135 DEBUG ceilometer.openstack.common.rpc.service [-] Creating Consumer connection for Service ceilometer.agent.central start /usr/local/lib/python2.7/dist-packages/ceilometer-2013.1.a318.g93e27b9-py2.7.egg/ceilometer/openstack/common/rpc/service.py:48
2013-02-19 23:03:13.304 13135 INFO ceilometer.agent [-] Polling and publishing objectstore
2013-02-19 23:03:13.340 13135 AUDIT ceilometer.pipeline [-] Flush pipeline meter_pipeline
2013-02-19 23:03:13.341 13135 WARNING ceilometer.agent [-] Continuing after error from objectstore: Unable to communicate with identity service: {"error": {"message": "An unexpected error prevented the server from fulfilling your request. coercing to Unicode: need string or buffer, NoneType found", "code": 500, "title": "Internal Server Error"}}. (HTTP 500)
2013-02-19 23:03:13.342 13135 ERROR ceilometer.agent [-] Unable to communicate with identity service: {"error": {"message": "An unexpected error prevented the server from fulfilling your request. coercing to Unicode: need string or buffer, NoneType found", "code": 500, "title": "Internal Server Error"}}. (HTTP 500)
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent Traceback (most recent call last):
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/ceilometer-2013.1.a318.g93e27b9-py2.7.egg/ceilometer/agent.py", line 52, in publish_counters_from_one_pollster
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent p(ext.obj.get_counters(manager, *args, **kwargs))
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/ceilometer-2013.1.a318.g93e27b9-py2.7.egg/ceilometer/pipeline.py", line 79, in p
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent self.source)
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/ceilometer-2013.1.a318.g93e27b9-py2.7.egg/ceilometer/pipeline.py", line 251, in publish_counters
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent sorted(counters, key=lambda c: c.name),
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/ceilometer-2013.1.a318.g93e27b9-py2.7.egg/ceilometer/objectstore/swift.py", line 55, in get_counters
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent for tenant, account in self.iter_accounts(manager.keystone):
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/ceilometer-2013.1.a318.g93e27b9-py2.7.egg/ceilometer/objectstore/swift.py", line 107, in iter_accounts
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent for t in ksclient.tenants.list():
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/python_keystoneclient-0.2.2-py2.7.egg/keystoneclient/v2_0/tenants.py", line 116, in list
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent tenant_list = self._list("/tenants%s" % query, "tenants")
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/python_keystoneclient-0.2.2-py2.7.egg/keystoneclient/base.py", line 67, in _list
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent resp, body = self.api.get(url)
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/python_keystoneclient-0.2.2-py2.7.egg/keystoneclient/client.py", line 380, in get
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent return self._cs_request(url, 'GET', **kwargs)
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/python_keystoneclient-0.2.2-py2.7.egg/keystoneclient/client.py", line 376, in _cs_request
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent **kwargs)
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent File "/usr/local/lib/python2.7/dist-packages/python_keystoneclient-0.2.2-py2.7.egg/keystoneclient/client.py", line 338, in request
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent raise exceptions.from_response(resp, resp.text)
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent ClientException: Unable to communicate with identity service: {"error": {"message": "An unexpected error prevented the server from fulfilling your request. coercing to Unicode: need string or buffer, NoneType found", "code": 500, "title": "Internal Server Error"}}. (HTTP 500)
2013-02-19 23:03:13.342 13135 TRACE ceilometer.agent
---------------------

Revision history for this message
Tomasz Wanczyk (twanczyk) said :
#21

Thanks Julien Danjou, that solved my question.