Openstack NEWTON ceilometer service not available if aodh startet

Asked by Christian Petrasch on 2017-12-06

Hi,

i'm running a openstack newton cluster and i have the problem, that any time i start aodh services, the ceilomter services returns "Service unavailable 503 " after a ceilometer service call like "ceilometer meter-list".

It is reproduceable if i uninstall aodh everything is working fine

I see the errors in the following logs, but i don't know what are the connection to the unavailable service of ceilometer ??
Ceilometer

Does anybody has an idea ?

It would be really appreciated...

thanks a lot in advance..

greets

Christian

##########

Ubuntu 16.04-3

aodh-api (3.0.4-0ubuntu1~cloud1) ...
aodh-evaluator (3.0.4-0ubuntu1~cloud1) ...
aodh-expirer (3.0.4-0ubuntu1~cloud1) ...
aodh-listener (3.0.4-0ubuntu1~cloud1) ...
aodh-notifier (3.0.4-0ubuntu1~cloud1) ...

ceilometer-agent-central 1:7.1.1-0ubuntu1~cloud1
ceilometer-agent-notification 1:7.1.1-0ubuntu1~cloud1
ceilometer-api 1:7.1.1-0ubuntu1~cloud1
ceilometer-collector 1:7.1.1-0ubuntu1~cloud1
ceilometer-common 1:7.1.1-0ubuntu1~cloud1

Conf aodh:

[DEFAULT]

#
# From aodh
#

auth_strategy = keystone
transport_url = rabbit://os_rabbit_user:PW@controller

[api]

port = 8042
host = 172.24.104.21
paste_config = api_paste.ini

[database]

connection = mysql+pymysql://aodh_db_user:PW@controller/aodh_db

[keystone_authtoken]

#
# From keystonemiddleware.auth_token
#

project_domain_name = default
user_domain_name = default
project_name = service
user_name = aodh_user
password = PW
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password

[service_credentials]

region_name = RegionOne
interface = internalURL
auth_type = password
auth_url = http://controller:5000/v3
project_name = service
username = aodh_user
user_domain_name = default
password = PW

##############

api-paste.ini

# aodh API WSGI Pipeline
# Define the filters that make up the pipeline for processing WSGI requests
# Note: This pipeline is PasteDeploy's term rather than aodh's pipeline
# used for processing samples

# Remove authtoken from the pipeline if you don't want to use keystone authentication
[pipeline:main]
pipeline = cors request_id authtoken api-server

[app:api-server]
paste.app_factory = aodh.api.app:app_factory

[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
oslo_config_project = aodh

[filter:request_id]
paste.filter_factory = oslo_middleware:RequestId.factory

[filter:cors]
paste.filter_factory = oslo_middleware.cors:filter_factory
oslo_config_project = aodh

############

Logs aodh:

syslog

Dec 6 11:40:18 x01123 aodh-api[5905]: ********************************************************************************
Dec 6 11:40:18 x01123 aodh-api[5905]: STARTING test server aodh.api.app.build_wsgi_app
Dec 6 11:40:18 x01123 aodh-api[5905]: Available at http://127.0.1.1:8042/
Dec 6 11:40:18 x01123 aodh-api[5905]: DANGER! For testing only, do not use in production
Dec 6 11:40:18 x01123 aodh-api[5905]: ********************************************************************************
Dec 6 11:40:18 x01123 systemd[1]: Reloading.
Dec 6 11:40:18 x01123 systemd[1]: Started ACPI event daemon.
Dec 6 11:40:18 x01123 systemd[1]: Reloading.
Dec 6 11:40:18 x01123 systemd[1]: Started ACPI event daemon.
Dec 6 11:40:18 x01123 systemd[1]: Starting OpenStack Aodh Notifier...
Dec 6 11:40:18 x01123 systemd[1]: Started OpenStack Aodh Notifier.
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.096 6330 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.172 6330 INFO aodh.evaluator [-] initiating evaluation cycle on 2 alarms
Dec 6 11:40:19 x01123 aodh-expirer[6120]: 2017-12-06 11:40:19.320 6120 INFO aodh.cmd.storage [-] Nothing to clean, database alarm history time to live is disabled
Dec 6 11:40:19 x01123 aodh-api[5905]: 2017-12-06 11:40:19.636 5905 ERROR keystonemiddleware.auth_token [-] Bad response code while validating token: 400
Dec 6 11:40:19 x01123 aodh-api[5905]: 2017-12-06 11:40:19.650 5905 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.", "code": 400, "title": "Bad Request"}}
Dec 6 11:40:19 x01123 aodh-api[5905]: 2017-12-06 11:40:19.651 5905 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Failed to fetch token data from identity server
Dec 6 11:40:19 x01123 aodh-api[5905]: 172.24.104.21 - - [06/Dec/2017 11:40:19] "GET / HTTP/1.1" 503 170
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold [-] alarm stats retrieval failed
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold Traceback (most recent call last):
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/aodh/evaluator/threshold.py", line 127, in _statistics
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold return self.cm_client.statistics.list(
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/aodh/evaluator/threshold.py", line 72, in cm_client
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold interface=auth_config.interface,
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 366, in get_client
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold return Client(version, endpoint, **kwargs)
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 313, in Client
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold return client_class(*args, **client_kwargs)
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 65, in __init__
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold self.alarm_client = self._get_alarm_client(**kwargs)
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 120, in _get_alarm_client
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold c.get("/")
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 187, in get
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold return self.request(url, 'GET', **kwargs)
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 473, in request
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold raise exc.from_response(resp, body)
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold HTTPServiceUnavailable: HTTPServiceUnavailable (HTTP 503)
Dec 6 11:40:19 x01123 aodh-evaluator[6021]: 2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold

aodh-api

2017-12-06 11:40:17.696 5905 INFO aodh.api.app [-] Full WSGI config used: /etc/aodh/api_paste.ini
2017-12-06 11:40:19.636 5905 ERROR keystonemiddleware.auth_token [-] Bad response code while validating token: 400
2017-12-06 11:40:19.650 5905 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.", "code": 400, "title": "Bad Request"}}
2017-12-06 11:40:19.651 5905 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Failed to fetch token data from identity server
2017-12-06 11:40:19.956 5905 ERROR keystonemiddleware.auth_token [-] Bad response code while validating token: 400
2017-12-06 11:40:19.957 5905 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.", "code": 400, "title": "Bad Request"}}

aodh-evaluator

2017-12-06 11:40:19.096 6330 WARNING oslo_db.sqlalchemy.utils [-] Unique keys not in sort_keys. The sorting order may be unstable.
2017-12-06 11:40:19.172 6330 INFO aodh.evaluator [-] initiating evaluation cycle on 2 alarms
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold [-] alarm stats retrieval failed
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold Traceback (most recent call last):
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/aodh/evaluator/threshold.py", line 127, in _statistics
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold return self.cm_client.statistics.list(
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/aodh/evaluator/threshold.py", line 72, in cm_client
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold interface=auth_config.interface,
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 366, in get_client
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold return Client(version, endpoint, **kwargs)
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 313, in Client
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold return client_class(*args, **client_kwargs)
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 65, in __init__
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold self.alarm_client = self._get_alarm_client(**kwargs)
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 120, in _get_alarm_client
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold c.get("/")
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 187, in get
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold return self.request(url, 'GET', **kwargs)
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 473, in request
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold raise exc.from_response(resp, body)
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold HTTPServiceUnavailable: HTTPServiceUnavailable (HTTP 503)
2017-12-06 11:40:19.656 6330 ERROR aodh.evaluator.threshold
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold [-] alarm stats retrieval failed
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold Traceback (most recent call last):
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/aodh/evaluator/threshold.py", line 127, in _statistics
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold return self.cm_client.statistics.list(
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/aodh/evaluator/threshold.py", line 72, in cm_client
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold interface=auth_config.interface,
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 366, in get_client
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold return Client(version, endpoint, **kwargs)
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 313, in Client
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold return client_class(*args, **client_kwargs)
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 65, in __init__
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold self.alarm_client = self._get_alarm_client(**kwargs)
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 120, in _get_alarm_client
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold c.get("/")
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 187, in get
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold return self.request(url, 'GET', **kwargs)
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 473, in request
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold raise exc.from_response(resp, body)
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold HTTPServiceUnavailable: HTTPServiceUnavailable (HTTP 503)
2017-12-06 11:40:19.962 6330 ERROR aodh.evaluator.threshold

######################

ceilometer conf

[DEFAULT]

auth_strategy = keystone

[api]

port = 8777

[database]

connection = mongodb://ceilometer_db_user:PW@controller:27017/ceilometer_db

[keystone_authtoken]

auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer_user
password = PW

[oslo_messaging_rabbit]

rabbit_host = controller
rabbit_userid = os_rabbit_user
rabbit_password = PW

[service_credentials]

auth_type = password
auth_url = http://controller:5000/v3
project_domain_name = default
user_domain_name = default
project_name = service
username = ceilometer_user
password = PW
interface = internalURL
region_name = RegionOne

#########################

api-paste.ini ceilometer

# Ceilometer API WSGI Pipeline
# Define the filters that make up the pipeline for processing WSGI requests
# Note: This pipeline is PasteDeploy's term rather than Ceilometer's pipeline
# used for processing samples

# Remove authtoken from the pipeline if you don't want to use keystone authentication
[pipeline:main]
pipeline = cors http_proxy_to_wsgi request_id authtoken api-server

[app:api-server]
paste.app_factory = ceilometer.api.app:app_factory

[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory

[filter:request_id]
paste.filter_factory = oslo_middleware:RequestId.factory

[filter:cors]
paste.filter_factory = oslo_middleware.cors:filter_factory
oslo_config_project = ceilometer

[filter:http_proxy_to_wsgi]
paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory
oslo_config_project = ceilometer

########################

wsgi file apache2

Listen 8777

<VirtualHost *:8777>
    WSGIDaemonProcess ceilometer-api processes=2 threads=10 user=ceilometer group=ceilometer display-name=%{GROUP}
    WSGIProcessGroup ceilometer-api
    WSGIScriptAlias / "/var/www/cgi-bin/ceilometer/app"
    WSGIApplicationGroup %{GLOBAL}
    ErrorLog /var/log/apache2/ceilometer_error.log
    CustomLog /var/log/apache2/ceilometer_access.log combined
</VirtualHost>

WSGISocketPrefix /var/run/apache2

########################
ceilometer logs

I can't find any errors.. neither in ceilomter logs nor apache ceilo wsgi logs or in syslog :(

Question information

Language:
English Edit question
Status:
Open
For:
Ubuntu ceilometer Edit question
Assignee:
No assignee Edit question
Last query:
2017-12-06
Last reply:

Can you help with this problem?

Provide an answer of your own, or ask Christian Petrasch for more information if necessary.

To post a message you must log in.