OpenStack Heat

Could not get secret on group scaling

Asked by Davide Michelino

When heat tries to scale a group after an alarm, it is unable to launch the new instance.
I think that it can't get the right credentials and uses a null password.

engine.log reports:
2013-07-31 17:14:33.956 4279 DEBUG heat.engine.service [-] Periodic watcher task for stack 8b1a4382-a09f-473f-b197-c41531694484 _periodic_watcher_task /usr/lib/python2.6/site-packages/heat/engine/service.py:511
2013-07-31 17:14:33.966 4279 INFO heat.engine.watchrule [-] WATCH: stack:8b1a4382-a09f-473f-b197-c41531694484, watch_name:wp-single-HA-test261.MEMAlarmHigh ALARM
2013-07-31 17:14:34.017 4279 INFO heat.engine.watchrule [-] WATCH: stack:8b1a4382-a09f-473f-b197-c41531694484, watch_name:wp-single-HA-test261.MEMAlarmLow NORMAL
2013-07-31 17:14:34.017 4279 INFO heat.engine.watchrule [-] no action for new state NORMAL
2013-07-31 17:14:34.023 4279 INFO heat.engine.resources.autoscaling [-] WebServerScaleUpPolicy Alarm, adjusting Group WebServerGroup by 1
2013-07-31 17:14:34.026 4279 DEBUG heat.engine.resources.autoscaling [-] adjusting capacity from 1 to 2 resize /usr/lib/python2.6/site-packages/heat/engine/resources/autoscaling.py:182
2013-07-31 17:14:34.033 4279 INFO heat.engine.resources.autoscaling [-] creating inst
2013-07-31 17:14:34.033 4279 INFO heat.engine.resource [-] creating GroupedInstance "WebServerGroup-1"
2013-07-31 17:14:34.063 4279 WARNING heat.engine.resources.user [-] could not get secret for wp-single-HA-test261.CfnUser Error:Authorization Failed: Unable to communicate with identity service: {"error": {"message": "crypt() argument 1 must be string without null bytes, not str", "code": 400, "title": "Bad Request"}}. (HTTP 400)
2013-07-31 17:14:34.063 4279 INFO heat.engine.resources.user [-] wp-single-HA-test261.WebServerKeys.GetAtt(SecretAccessKey) == <SANITIZED>
2013-07-31 17:14:34.096 4279 ERROR heat.engine.resource [-] create GroupedInstance "WebServerGroup-1"
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource Traceback (most recent call last):
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 320, in create
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource self.handle_create()
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resources/instance.py", line 253, in handle_create
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource keypairs = [k.name for k in self.nova().keypairs.list()]
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 286, in nova
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource return self.stack.clients.nova(service_type)
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/clients.py", line 104, in nova
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource client.authenticate()
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/v1_1/client.py", line 169, in authenticate
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource self.client.authenticate()
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 330, in authenticate
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource auth_url = self._v2_auth(auth_url)
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 407, in _v2_auth
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource self._authenticate(url, body)
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 419, in _authenticate
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource **kwargs)
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 199, in _time_request
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource resp, body = self.request(url, method, **kwargs)
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 193, in request
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource raise exceptions.from_response(resp, body, url, method)
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource BadRequest: crypt() argument 1 must be string without null bytes, not str (HTTP 400)
2013-07-31 17:14:34.096 4279 TRACE heat.engine.resource

and keystone.log reports:
2013-07-31 17:14:35 DEBUG [keystone.common.wsgi] {"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "4\u0000\u0000\u00002\u0000\u0000\u00009\u0000\u0000\u0000f\u0000\u0000\u00003\u0000\u0000\u00002\u0000\u0000\u0000b\u0000\u0000\u00008\u0000\u0000\u00006\u0000\u0000\u00008\u0000\u0000\u0000a\u0000\u0000\u0000d\u0000\u0000\u00004\u0000\u0000\u00002\u0000\u0000\u00009\u0000\u0000\u0000a\u0000\u0000\u0000"}}}

I think that everything is configured fine, heat can create successfully CfnUsers on the keystone, and the instances are able to get ec2 credentials without any problem connecting to cloudwatch.

Why it is unable to get the credentials in this case? Why it tries to authenticate using the "admin" username?

Thanks!

Question information

Language:
English Edit question
Status:
Open
For:
OpenStack Heat Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Steve Baker (steve-stevebaker) said : 		#1

2 questions:
- what version of heat are you running?
- did you specify a username and password when you launched the stack? (not just an auth_token?)

Revision history for this message
Davide Michelino (davide-m) said : 		#2

I'm running heat 2013.1.2-3 from grizzly rdo repository.

I'm using the administration credentials from my system environment.

My environment:
OS_AUTH_URL="http://CONTROLLER_HOST:35357/v2.0/"
OS_PASSWORD="ADMIN_PASSWD"
OS_TENANT_NAME="admin"
OS_USERNAME="admin"

Client command:

$ heat stack-create -f AutoScalingGroupTest.tempalte -P "DBUsername=user;DBPassword=pass;KeyName=mykey;" test-stack

This is my context debug (if it could be useful):

2013-08-01 10:40:02.488 4279 DEBUG heat.openstack.common.rpc.amqp [-] unpacked context: {'username': None, 'service_user': u'heat', 'service_tenant': u'services', 'roles': [u'_member_', u'heat_stack_user'], 'aws_auth_uri': u'http://CONTROLLER_HOST:5000/v2.0/ec2tokens', 'tenant_id': u'ADMIN_TENANT_ID', 'auth_token': '<SANITIZED>', 'service_password': u'HEAT_PASS', 'auth_url': u'http://CONTROLLER_HOST:5000/v2.0', 'is_admin': True, 'password': None, 'aws_creds': u'{"ec2Credentials": {"access": "EC2_ACCESS", "signature": "EC2_SIGNATURE"}}', 'tenant': u'admin'} _safe_log /usr/lib/python2.6/site-packages/heat/openstack/common/rpc/common.py:276

Is it right to use administration credentials (admin user on admin tenant) to launch the stack?

Revision history for this message
Davide Michelino (davide-m) said : 		#3

Reading carefully the keystone log I realize that the username\password that heat provides are right: "4\u0000\u0000\u00002\u0000\u0000\u00009\u0000\u0000\u0000f\u0000\u0000\u00003\u0000\u0000\u00002\u0000\u0000\u0000b\u0000\u0000\u00008\u0000\u0000\u00006\u0000\u0000\u00008\u0000\u0000\u0000a\u0000\u0000\u0000d\u0000\u0000\u00004\u0000\u0000\u00002\u0000\u0000\u00009\u0000\u0000\u0000a\u0000\u0000\u0000" is exactly my admin password, but with three unicode null-char between each character.

Maybe the problem is not heat?

This is the keystone log:

2013-07-30 13:54:51 ERROR [root] crypt() argument 1 must be string without null bytes, not str
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 236, in __call__
    result = method(context, **params)
  File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 79, in authenticate
    context, auth)
  File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 301, in _authenticate_local
    tenant_id=tenant_id)
  File "/usr/lib/python2.6/site-packages/keystone/common/manager.py", line 47, in _wrapper
    return f(*args, **kw)
  File "/usr/lib/python2.6/site-packages/keystone/identity/backends/sql.py", line 189, in authenticate
    if not self._check_password(password, user_ref):
  File "/usr/lib/python2.6/site-packages/keystone/identity/backends/sql.py", line 170, in _check_password
    return utils.check_password(password, user_ref.get('password'))
  File "/usr/lib/python2.6/site-packages/keystone/common/utils.py", line 135, in check_password
    return passlib.hash.sha512_crypt.verify(password_utf8, hashed)
  File "/usr/lib/python2.6/site-packages/passlib/utils/handlers.py", line 465, in verify
    return self.checksum == self.calc_checksum(secret)
  File "/usr/lib/python2.6/site-packages/passlib/handlers/sha2_crypt.py", line 500, in _calc_checksum_os_crypt
    ok, result = safe_os_crypt(secret, self.to_string(native=False))
  File "/usr/lib/python2.6/site-packages/passlib/utils/__init__.py", line 197, in safe_os_crypt
    return True, os_crypt(secret, hash).decode("ascii")
  File "/usr/lib64/python2.6/crypt.py", line 55, in crypt
   return _crypt.crypt(word, salt)
TypeError: crypt() argument 1 must be string without null bytes, not str

Maybe I have wrong version of any library?

Thanks!

Revision history for this message
Davide Michelino (davide-m) said : 		#4

I did a little and quick hack to fix this issue.

I modified the code of heat/db/sqlalchemy/api.py at line 221, stripping out the null-chars

from

result['password'] = crypt.decrypt(result['password'])

to

result['password'] = crypt.decrypt(result['password']).replace("\x00", "")

Now seems to work fine.
But why I have this issue? Is it heat bug?

Revision history for this message
Zane Bitter (zaneb) said : 		#5

This is clearly an encoding problem - we're getting unicode out but treating it as an ascii/utf-8 string. The answer is definitely not just stripping out the null characters, the correct fix would be to make sure the string is encoded as utf-8 _before_ it is encrypted.

Revision history for this message
Davide Michelino (davide-m) said : 		#6

Yes, it works as well modifying user_creds_create() in heat/db/sqlalchemy/api.py at line 209

from

user_creds_ref.password = crypt.encrypt(values['password'])

to

user_creds_ref.password = crypt.encrypt(values['password'].encode('utf-8'))

Should I create a bug report?

Revision history for this message
Masaru Hoshi (hoshi-h) said : 		#7

I'm facing a similar problem with my password.

I tried davide suggestion but even though my password still contains the \x00 characters.

After browsing this problem around, I found out we may all be facing a unicode conversion problem caused probably by the crypt module.

http://stackoverflow.com/questions/10253405/python-x00-filled-utf-32-string-from-cstringio

Revision history for this message
Davide Michelino (davide-m) said : 		#8

Try replacing that line with

user_creds_ref.password = crypt.encrypt(unicode(values['password']).encode('utf-8'))

and then restart all heat services.

This workaround for me is working well.

Here there is a proposed patch: https://review.openstack.org/#/c/46287/1/heat/db/sqlalchemy/api.py

Can you help with this problem?

Provide an answer of your own, or ask Davide Michelino for more information if necessary.

To post a message you must log in.