Sahara

Got "INTERNAL SERVER ERROR" when trying to create a cluster

Asked by Senhua Huang

10.10.1.2 - - [22/Apr/2013 17:32:27] "POST /v0.2/d44ee75e1ae746d899f0c8703ddb7946/clusters HTTP/1.1" 500 258 0.047119
(30704) accepted ('10.10.1.2', 38939)
2013-04-22 17:35:07 ERROR [savanna.utils.api] Request aborted with status code 500 and message 'Error occurred during validation'
2013-04-22 17:35:07 ERROR [savanna.utils.api] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/savanna/service/validation.py", line 106, in handler
    validate_func(api_u.request_data())
  File "/usr/local/lib/python2.7/dist-packages/savanna/service/validation.py", line 146, in validate_cluster_create
    jsonschema.validate(cluster_values, CLUSTER_CREATE_SCHEMA)
  File "/usr/local/lib/python2.7/dist-packages/jsonschema.py", line 791, in validate
    cls.check_schema(schema)
  File "/usr/local/lib/python2.7/dist-packages/jsonschema.py", line 163, in check_schema
    error.message, validator=error.validator, path=error.path,
SchemaError: ['name', 'base_image_id', 'node_templates'] is not of type u'boolean'

The Request I sent was constructed using Python.
     url = "http://%s:8080/v0.2/%s/clusters" %(host, tenant_id)
    print "URL: %s" %url
    req = urllib2.Request(url)
    req.add_header('Content-Type', 'application/json')
    req.add_header('Accept', 'application/json')
    req.add_header('X-Auth-Token', auth_token)
    cluster_data = {"cluster": {
                           "name": "hdp",
                           "node_templates": {
                               "jt_nn.small": 1,
                               "tt_dn.small": 3
                           },
                           "base_image_id": "39e2dcda-8104-4343-aa4d-f701f7012ec2"
                       }
                   }
    req.add_data(json.dumps(cluster_data))
    r1 = urllib2.urlopen(req)
    data1 = json.loads(r1.read())

The client side got the following error:
  File "tools/create_cluster.py", line 93, in main
    r1 = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

Any hints?

Question information

Language:
English Edit question
Status:
Solved
For:
Sahara Edit question
Assignee:
Sergey Lukjanov Edit question
Last query:
Last reply:
Revision history for this message
Sergey Lukjanov (slukjanov) said : 		#1

Hi Senhua,

it looks like validation error, but the validation error isn't useful:
SchemaError: ['name', 'base_image_id', 'node_templates'] is not of type u'boolean'.

Could you, please, specify which version of savanna you are using? release number or commit hash if ok.

Sergey.

Revision history for this message
Senhua Huang (senhuang) said : 		#2

Hi Sergey,

The commit id is:
commit af58173a4ce7ef439c6e9b50af485b8758079542
Merge: 02ce954 db7b8df
Author: Jenkins <email address hidden>
Date: Mon Apr 22 12:46:51 2013 +0000

    Merge "Tenants support implemented for clusters."

Thanks,
Senhua

On 4/25/13 5:16 AM, "Sergey Lukjanov"
<email address hidden> wrote:

>Your question #227258 on Savanna changed:
>https://answers.launchpad.net/savanna/+question/227258
>
> Status: Open => Answered
>
>Sergey Lukjanov proposed the following answer:
>Hi Senhua,
>
>it looks like validation error, but the validation error isn't useful:
>SchemaError: ['name', 'base_image_id', 'node_templates'] is not of type
>u'boolean'.
>
>Could you, please, specify which version of savanna you are using?
>release number or commit hash if ok.
>
>Sergey.
>
>--
>If this answers your question, please go to the following page to let us
>know that it is solved:
>https://answers.launchpad.net/savanna/+question/227258/+confirm?answer_id=
>0
>
>If you still need help, you can reply to this email or go to the
>following page to enter your feedback:
>https://answers.launchpad.net/savanna/+question/227258
>
>You received this question notification because you asked the question.

Revision history for this message
Sergey Lukjanov (slukjanov) said : 		#3

Could you, please, execute "pip freeze" in your virtual environment?

It's a very strange error... Currently I have now ideas about the reason of it... Is it reproducible?

Revision history for this message
nanhaihappy (nanhaihappy) said : 		#4

I commented out 'jsonschema.validate(cluster_values, CLUSTER_CREATE_SCHEMA)' statement,I do not know that there is no effect on,but the problem is solved.

Revision history for this message
Alexander Ignatov (aignatov) said : 		#5

I have an idea what happens with jsonschema.
It seems nanhaihappy and Senhua Huang had version of jsonschema lower than 1.0.
The issue is that before version 1.0 "required" attributes (boolean values) were in the schema field defenition. After 1.0 required attribute are the list of fields of schema parameters to validate. We are using this way.
To confirm this idea we need to see "pip freeze" on the environment where issue happened.
Also we need to set range > 1.0 in the piprequires for jsonschema lib.

Revision history for this message
Sergey Lukjanov (slukjanov) said : 		#6

BTW: https://github.com/stackforge/savanna/blob/master/tools/pip-requires#L4

Revision history for this message
kesten broughton (dathomir) said : 		#7

The trace is fairly helpful.
The error comes from _add_details on the setattr line.
Keypairs inherits from base.Resource in base.py.
Note the difference and the comment (setattr is surrounded with a try catch in base.Resource.
If you wrap the setattr in KeyPair's _add_details, the error goes away.

To fix:
locate v1_1/keypairs.py
for me they were located here:
/opt/stack/python-novaclient/novaclient/v1_1/keypairs.py
/home/stack/savanna-venv/lib/python2.7/site-packages/novaclient/v1_1/keypairs.py

I also deleted the .pyc versions of these files, though that shouldn't be necessary.

Test it first on
nova keypair-list
if that succeeds then stop the savana process and restart (from the parent directory to savanna-venv) with
savanna-venv/bin/python savanna-venv/bin/savanna-api --config-file savanna-venv/etc/savanna.conf

# KeyPair in keypairs.py
    def _add_details(self, info):
        dico = 'keypair' in info and \
            info['keypair'] or info
        for (k, v) in dico.items(): ## wrap this for loop in a try/catch like in base.Resource
            setattr(self, k, v)

# base.Resource in base.py
    def _add_details(self, info):
        for (k, v) in six.iteritems(info):
            try:
                setattr(self, k, v)
                self._info[k] = v
            except AttributeError:
                # In this case we already defined the attribute on the class
                pass