internal server error

Asked by meenakshi

Hi,

    I installed openstack using devstack (single node install). And i am able to use the openstack dashboard. I could successfully create users as well, but i am not able to create networks. On submitting the 'create network' button, i got the following error :-

   There was an error submitting the form. Please try again.

Also, i tried creating a network using the cli :-

    quantum net-create net1

But it gave the following error:-

   Request failed: internal server error while processing your request.

Question information

Language:
English Edit question
Status:
Answered
For:
neutron Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
meenakshi (msarma00) said :
#1

Anyone.. please help me solve this problem.

Revision history for this message
Eugene Nikanorov (enikanorov) said :
#2

Can you go to devstack directory and run ./rejoin_stack.sh
That will bring up screen session with tabs related to all openstack services running.
Switch to the tab with name q-svc (ctrl-a-<num> to switch to tab #N, or ctrl-a-n to switch to the next tab)
Then try to create network again.
You should see some errors printed in quantum-server log.
Please provide them for further analyzis

Revision history for this message
meenakshi (msarma00) said :
#4

On giving the command : quantum net-create net1
This is the error log:-

2013-03-21 09:42:36 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/opt/stack/quantum/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/opt/stack/quantum/quantum/common/rpc.py", line 43, in dispatch
    quantum_ctxt, version, method, **kwargs)
  File "/opt/stack/quantum/quantum/openstack/common/rpc/dispatcher.py", line 136, in dispatch
    raise AttributeError("No such RPC function '%s'" % method)
AttributeError: No such RPC function 'sync_routers'
2013-03-21 09:42:36 ERROR [quantum.openstack.common.rpc.common] Returning exception No such RPC function 'sync_routers' to caller
2013-03-21 09:42:36 ERROR [quantum.openstack.common.rpc.common] ['Traceback (most recent call last):\n', ' File "/opt/stack/quantum/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data\n rval = self.proxy.dispatch(ctxt, version, method, **args)\n', ' File "/opt/stack/quantum/quantum/common/rpc.py", line 43, in dispatch\n quantum_ctxt, version, method, **kwargs)\n', ' File "/opt/stack/quantum/quantum/openstack/common/rpc/dispatcher.py", line 136, in dispatch\n raise AttributeError("No such RPC function \'%s\'" % method)\n', "AttributeError: No such RPC function 'sync_routers'\n"]
2013-03-21 09:42:36 DEBUG [quantum.openstack.common.rpc.amqp] UNIQUE_ID is 0906bb7294444087b1bbbd4c2227f3ef.

And on creating a network using the dashboard, this is the error log:-

2013-03-21 09:48:36 DEBUG [quantum.openstack.common.rpc.amqp] received {u'_context_roles': [u'admin'], u'_msg_id': u'139770d475e340698e277f7863d24548', u'_context_read_deleted': u'no', u'_context_tenant_id': None, u'args': {u'host': u'tcs-VirtualBox', u'router_ids': None, u'fullsync': None}, u'_unique_id': u'c3f801c97a3d4537949194c1b0ad60fc', u'_context_is_admin': True, u'version': u'1.0', u'_context_project_id': None, u'_context_timestamp': u'2013-03-21 04:18:36.414111', u'_context_user_id': None, u'method': u'sync_routers'}
2013-03-21 09:48:36 DEBUG [quantum.openstack.common.rpc.amqp] unpacked context: {'user_id': None, 'roles': [u'admin'], 'tenant_id': None, 'is_admin': True, 'timestamp': u'2013-03-21 04:18:36.414111', 'project_id': None, 'read_deleted': u'no'}
2013-03-21 09:48:36 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/opt/stack/quantum/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/opt/stack/quantum/quantum/common/rpc.py", line 43, in dispatch
    quantum_ctxt, version, method, **kwargs)
  File "/opt/stack/quantum/quantum/openstack/common/rpc/dispatcher.py", line 136, in dispatch
    raise AttributeError("No such RPC function '%s'" % method)
AttributeError: No such RPC function 'sync_routers'
2013-03-21 09:48:36 ERROR [quantum.openstack.common.rpc.common] Returning exception No such RPC function 'sync_routers' to caller
2013-03-21 09:48:36 ERROR [quantum.openstack.common.rpc.common] ['Traceback (most recent call last):\n', ' File "/opt/stack/quantum/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data\n rval = self.proxy.dispatch(ctxt, version, method, **args)\n', ' File "/opt/stack/quantum/quantum/common/rpc.py", line 43, in dispatch\n quantum_ctxt, version, method, **kwargs)\n', ' File "/opt/stack/quantum/quantum/openstack/common/rpc/dispatcher.py", line 136, in dispatch\n raise AttributeError("No such RPC function \'%s\'" % method)\n', "AttributeError: No such RPC function 'sync_routers'\n"]
2013-03-21 09:48:36 DEBUG [quantum.openstack.common.rpc.amqp] UNIQUE_ID is 4378d9705e96435983a7931bd04b01df.

Revision history for this message
Eugene Nikanorov (enikanorov) said :
#5

That seems strange as if l3 agent code was newer than quantum server code

Could you try to do the following:
1) run ./unstack.sh
2) sudo rm -rf /opt/stack/quantum
3) ./stack.sh

Check if issue still exist.

Revision history for this message
meenakshi (msarma00) said :
#6

okay.. will do that. Also, do i have to enable internet connectivity this time when i run ./stack.sh ?

Revision history for this message
Eugene Nikanorov (enikanorov) said :
#7

Yes, internet access should be enabled as devstack checks version of all packages it needs and also pull openstack sources from github.com.

Revision history for this message
meenakshi (msarma00) said :
#8

stack.sh ended with the following :-

++ [[ stack == \u\n\s\t\a\c\k ]]
+ [[ -x /home/stack/devstack/local.sh ]]
+ service_check
+ local service
+ local failures
+ SCREEN_NAME=stack
+ SERVICE_DIR=/opt/stack/status
+ [[ ! -d /opt/stack/status/stack ]]
++ ls /opt/stack/status/stack/q-agt.failure
+ failures=/opt/stack/status/stack/q-agt.failure
+ for service in '$failures'
++ basename /opt/stack/status/stack/q-agt.failure
+ service=q-agt.failure
+ service=q-agt
+ echo 'Error: Service q-agt is not running'
Error: Service q-agt is not running
+ '[' -n /opt/stack/status/stack/q-agt.failure ']'
+ echo 'More details about the above errors can be found with screen, with ./rejoin-stack.sh'
More details about the above errors can be found with screen, with ./rejoin-stack.sh
+ set +o xtrace

Horizon is now available at http://10.0.2.15/
Keystone is serving at http://10.0.2.15:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: password
This is your host ip: 10.0.2.15
stack.sh completed in 727 seconds.

So q-agt is not working. How do i resolve that? Tell me if you require any more logs.

Revision history for this message
meenakshi (msarma00) said :
#9

And creating a network gives the same error:-

2013-03-21 12:28:19 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/opt/stack/quantum/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/opt/stack/quantum/quantum/common/rpc.py", line 43, in dispatch
    quantum_ctxt, version, method, **kwargs)
  File "/opt/stack/quantum/quantum/openstack/common/rpc/dispatcher.py", line 136, in dispatch
    raise AttributeError("No such RPC function '%s'" % method)
AttributeError: No such RPC function 'sync_routers'
2013-03-21 12:28:19 ERROR [quantum.openstack.common.rpc.common] Returning exception No such RPC function 'sync_routers' to caller
2013-03-21 12:28:19 ERROR [quantum.openstack.common.rpc.common] ['Traceback (most recent call last):\n', ' File "/opt/stack/quantum/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data\n rval = self.proxy.dispatch(ctxt, version, method, **args)\n', ' File "/opt/stack/quantum/quantum/common/rpc.py", line 43, in dispatch\n quantum_ctxt, version, method, **kwargs)\n', ' File "/opt/stack/quantum/quantum/openstack/common/rpc/dispatcher.py", line 136, in dispatch\n raise AttributeError("No such RPC function \'%s\'" % method)\n', "AttributeError: No such RPC function 'sync_routers'\n"]
2013-03-21 12:28:19 DEBUG [quantum.openstack.common.rpc.amqp] UNIQUE_ID is 48dc09d075a949b08c34267befb00e52.

Revision history for this message
Eugene Nikanorov (enikanorov) said :
#10

Please run ./rejoin-stack.sh
That will start screen session with all services that devstack starts. q-svc (quantum server) is usually on 6th tab, you can switch there by pressing ctrl-a-6 ("ctrl-a-n" to switch to the next tab, "ctrl-a-p" to switch to previous)
Check if there are logs explaining quantum server failure.
Also, could you provide contents your localrc file?

Revision history for this message
Eugene Nikanorov (enikanorov) said :
#11

I've mistaken, it's not q-svc, but q-agt (quantum agent) is failing, so you need to check logs on corresponding tab (it should be near tab with q-svc)

Revision history for this message
meenakshi (msarma00) said :
#12

This is the localrc file:-

ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-meta
enable_service q-l3
enable_service quantum
enable_service bigswitch_floodlight
Q_PLUGIN=bigswitch_floodlight

Revision history for this message
meenakshi (msarma00) said :
#13

and there is no q-agt tab there. Am i doing something wrong?

Revision history for this message
Salvatore Orlando (salvatore-orlando) said :
#14

The Big Switch plugin is a proxy to floodlight, which manages L2 & L3.
Agents should be not running at all, as I think tney're not required.

Your agents are crashing because the plugin does not implement the RPC functions they call for synchronization (e.g.: sync-routers)

Try removing q-agt, and q-l3 from your localrc file.

Also, try and have a look at this document: http://www.openflowhub.org/display/floodlightcontroller/Install+Floodlight+and+OpenStack+on+Your+Own+Ubuntu+VM

Revision history for this message
meenakshi (msarma00) said :
#15

Hi,
  i removed q-agt and q-l3 from localrc and then run stack.sh again. This time there was no error. But on trying to create a network, this was the error :-

  \n </div>\n\n <div id="explanation">\n <p>\n You\'re seeing this error because you have <code>DEBUG = True</code> in\n your Django settings file. Change that to <code>False</code>, and Django\n will display a standard 404 page.\n </p>\n </div>\n</body>\n</html>\n'
2013-03-21 14:22:29 DEBUG [quantum.openstack.common.lockutils] Released file lock "rest_call" at /tmp/tmpWPEamV/bsn-rest_call for method "rest_call"...
2013-03-21 14:22:29 ERROR [quantum.plugins.bigswitch.plugin] ServerProxy: POST failure for servers: ('localhost', 80)
2013-03-21 14:22:29 ERROR [quantum.plugins.bigswitch.plugin] ServerProxy: POST failure for all servers: (('localhost', 80),)
2013-03-21 14:22:29 ERROR [quantum.plugins.bigswitch.plugin] QuantumRestProxyV2:Unable to create remote network: Error in REST call to remote network controller: None
2013-03-21 14:22:29 ERROR [quantum.api.v2.resource] create failed
Traceback (most recent call last):
  File "/opt/stack/quantum/quantum/api/v2/resource.py", line 82, in resource
    result = method(request=request, **args)
  File "/opt/stack/quantum/quantum/api/v2/base.py", line 369, in create
    obj = obj_creator(request.context, **kwargs)
  File "/opt/stack/quantum/quantum/plugins/bigswitch/plugin.py", line 387, in create_network
    raise
TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType

Revision history for this message
meenakshi (msarma00) said :
#16

What's wrong with quantum now? Can anyone please help?

Revision history for this message
Salvatore Orlando (salvatore-orlando) said :
#17

I am afraid this traceback is not really helpful (probably a little bug in the plugin, but I'd check this with Sumit Naiksatam first).
Looking at the code it seems the floodlight proxy is returning an error code (anything not in 200-207 range), but this puzzles me too as in that case you should have more meesages in the log.

I am sorry but we need to hear from somebody more informed on the BSN plugin.

Revision history for this message
Salvatore Orlando (salvatore-orlando) said :
#18

Sorry, but I missed a few bits in your log that clearly state the REST call is failing,
Unfortunately, the log does not return the exact HTTP error code.

One thing to check would be the correctness of the information in /etc/quantum/plugins/bigswitch/restproxy.ini
In particular the 'servers' and 'server_auth' parameters.

Revision history for this message
meenakshi (msarma00) said :
#19

This is the restproxy.ini file :-

# Config file for quantum-proxy-plugin.

[DATABASE]
# This line MUST be changed to actually run the plugin.
# Example:
# sql_connection = mysql://root:pass@127.0.0.1:3306/restproxy_quantum
# Replace 127.0.0.1 above with the IP address of the database used by the
# main quantum server. (Leave it as is if the database runs on this host.)
sql_connection = mysql://root:password@localhost/restproxy_quantum?charset=utf8
# Database reconnection retry times - in event connectivity is lost
# set to -1 implies an infinite retry count
# sql_max_retries = 10
# Database reconnection interval in seconds - if the initial connection to the
# database fails
reconnect_interval = 2
# Enable the use of eventlet's db_pool for MySQL. The flags sql_min_pool_size,
# sql_max_pool_size and sql_idle_timeout are relevant only if this is enabled.
# sql_dbpool_enable = False
# Minimum number of SQL connections to keep open in a pool
# sql_min_pool_size = 1
# Maximum number of SQL connections to keep open in a pool
# sql_max_pool_size = 5
# Timeout in seconds before idle sql connections are reaped
# sql_idle_timeout = 3600

[RESTPROXY]
servertimeout = 10
# All configuration for this plugin is in section '[restproxy]'
#
# The following parameters are supported:
# servers : <host:port>[,<host:port>]* (Error if not set)
# server_auth : <username:password> (default: no auth)
# server_ssl : True | False (default: False)
# sync_data : True | False (default: False)
# server_timeout : 10 (default: 10 seconds)
#
servers=127.0.0.1:80
#server_auth=username:password
#server_ssl=True
#sync_data=True
#server_timeout=10

Revision history for this message
Eugene Nikanorov (enikanorov) said :
#20

Just wondering why [RESTPROXY] servers parameter is 127.0.0.1:80
Isn't this address used by Horizon dashboard?
Is it possible that your floodlight service is trying to use the same address as Horizon?
If so, you need to reconfigure either Horizon or floodlight service to use different ports.

Can you help with this problem?

Provide an answer of your own, or ask meenakshi for more information if necessary.

To post a message you must log in.