Two errors: 1) 403 Forbidden, Authentication Failure. 2[Errno 111] Connection refused)

Asked by HP

I followed exactly instructions in the admin manual on openstack website, and got these errors. I have seen several people ask those questions, but did not see answers, so please help.

1. The very first time when I installed nova, when I used "euca-authorize -P icmp -t -1:-1 default" command, it seemed to be able to connect to nova-api, but nova-api threw an error: 403 Forbidden to resource. The nova-api lof file says: Authentication failed. The access key "..." could not be found. I know this is related to credentials, but I just did the zip file and unzip it, and append the novarc to .bashrc, and source .bashrc. What could possibly be wrong here? I have only a single node with nova api, network, objectstore, scheduler, compute rabbitmq...in a single Ubuntu 10.10 computer, no proxy. I also saw that openstack wiki website list this error, but did not see a specific answer of how to correct it.

2. Now second, this error has been driving me nuts for almost two weeks. As I said, the very first time I installed nova without rebooting my computer, I got the error in 1). If I reboot my computer or restart any nova-services (api, network...) then the euca-authorize throws this: [Errno 111] Connection refused. I checked all services: all are up, including rabbitmq. No proxy, all .bashrc has been sourced. Check all IPs in /etc/nova/nova.conf and in .bashrc, they are all the same; just one IP. Tried with both mysql and sqlite, still the same error. My internet is wifi, but I think this should not matter since everything is local so far. Also, after rebooting my computer or any nova services, I see ABSOLUTELY no new log info in nova-api.log in /var/log/nova/nova-api.log; the only log info here was added before I restarted my computer after installing nova. Rabbitmq is up and no sign of error in its log.

Below are the portion in .bashrc file that was copied from novarc in novacreds.zip. I also copied the other files from this zip to /root/ because when I echoed $EC2_PRIVATE_KEY I got "/root/pk.pem"... so I copied those to /root/. User name and project name are the same: "test"

.bashrc:
NOVARC=$(readlink -f "${BASH_SOURCE:-${0}}" 2>/dev/null) ||
    NOVARC=$(python -c 'import os,sys; print os.path.abspath(os.path.realpath(sys.argv[1]))' "${BASH_SOURCE:-${0}}")
NOVA_KEY_DIR=${NOVARC%/*}
export EC2_ACCESS_KEY="f79ef6a5-6815-484c-99cb-2d6885f54520:test"
export EC2_SECRET_KEY="5ee0fad7-a236-432c-9110-8eeb9f1c9744"
export EC2_URL="http://192.168.23.93:8773/services/Cloud"
export S3_URL="http://192.168.23.93:3333"
export EC2_USER_ID=42 # nova does not use user id, but bundling requires it
export EC2_PRIVATE_KEY=${NOVA_KEY_DIR}/pk.pem
export EC2_CERT=${NOVA_KEY_DIR}/cert.pem
export NOVA_CERT=${NOVA_KEY_DIR}/cacert.pem
export EUCALYPTUS_CERT=${NOVA_CERT} # euca-bundle-image seems to require this set
alias ec2-bundle-image="ec2-bundle-image --cert ${EC2_CERT} --privatekey ${EC2_PRIVATE_KEY} --user 42 --ec2cert ${NOVA_CERT}"
alias ec2-upload-bundle="ec2-upload-bundle -a ${EC2_ACCESS_KEY} -s ${EC2_SECRET_KEY} --url ${S3_URL} --ec2cert ${NOVA_CERT}"
export NOVA_API_KEY="f79ef6a5-6815-484c-99cb-2d6885f54520"
export NOVA_USERNAME="test"
export NOVA_PROJECT_ID="test"
export NOVA_URL="http://192.168.23.93:8774/v1.0/"

nova.conf
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose=1
--s3_host=192.168.23.93
--rabbit_host=192.168.23.93
--ec2_api=192.168.23.93
--ec2_url=http://192.168.23.93:8773/services/Cloud
--fixed_range=192.168.22.0/23
--network_size=8
--routing_source_ip=192.168.23.93
--sql_connection=sqlite:////root/dev/sqlite/nova1.sqlite

Question information

Language:
English Edit question
Status:
Expired
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
HP (huy-pham-sc) said :
#1

Oops, a typo: In the EC2_ACCESS_KEY, the project name is appended to it, which is test. I forgot to change it to hide my name when I post here. plz, do not pay attention to that.

Revision history for this message
HP (huy-pham-sc) said :
#2

More info. I also tried deleting the entire "nova" database, then create a new database, create new nova user and nova project, re-generate new credentials again and again...still crying :(

Revision history for this message
Shweta P (shweta-ap05) said :
#3

The main issue in my case was the the proxy was set so i had to unset it.
If the port is open, then unset the proxy on the machine where the euca-authorize cmmand is run
export http_proxy=

But in general I guess you can do this to find where the problem occurs.
Check if the port 8773 is open on the controller
telnet contoller_ip 8773
Or on the controller
netstat -nap | grep 8773
If the port is not listening then there is a problem with the nova-api.Try to restart the services or check the nova.conf file. Sometimes an error in it may cause the services to not start

Thats a good link for the install

http://cssoss.wordpress.com/2011/04/27/openstack-beginners-guide-for-ubuntu-11-04-installation-and-configuration/

Revision history for this message
HP (huy-pham-sc) said :
#4

ugh, port 8773 is not being listened. The nova.conf is fine, and it should be because nova-api was running before I rebooted my computer. The bullsh*t Errno 111 only happened after rebooting.

So your previous problem was because of proxy? I saw that post.

Revision history for this message
HP (huy-pham-sc) said :
#5

I cannot even telnet to my computer, even though it's the same machine, but I can ping to it though. The port is open anyway, even after I re-installed nova-api. I think nova-api might use some info in the var/lib/nova.
 Did you follow the installation instruction in the website whose link is the one you gave above?

Revision history for this message
HP (huy-pham-sc) said :
#6

so now I unintalled all nova-services and try not to reboot them or reboot my computer, the euca now can connect to nova-api, but still "403 Forbidden":

Termial:
    Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
    EC2ResponseError: 403 Forbidden
    Access was denied to this source.

nova-api log:
    Authentication Failure: Access key ...... could not be found.
    192.168.0.140 GET /services/Cloud/ None:None 403 [Boto/1.9b (linux2)] text/plain text/html.

what the heck is wrong with my credentials? I found the access key in the database.
All my love for people who solve these problems. (Still 2 problems unsolved, but trying to fix this "403..." first).

Revision history for this message
haynes davis (haynes-davis) said :
#7

reinstalling didnt help me. nova-api is dying after sometime. any hint how to debug?

Revision history for this message
swap-zone (gros-sascha) said :
#8

It's a bit late, but maybe I can help you:

1) The "403 Forbidden" error is caused by missing credentials. Just specify the directory of your credential files manually in the ~/.bashrc:
Delete the line:
NOVA_KEY_DIR=${NOVARC%/*}
And write instead:
NOVA_KEY_DIR=/root/creds/
Log off and back in again and the error should be gone.

2) I'm not sure about that, but the following line in the nova.conf might cause problems:
--sql_connection=sqlite:////root/dev/sqlite/nova1.sqlite
You should specify an URL since it several nodes have to access the database (I know, not in your single node config). So try the use the URL of your controller node.

I hope this helped.

Revision history for this message
Launchpad Janitor (janitor) said :
#9

This question was expired because it remained in the 'Needs information' state without activity for the last 15 days.