Red Dwarf DevStack int-tests Hangs on WaitForGuestInstallationToFinish

Asked by suntr

After following https://wiki.openstack.org/wiki/Reddwarf-installation

and executing:

$ ~/reddwarf-integration/scripts/redstack post-devstack mysql
$ sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

$ ~/reddwarf-integration/scripts/redstack int-tests

Hangs forever until timeout @:

WaitForGuestInstallationToFinish
        test_instance_created

It seems this is occurring because the poller is convinced that the state is still BUILD.

$ ~/reddwarf-integration/scripts/redstack nova-client list

+--------------------------------------+---------------------------------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+---------------------------------+--------+------------------+
| 9f879821-566d-4ad7-a44f-3018b22d1314 | TEST_2013-04-10 17:00:50.986100 | ACTIVE | private=10.0.0.2 |
+--------------------------------------+---------------------------------+--------+------------------+

$ ~/reddwarf-integration/scripts/redstack nova-client show

+-------------------------------------+---------------------------------------------------------------+
| Property | Value |
+-------------------------------------+---------------------------------------------------------------+
| status | ACTIVE |
| updated | 2013-04-11T00:02:31Z |
| OS-EXT-STS:task_state | None |
| OS-EXT-SRV-ATTR:host | suntr-vb |
| key_name | None |
| image | precise_mysql_image (2d1b5fa5-caa4-4bbc-a00c-9447f0ca05cf) |
| private network | 10.0.0.2 |
| hostId | a819f57851bc97a4c47dae336478b3bfdd7aee769d6175c76002ecf6 |
| OS-EXT-STS:vm_state | active |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | suntr-vb |
| flavor | m1.tiny (1) |
| id | 9f879821-566d-4ad7-a44f-3018b22d1314 |
| security_groups | [{u'name': u'SecGroup_0354c20c-feff-4fed-822d-a8a0e651e983'}] |
| user_id | 558e6b73ae754a08aeea5d5ec40d6f2d |
| name | TEST_2013-04-10 17:00:50.986100 |
| created | 2013-04-11T00:00:54Z |
| tenant_id | 1e709b78937447ad971cc286e196752a |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+-------------------------------------+---------------------------------------------------------------+

$ ~/reddwarf-integration/scripts/redstack rd-client instance list

{
    "flavor": {
        "id": "1",
        "links": [
            {
                "href": "https://10.0.2.15:8779/v1.0/1e709b78937447ad971cc286e196752a/flavors/1",
                "rel": "self"
            },
            {
                "href": "https://10.0.2.15:8779/flavors/1",
                "rel": "bookmark"
            }
        ]
    },
    "id": "0354c20c-feff-4fed-822d-a8a0e651e983",
    "links": [
        {
            "href": "https://10.0.2.15:8779/v1.0/1e709b78937447ad971cc286e196752a/instances/0354c20c-feff-4fed-822d-a8a0e651e983",
            "rel": "self"
        },
        {
            "href": "https://10.0.2.15:8779/instances/0354c20c-feff-4fed-822d-a8a0e651e983",
            "rel": "bookmark"
        }
    ],
    "name": "TEST_2013-04-10 17:00:50.986100",
    "status": "BUILD",
    "volume": {
        "size": 1
    }
}

Trying to do a real mysql deployment stays hung in BUILD as well (so it's not isolated to int-tests)

Any ideas?

Specs:

> VirtualBox 4.2.10
> Host => Mac OS X 10.6.8 (Core i7) w/ 8 GB Mem
> Guest => Ubuntu 12.04.2 LTS (64-bit)

Red Dwarf DB Data:

mysql> select * from instances \G;
*************************** 1. row ***************************
                 id: 0354c20c-feff-4fed-822d-a8a0e651e983
            created: 2013-04-11 00:00:51
            updated: 2013-04-11 00:00:54
               name: TEST_2013-04-10 17:00:50.986100
           hostname: NULL
compute_instance_id: 9f879821-566d-4ad7-a44f-3018b22d1314
            task_id: 1
   task_description: No tasks for the instance.
    task_start_time: NULL
          volume_id: 984cdeb0-2bbc-482b-89fe-46030ca7db62
          flavor_id: 1
        volume_size: 1
          tenant_id: 1e709b78937447ad971cc286e196752a
      server_status: NULL
            deleted: 0
         deleted_at: NULL
1 row in set (0.00 sec)

ERROR:
No query specified

mysql> select * from service_statuses \G;
*************************** 1. row ***************************
                id: 04d0d7a2-d021-44d4-a272-a6c68cab6227
       instance_id: 0354c20c-feff-4fed-822d-a8a0e651e983
         status_id: 23
status_description: new
        updated_at: NULL
1 row in set (0.00 sec)

Thanks!

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack DBaaS (Trove) Edit question
Assignee:
No assignee Edit question
Solved by:
suntr
Solved:
Last query:
Last reply:
Revision history for this message
Ed Cranford (ed--cranford) said :
#1

It could be that your instance can't get a line out and can't finish its apt-get upgrade/install jobs.

try
sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
from the host (not the instance)

Revision history for this message
Michael Basnight (hubcap) said :
#2

U might also try ssh'ing in to the vm (assuming 10.0.0.2 is the vm ip, just ssh 10.0.0.2). Try an apt-get update. Also check /tmp/logfile.txt to see if the guest ever got up and running. Then give us your findings.

Revision history for this message
suntr (suntr) said :
#3

After starting from scratch (you'll see that the users have changed below), I was finally able to SSH into the instance, but the status remains BUILD via rd-client.

From /var/log/upstart/reddwarf-guest.log (on the reddwarf instance):

++++++++++++
stack@ubuntu:/var/log/upstart$ sudo cat reddwarf-guest.log
bash: /home/stack/reddwarf/bin/reddwarf-guestagent: No such file or directory
Traceback (most recent call last):
  File "/home/stack/reddwarf/bin/reddwarf-guestagent", line 40, in <module>
    from reddwarf.common import rpc
  File "/home/stack/reddwarf/reddwarf/common/rpc.py", line 25, in <module>
    from reddwarf.openstack.common.rpc import service as rpc_service
  File "/home/stack/reddwarf/reddwarf/openstack/common/rpc/service.py", line 24, in <module>
    from reddwarf.openstack.common import service
  File "/home/stack/reddwarf/reddwarf/openstack/common/service.py", line 30, in <module>
    import extras
ImportError: No module named extras
++++++++++++

It looks like the guest failed to init properly, causing the message to never be peeled off the queue.

Any ideas on why this is occurring? I'll be investigating in parallel....

Thanks!

Revision history for this message
suntr (suntr) said :
#4

Is this related to https://review.openstack.org/#/c/20145/ ?

/reddwarf/tools/test-requires has "testtools>=0.9.22"
/reddwarf/tools/pip-requires has "extras"

Perhaps changing the testtools value to "testtools>=0.9.22,<=0.9.24" will fix it?

Revision history for this message
suntr (suntr) said :
#5

Disregard, I needed to re-exec "sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE".

Revision history for this message
Michael Basnight (hubcap) said :
#6

awesome. iptables rules generally disappear upon host reboots so you need to make sure you 1) save that rule on the host, or 2) rerun it when u reboot.