fuel hanged on provisioning phase with fuel plugins

Asked by Maple Wang

Hi,

recently I just want to deploy mirantis openstack with some fuel plugins in fuel 7.0 as I did in fuel 6.1. But surprisingly, fuel even can't finish the provisioning, all nodes were with status like "INSTALLING UBUNTU: 100%", but just hanged there without proceeding to next deploying phase.

I checked the log in fuel master from web backend like below:
--------------------------------------------------------------------------------------------------------------------------------------------------------------
2015-11-27 12:11:15 INFO [7fc13ae5f740] (node) Interfaces are locked for update on node CCCM-1
[pid: 708|app: 0|req: 924/5011] 10.20.0.4 () {36 vars in 522 bytes} [Fri Nov 27 12:11:15 2015] PUT /api/nodes/agent/ => generated 9 bytes in 133 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 710|app: 0|req: 482/5012] 150.236.223.74 () {44 vars in 854 bytes} [Fri Nov 27 12:11:19 2015] GET /api/tasks/15?_=1448620462295 => generated 158 bytes in 11 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 703|app: 0|req: 383/5013] 150.236.223.74 () {44 vars in 874 bytes} [Fri Nov 27 12:11:19 2015] GET /api/nodes?cluster_id=1&_=1448620462296 => generated 23544 bytes in 337 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 704|app: 0|req: 1621/5014] 150.236.223.74 () {44 vars in 854 bytes} [Fri Nov 27 12:11:25 2015] GET /api/tasks/15?_=1448620462297 => generated 158 bytes in 11 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
[pid: 704|app: 0|req: 1622/5015] 150.236.223.74 () {44 vars in 874 bytes} [Fri Nov 27 12:11:25 2015] GET /api/nodes?cluster_id=1&_=1448620462298 => generated 23544 bytes in 336 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
2015-11-27 12:11:15 INFO [7fc13ae5f740] (node) Volume information is locked for update on node CCCM-1
2015-11-27 12:11:15 DEBUG [7fc13ae5f740] (node) Node CCCM-1 has provisioning or error status - status not updated by agent
--------------------------------------------------------------------------------------------------------------------------------------------------------------

it seems that agent is not working, but this is log from fuel-agent in CCCM-1 node:
--------------------------------------------------------------------------------------------------------------------------------------------------------------
2015-11-27 11:25:16 INFO fuel_agent.manager [-] --- Provisioning END (do_provisioning) ---
2015-11-27 11:25:16 INFO fuel_agent.utils.utils [-] Trying to execute command: umount /tmp/target/
2015-11-27 11:25:16 INFO fuel_agent.utils.fs [-] Trying to umount /tmp/target/
2015-11-27 11:25:16 INFO fuel_agent.utils.utils [-] Trying to execute command: mountpoint -q /tmp/target/
2015-11-27 11:25:16 INFO fuel_agent.utils.utils [-] Trying to execute command: umount /tmp/target/boot
2015-11-27 11:25:16 INFO fuel_agent.utils.fs [-] Trying to umount /tmp/target/boot
2015-11-27 11:25:16 INFO fuel_agent.utils.utils [-] Trying to execute command: mountpoint -q /tmp/target/boot
2015-11-27 11:25:16 INFO fuel_agent.utils.utils [-] Trying to execute command: umount /tmp/target/var/log
2015-11-27 11:25:16 INFO fuel_agent.utils.fs [-] Trying to umount /tmp/target/var/log
--------------------------------------------------------------------------------------------------------------------------------------------------------------
it looks like provisioning has already ended, but why fuel master can't received the status? but once I removed the plugins, the provisioning just work properly, does plugins cause this problem? anyone met the same problem as me?

by the way, i have not found the error in the execution, who can tell me what happened?

I almost installed all OPERATIONS MANAGEMENT PLUGINS i the link below:
https://www.mirantis.com/products/openstack-drivers-and-plugins/fuel-plugins/

I suspected maybe just someone of them introduced this problem, but even I tried to install one plugin in one time, this problem always happened as long as I installed it.

if you want to have any info or logs to trace, feel free to ask.

thanks in advance.

Question information

Language:
English Edit question
Status:
Expired
For:
Fuel for OpenStack Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Maple Wang (maple-feng-wang) said :
#1

here is tcpdump log from fuel master:(10.10.10.2 is external IP on fuel master)

20:41:31.200664 ip: 10.10.10.2.33961 > 150.236.34.180.53: 54364+ AAAA? CCCM-1.domain.tld. (35)
20:41:31.261505 ip: 150.236.34.180.53 > 10.10.10.2.33961: 54364 NXDomain 0/1/0 (110)
20:41:31.264227 ip: 10.10.10.2.33961 > 150.236.34.181.53: 54364+ AAAA? CCCM-1.domain.tld. (35)
20:41:31.264352 ip: 10.10.10.2.33961 > 146.11.115.200.53: 54364+ AAAA? CCCM-1.domain.tld. (35)
20:41:31.265784 ip: 150.236.34.181.53 > 10.10.10.2.33961: 54364 NXDomain 0/1/0 (110)
20:41:31.268487 ip: 10.10.10.2 > 150.236.34.181: ICMP 10.10.10.2 udp port 33961 unreachable, length 146
20:41:31.340261 ip: 146.11.115.200.53 > 10.10.10.2.33961: 54364 NXDomain 0/1/0 (110)
20:41:31.343223 ip: 10.10.10.2 > 146.11.115.200: ICMP 10.10.10.2 udp port 33961 unreachable, length 146
20:43:54.305806 ip: 10.10.10.2.27376 > 150.236.34.180.53: 57017+ AAAA? CCCM--2.domain.tld. (36)
20:43:54.306021 ip: 10.10.10.2.27376 > 150.236.34.181.53: 57017+ AAAA? CCCM--2.domain.tld. (36)
20:43:54.307034 ip: 150.236.34.181.53 > 10.10.10.2.27376: 57017 NXDomain 0/1/0 (111)
20:43:54.307844 ip: 150.236.34.180.53 > 10.10.10.2.27376: 57017 NXDomain 0/1/0 (111)
20:43:54.310429 ip: 10.10.10.2.27376 > 146.11.115.200.53: 57017+ AAAA? CCCM--2.domain.tld. (36)
20:43:54.313339 ip: 10.10.10.2 > 150.236.34.180: ICMP 10.10.10.2 udp port 27376 unreachable, length 147
20:43:54.418131 ip: 146.11.115.200.53 > 10.10.10.2.27376: 57017 NXDomain 0/1/0 (111)
20:43:54.422098 ip: 10.10.10.2 > 146.11.115.200: ICMP 10.10.10.2 udp port 27376 unreachable, length 147
20:43:57.940486 ip: 10.10.10.2.2049 > 150.236.34.180.53: 46173+ AAAA? LMA.domain.tld. (32)
20:43:57.942228 ip: 150.236.34.180.53 > 10.10.10.2.2049: 46173 NXDomain 0/1/0 (107)
20:43:57.944535 ip: 10.10.10.2.2049 > 150.236.34.181.53: 46173+ AAAA? LMA.domain.tld. (32)
20:43:57.944725 ip: 10.10.10.2.2049 > 146.11.115.200.53: 46173+ AAAA? LMA.domain.tld. (32)
20:43:57.945341 ip: 150.236.34.181.53 > 10.10.10.2.2049: 46173 NXDomain 0/1/0 (107)
20:43:57.948525 ip: 10.10.10.2 > 150.236.34.181: ICMP 10.10.10.2 udp port 2049 unreachable, length 143
20:43:58.066072 ip: 146.11.115.200.53 > 10.10.10.2.2049: 46173 NXDomain 0/1/0 (107)
20:43:58.069806 ip: 10.10.10.2 > 146.11.115.200: ICMP 10.10.10.2 udp port 2049 unreachable, length 143
20:45:51.459151 ip: 10.10.10.2.60790 > 150.236.34.180.53: 42208+ AAAA? OPENBOOK.domain.tld. (37)
20:45:51.460879 ip: 150.236.34.180.53 > 10.10.10.2.60790: 42208 NXDomain 0/1/0 (112)
20:45:51.463588 ip: 10.10.10.2.60790 > 150.236.34.181.53: 42208+ AAAA? OPENBOOK.domain.tld. (37)
20:45:51.463734 ip: 10.10.10.2.60790 > 146.11.115.200.53: 42208+ AAAA? OPENBOOK.domain.tld. (37)
20:45:51.464616 ip: 150.236.34.181.53 > 10.10.10.2.60790: 42208 NXDomain 0/1/0 (112)
20:45:51.507155 ip: 10.10.10.2 > 150.236.34.181: ICMP 10.10.10.2 udp port 60790 unreachable, length 148
20:45:51.568014 ip: 146.11.115.200.53 > 10.10.10.2.60790: 42208 NXDomain 0/1/0 (112)
20:45:51.571974 ip: 10.10.10.2 > 146.11.115.200: ICMP 10.10.10.2 udp port 60790 unreachable, length 148

I don't know if I understand it right? it looks like 150.236.34.181 and 146.11.115.200 want to access 10.10.10.2's udp port initially? for some reasons, fuel master is behind the NAT device, so it's impossible for outside to visit the service of fuel master initially in my environment. please correct me if I'm wrong.

Revision history for this message
Maple Wang (maple-feng-wang) said :
#2

Hi,

is there any support got here? or I just post on wrong place?

best regards.

Revision history for this message
Launchpad Janitor (janitor) said :
#3

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Maple Wang (maple-feng-wang) said :
#4
Revision history for this message
Launchpad Janitor (janitor) said :
#5

This question was expired because it remained in the 'Open' state without activity for the last 15 days.