Deployment fails for fuel 7.0 on virtualbox

Asked by Makrand on 2016-02-07

Hi,

Setup
Master, 1 controller, 1 compute and 1 cinder. Deployment took long time. I saw READY status on 3 slaves. The progress bar master GUI shows 100%. But it just came with error on very last moment

============================
Deployment Failed
×Error
Deployment has failed. Method granular_deploy. Failed to execute hook 'shell' command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb
Task: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired
Task timeout: 180, Retries: 3

---
priority: 900
fail_on_error: true
type: shell
uids:
- '3'
parameters:
  retries: 3
  cmd: ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb
  timeout: 180
  interval: 20
.
Inspect Astute logs for the details
============================

On node1 compute, when I ran mentioned .rb file, it ran fine

root@node-1:~# ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb
Image 'TestVM' is already present!

Very strange. This has happened twice already. Kind of frustrating.

Any help?

A simple question, if I hit deploy again, will the whole process start from scratch or it will resume where it failed initially.

Question information

Language:
English Edit question
Status:
Answered
For:
Fuel for OpenStack Edit question
Assignee:
No assignee Edit question
Last query:
2016-02-28
Last reply:
2016-03-02
Makrand (makrandsanap) said : #1

Tail of logs for astute

2016-02-07T18:30:09 err: [575] command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb

Task: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired
Task timeout: 180, Retries: 3
2016-02-07T18:30:09 debug: [575] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"3", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"shell", "error_msg"=>"command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb\n\nTask: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired\nTask timeout: 180, Retries: 3"}], "error"=>"Failed to execute hook 'shell' command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb\n\nTask: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired\nTask timeout: 180, Retries: 3"}
2016-02-07T18:30:09 debug: [575] Data send by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"3", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"shell", "error_msg"=>"command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb\n\nTask: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired\nTask timeout: 180, Retries: 3"}], "error"=>"Failed to execute hook 'shell' command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb\n\nTask: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired\nTask timeout: 180, Retries: 3"}
2016-02-07T18:30:09 debug: [575] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}, {"uid"=>"2", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}, {"uid"=>"3", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}]}
2016-02-07T18:30:09 info: [575] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"db3c988b-678d-47f2-9800-a848cb61c729", "nodes"=>[{"uid"=>"3", "status"=>"error", "error_type"=>"deploy", "role"=>"hook", "hook"=>"shell", "error_msg"=>"command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb\n\nTask: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired\nTask timeout: 180, Retries: 3"}], "error"=>"Failed to execute hook 'shell' command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb\n\nTask: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired\nTask timeout: 180, Retries: 3"}}
2016-02-07T18:30:09 debug: [575] Data send by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"1", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}, {"uid"=>"2", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}]}
2016-02-07T18:30:09 err: [575] Error running RPC method granular_deploy: Failed to execute hook 'shell' command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb

Task: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired
Task timeout: 180, Retries: 3

---
priority: 900
fail_on_error: true
type: shell
uids:
- '3'
parameters:
  retries: 3
  cmd: ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb
  timeout: 180
  interval: 20
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/nailgun_hooks.rb:59:in `block in process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/nailgun_hooks.rb:26:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/nailgun_hooks.rb:26:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine/granular_deployment.rb:221:in `post_deployment_actions'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/deployment_engine.rb:75:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/orchestrator.rb:179:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/orchestrator.rb:45:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/dispatcher.rb:91:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:146:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:107:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:89:in `block in perform_main_job'"]

2016-02-07T18:30:09 info: [575] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"db3c988b-678d-47f2-9800-a848cb61c729", "nodes"=>[{"uid"=>"1", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}, {"uid"=>"2", "status"=>"error", "role"=>"hook", "error_type"=>"deploy"}]}}
2016-02-07T18:30:09 info: [575] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"db3c988b-678d-47f2-9800-a848cb61c729", "status"=>"error", "error"=>"Method granular_deploy. Failed to execute hook 'shell' command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb\n\nTask: db3c988b-678d-47f2-9800-a848cb61c729: shell timeout error: execution expired\nTask timeout: 180, Retries: 3\n\n---\npriority: 900\nfail_on_error: true\ntype: shell\nuids:\n- '3'\nparameters:\n retries: 3\n cmd: ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb\n timeout: 180\n interval: 20\n.\nInspect Astute logs for the details"}}

Fabrizio Soppelsa (fsoppelsa) said : #2

Greetings Makrand,
you can find further info in the linked bug. Do you think it describes your problem?

Also, you can't Deploy Changes again, you will need to Reset environment or to Delete it before.
But you can just redeploy the node, using the CLI: look at the online `fuel node --help`:

 Execute deployment tasks
            fuel node --node 2 --tasks hiera netconfig
            fuel node --node 2 --skip hiera netconfig
            fuel node --node 2 --skip rsync --end pre_deployment
            fuel node --node 2 --end netconfig
            fuel node --node 2 --start hiera --end neutron
            fuel node --node 2 --start post_deployment

Makrand (makrandsanap) said : #3

Hi Fabrizio,

Thanks for replying :). I thought no one would answer. I will check bug and
get back to you on this.

Cheers!

--
Best,
Makrand

On Fri, Feb 19, 2016 at 4:11 PM, Fabrizio Soppelsa <
<email address hidden>> wrote:

> Your question #284749 on Fuel for OpenStack changed:
> https://answers.launchpad.net/fuel/+question/284749
>
> Status: Open => Answered
>
> Fabrizio Soppelsa proposed the following answer:
> Greetings Makrand,
> you can find further info in the linked bug. Do you think it describes
> your problem?
>
> Also, you can't Deploy Changes again, you will need to Reset environment
> or to Delete it before.
> But you can just redeploy the node, using the CLI: look at the online
> `fuel node --help`:
>
> Execute deployment tasks
> fuel node --node 2 --tasks hiera netconfig
> fuel node --node 2 --skip hiera netconfig
> fuel node --node 2 --skip rsync --end pre_deployment
> fuel node --node 2 --end netconfig
> fuel node --node 2 --start hiera --end neutron
> fuel node --node 2 --start post_deployment
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https://answers.launchpad.net/fuel/+question/284749/+confirm?answer_id=1
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/fuel/+question/284749
>
> You received this question notification because you asked the question.
>

Makrand (makrandsanap) said : #4

Hi Fabrizio,

Just simple question. That is almost year old bug with fuel 6.1. I am
assuming it has been fixed already (Fix Committed) and should not be the
case with 7.0. Sorry if this sounds bit noobish, but I am new to Openstack
and the launchpad etc.

I will try with commands you suggested.

--
Best,
Makrand

On Sun, Feb 28, 2016 at 2:23 PM, Makrand <
<email address hidden>> wrote:

> Your question #284749 on Fuel for OpenStack changed:
> https://answers.launchpad.net/fuel/+question/284749
>
> Status: Answered => Open
>
> You are still having a problem:
> Hi Fabrizio,
>
> Thanks for replying :). I thought no one would answer. I will check bug and
> get back to you on this.
>
> Cheers!
>
> --
> Best,
> Makrand
>
>
> On Fri, Feb 19, 2016 at 4:11 PM, Fabrizio Soppelsa <
> <email address hidden>> wrote:
>
> > Your question #284749 on Fuel for OpenStack changed:
> > https://answers.launchpad.net/fuel/+question/284749
> >
> > Status: Open => Answered
> >
> > Fabrizio Soppelsa proposed the following answer:
> > Greetings Makrand,
> > you can find further info in the linked bug. Do you think it describes
> > your problem?
> >
> > Also, you can't Deploy Changes again, you will need to Reset environment
> > or to Delete it before.
> > But you can just redeploy the node, using the CLI: look at the online
> > `fuel node --help`:
> >
> > Execute deployment tasks
> > fuel node --node 2 --tasks hiera netconfig
> > fuel node --node 2 --skip hiera netconfig
> > fuel node --node 2 --skip rsync --end pre_deployment
> > fuel node --node 2 --end netconfig
> > fuel node --node 2 --start hiera --end neutron
> > fuel node --node 2 --start post_deployment
> >
> > --
> > If this answers your question, please go to the following page to let us
> > know that it is solved:
> > https://answers.launchpad.net/fuel/+question/284749/+confirm?answer_id=1
> >
> > If you still need help, you can reply to this email or go to the
> > following page to enter your feedback:
> > https://answers.launchpad.net/fuel/+question/284749
> >
> > You received this question notification because you asked the question.
> >
>
> --
> You received this question notification because you asked the question.
>

Fabrizio Soppelsa (fsoppelsa) said : #5

Hi Makrand,
you can get a try with 8.0 that was released yesterday.

Best regards,
Fabrizio

Can you help with this problem?

Provide an answer of your own, or ask Makrand for more information if necessary.

To post a message you must log in.