Interfaces are locked for update and Volume information is locked for update

Asked by Hugh Ma on 2015-10-22

I am constantly getting these log messages on fuel master logs. It starts to fill up the log space pretty quickly.
Any idea what is causing this?

This is Fuel 7.0

2015-10-22 15:49:31 INFO [7f91553a2740] (node) Interfaces are locked for update on node R1N29
[pid: 853|app: 0|req: 5867/42914] 172.18.0.22 () {36 vars in 527 bytes} [Thu Oct 22 15:49:31 2015] PUT /api/nodes/agent/ => generated 10 bytes in 131 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
2015-10-22 15:49:31 INFO [7f91553a2740] (node) Volume information is locked for update on node R1N29
2015-10-22 15:49:30 INFO [7f91553a2740] (node) Interfaces are locked for update on node R1N03
[pid: 853|app: 0|req: 5866/42913] 172.18.0.116 () {36 vars in 528 bytes} [Thu Oct 22 15:49:30 2015] PUT /api/nodes/agent/ => generated 10 bytes in 120 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)
2015-10-22 15:49:30 INFO [7f91553a2740] (node) Volume information is locked for update on node R1N03
2015-10-22 15:49:29 INFO [7f91553a2740] (node) Interfaces are locked for update on node R1N02
[pid: 853|app: 0|req: 5865/42912] 172.18.0.106 () {36 vars in 528 bytes} [Thu Oct 22 15:49:29 2015] PUT /api/nodes/agent/ => generated 11 bytes in 116 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)

Question information

Language:
English Edit question
Status:
Answered
For:
Fuel for OpenStack Edit question
Assignee:
No assignee Edit question
Last query:
2015-10-26
Last reply:
2015-10-27

This question was reopened

Fabrizio Soppelsa (fsoppelsa) said : #1

Greetings Hugh,
what is the log file including these entries? Nailgun.api? Astute.api?

Also, can you show us these outputs?
du -sh /var/log/docker-logs/remote/
du -sh /var/log
df -h /var/log

Fabrizio
Mirantis team

Hugh Ma (hughkma) said : #2

They are showing up under FuelMaster -> Web Backend

[root@fuel ~]# du -sh /var/log/docker-logs/remote/
12G /var/log/docker-logs/remote/
[root@fuel ~]# du -sh /var/log
14G /var/log
[root@fuel ~]# df -h /var/log
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-varlog
                       40G 14G 24G 38% /var/log

Fabrizio Soppelsa (fsoppelsa) said : #3

This is Nailgun https://github.com/openstack/fuel-web/blob/stable/7.0/nailgun/nailgun/objects/node.py#L466
Do you have a deployed and provisioned environment?

Seems that a node is flapping or has some issue: R1N03.

Hugh Ma (hughkma) said : #4

Yea it is deployed and provisioned. It throws up those messages for a lot of different nodes.

Fabrizio Soppelsa (fsoppelsa) said : #5

Anything on the nodes in /var/log/messages ?

Hugh Ma (hughkma) said : #6

a bunch of this:

<4>Oct 22 02:50:45 node-70 kernel: [1744861.000268] task: ffff8817fd376000 ti: ffff880bfd12e000 task.ti: ffff880bfd12e000
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000273] RIP: 0010:[<ffffffff813e98f8>] [<ffffffff813e98f8>] intel_idle+0xd8/0x140
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000274] RSP: 0018:ffff880bfd12fe28 EFLAGS: 00000046
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000275] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000276] RDX: 0000000000000000 RSI: ffffffff81c93e80 RDI: 0000000000000017
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000278] RBP: ffff880bfd12fe50 R08: ffff88183fd70310 R09: 0000000000000018
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000279] R10: 000000000000e02c R11: 000000000001902a R12: 0000000000000004
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000280] R13: 0000000000000020 R14: 0000000000000003 R15: ffffffff81c93ff8
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000282] FS: 0000000000000000(0000) GS:ffff88183fd60000(0000) knlGS:0000000000000000
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000283] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000284] CR2: 00000000008bc088 CR3: 0000000001c0e000 CR4: 00000000000007e0
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000285] Stack:
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000289] 00000017fd12fe50 ffff88183fd7a200 ffffffff81c93e80 000631f43dddad79
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000292] 0000000000000004 ffff880bfd12fe88 ffffffff815d5720 ffff880bfd12ff38
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000295] ffff88183fd7a200 0000000000000004 0000000000000017 ffffffff81c93e80
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000296] Call Trace:
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000300] [<ffffffff815d5720>] cpuidle_enter_state+0x40/0xc0
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000303] [<ffffffff815d5859>] cpuidle_idle_call+0xb9/0x1f0
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000307] [<ffffffff8101d3ee>] arch_cpu_idle+0xe/0x30
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000309] [<ffffffff810bf215>] cpu_startup_entry+0xc5/0x290
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000313] [<ffffffff810415cd>] start_secondary+0x21d/0x2d0
<4>Oct 22 02:50:45 node-70 kernel: [1744861.000332] Code: 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 0f ae f0 65 48 8b 04 25 30 b8 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 7a a5 8a 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 b2 a6 ce

Fabrizio Soppelsa (fsoppelsa) said : #7

Mh, can you attach a full dmesg and a full messages?

Hello,

These logs from the topic are just of "INFO" priority. It's Ok.
It means that nailgun gets new portions of information from nailgun-agent but is forbidden to make any changes in DB
as your environment has already been deployed.

Yes, may be it worth to change severity level for such sort of messages.

BTW, these _call_traces_ in your dmesq is a really bad sign.

Thanks,

PS: You can change a priority level for logs to be saved in rsyslog's configuration on your master node.

The bug was launched https://bugs.launchpad.net/fuel/+bug/1510151

Thanks,

Hugh Ma (hughkma) said : #11

 cool thanks.

Hugh Ma (hughkma) said : #12

btw what do you mean by

"BTW, these _call_traces_ in your dmesq is a really bad sign."

I mean a chunk of /var/log/messages you posted above.
Such "Call Trace:" records in logs may say about some HW/driver's problems.

But it's not related to the subject.

Regards,

Can you help with this problem?

Provide an answer of your own, or ask Hugh Ma for more information if necessary.

To post a message you must log in.