Problem with space on my Fuel server

Asked by mario on 2015-08-24

I have a problem with Fuel 6.0 because i have /var of 100%.

How to resolve the problem ?

Thanks

Question information

Language:
English Edit question
Status:
Answered
For:
Fuel for OpenStack Edit question
Assignee:
No assignee Edit question
Last query:
2015-08-24
Last reply:
2015-09-14

Hi,

Most probably you have to get rid of old log files. It's save to remove them.
Please run "df -sh /var/*" to get sure.

Please note (!) you should clear the space ASAP as in such situation there is a risk of corruption of FSs in your containers.

For the future you can
1. provide own partition for logging (if possible)
2. set up more aggressive logrotate policies (see "weekly/daily", "rotate", "compress" logrotate's options)

Regards,

mario (mariocastronovo) said : #2

Thanks.

How to clean space?

Mario

mario (mariocastronovo) said : #3

In this moment i have 16 Gb free. When i launch the command dockerctl list -l i receive the keystone: Stopped and the Postgres and Nailgun not created.

Michal Rostecki (mrostecki) said : #4

Hi,

Could you please check the output of "du -sh /var/*" and "du -sh /var/lib/*" commands? The significant amount of space should be mostly taken by
/var/lib/docker - which is normal and should take around 16 GB
/var/log - which are the logfiles mentioned by Evgeny, and you can clean it in the way that Evgeny suggested

If some other directories take more than ~50 MB, please inform us here about this.

After cleaning some space, please try to do:

dockerctl buld
dockerctl start

Regards,
Michal

PS. @ekozhemyakin: you probably meant du instead of df, right?

Hi,

Please see here is described the recovery proccess for your case:
https://docs.mirantis.com/openstack/fuel/fuel-6.1/operations.html#fuel-master-and-docker-disk-space-troubleshooting

@Michal: "du" surely. Thanks for the correction.

mario (mariocastronovo) said : #6

Hi, when i execute the dockerctl build i receive this errors:

Error: No such image or container: fuel-core-6.0-nailgun
Error: No such image or container: fuel-core-6.0-postgresql

Thanks

Michal Rostecki (mrostecki) said : #7

Hi,

That means that base images are missing. Try to execute the following command:

bootstrap_admin_node.sh

If it will fail, let me know, then I will prepare other solution.

Cheers,
Michal

mario (mariocastronovo) said : #8

Hi Michal, after i execute the command "bootstrap_admin_node.sh" i receive the same errors.

If i execute dockerctl check i receive this messages:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
410a1e7daaf1 fuel/mcollective_6.0:latest /usr/local/bin/start 6 months ago Up 17 minutes fuel-core-6.0-mcollective
0a830c919b7e fuel/cobbler_6.0:latest /bin/sh -c /usr/loca 6 months ago Up 18 minutes 53/tcp, 192.168.245.11:53->53/udp, 127.0.0.1:53->53/udp, 67/tcp, 67/udp, 127.0.0.1:69->69/udp, 192.168.245.11:69->69/udp, 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp fuel-core-6.0-cobbler
ed1f069f7d0d fuel/nginx_6.0:latest /usr/local/bin/start 6 months ago Up 18 minutes 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp fuel-core-6.0-nginx
460540add433 fuel/ostf_6.0:latest /bin/sh -c /usr/loca 6 months ago Up 18 minutes 192.168.245.11:8777->8777/tcp, 127.0.0.1:8777->8777/tcp fuel-core-6.0-ostf
c75e805f391b fuel/rsyslog_6.0:latest /usr/local/bin/start 6 months ago Up 17 minutes 192.168.245.11:514->514/tcp, 127.0.0.1:514->514/tcp, 192.168.245.11:514->514/udp, 127.0.0.1:514->514/udp, 192.168.245.11:25150->25150/tcp, 127.0.0.1:25150->25150/tcp fuel-core-6.0-rsyslog
28e62586eb7a fuel/astute_6.0:latest /bin/sh -c /usr/loca 6 months ago Up 18 minutes fuel-core-6.0-astute
f088e37b78d9 fuel/rsync_6.0:latest /bin/sh -c /usr/loca 6 months ago Up 17 minutes 192.168.245.11:873->873/tcp, 127.0.0.1:873->873/tcp fuel-core-6.0-rsync
5be21fe1b9b3 fuel/rabbitmq_6.0:latest /usr/local/bin/start 6 months ago Up 18 minutes 192.168.245.11:4369->4369/tcp, 127.0.0.1:4369->4369/tcp, 192.168.245.11:5672->5672/tcp, 127.0.0.1:5672->5672/tcp, 192.168.245.11:15672->15672/tcp, 127.0.0.1:15672->15672/tcp, 192.168.245.11:61613->61613/tcp, 127.0.0.1:61613->61613/tcp fuel-core-6.0-rabbitmq

Thanks

Mario

mario (mariocastronovo) said : #9

Hi Michal, are there news about this problem?

Thanks

Michal Rostecki (mrostecki) said : #10

Hi,

Maybe try to do:

docker ps -aq | xargs -i docker rm -f {}
docker images -aq | xargs -i docker rmi -f {}

And then:

bootstrap_admin_node.sh

Cheers,
Michal

mario (mariocastronovo) said : #11

Sorry but i receive the same errors:

Error: No such image or container: postgres
Could not get docker ID for container postgres. Is it running?
Could not get docker ID for all. Is it running?
try number 83
return code is 1

And before i reive this errors:

Error: Conflict, 60231f81bfe1 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 1013fa291ef2 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Untagged: fuel/rsync_6.0:latest
Error: Conflict, a4810ad8da05 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 6bf667202a2d wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 362f11c4e6f1 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 82d8bf5b7319 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 80313f95bc28 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, c6670015d36a wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, cannot delete 96098ec431cb because the container 965adbc66925 is using it
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 47992f26e8f0 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 252f6f9d6816 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, ee38b4d1bb40 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, fabf031f337c wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 3c41fd48ffa5 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 952d0365303d wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, cannot delete a4b579fd3e3b because the container fa3b7d73ec90 is using it
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, bb488ff8c87d wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 1073c8944675 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, d5f4b2da1855 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, d7399d704461 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 26057cf625e7 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 6709af80a9a2 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, e91a67cf4784 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, cannot delete 3d07100014c1 because the container 65852e740ed1 is using it
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 35237eeaaa3f wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 40aae4d25eda wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, e7ed31f3e08a wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 3ecee02c8543 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 4e2d28860d97 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, e117ebcdf8c8 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 547c8c68a76e wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, ade41f2aa2a2 wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, cc1a22b100e8 wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, 79ca48f7cce3 wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, 2d767baf3da6 wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, 7ebe67bcf12b wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, c0a05a6ab02f wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Untagged: fuel/nailgun_6.0:latest
Deleted: 2dfbdc8c01526ab2e659f6fa18ec407eaabc36e29aacfc84a668cd3f27053b70
Deleted: 95c45df1e4bb79f4f3d12695bac0f97087d2bb2384e082ba6c0c6e55d12c60e8
Deleted: ade53f9727e772dae83806b343f2f8d54e844054b2e72fabf1b9ba197114ac7c
Deleted: 718546056f5e1b40ae46a3cb6fb69fe197fd60fe5d1cc66d2e5f5fcaa7d5f8d5
Deleted: 694ea6c8964f4d01e64f60d36c06d7b3c2a1c9a70d9be6aa9124f8088784ba30
Deleted: baf4543ca7d1532cefc9c3b4d20e6fd97cbd67f2875f34159b0413236717465c
Deleted: ed473ae9a77c074ba2c6276a7080aa6531fec40abdb32cc7979bc82d90e993ef
Deleted: 2df3374482ddaf57b1cb98c93040002f26f1000b10004c2a39d4e97ad22b011c
Deleted: 44d577c28426154d3f159cea9a38d5d9af9c35dfa69450fa7193beec8795d89f
Deleted: d7b94927fbad4bf0951aaa9285bdc37b890eea571e02d3ee9c1eb8123f5deccf
Deleted: 60d043dafa118397a11e4e8b99f3eed94e97f84b2e2cb5fdf51324576eb24a8e
Deleted: a8eae6bae18ac06d4777dfb9ec9c09bc754b96de6a003b4a8ff08e9d4925cda6
Error: No such image: 718546056f5e
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 95c45df1e4bb
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: ade53f9727e7
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: baf4543ca7d1
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 694ea6c8964f
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: ed473ae9a77c
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 44d577c28426
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 2df3374482dd
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: d7b94927fbad
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 60d043dafa11
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: a8eae6bae18a
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, 03d2f86d1a14 wasn't deleted
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, 73287e15959a wasn't deleted
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, cannot delete 7fab7bd8a297 because the container 410a1e7daaf1 is using it
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, bcf870ec7125 wasn't deleted
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, 2dbebf6d4d75 wasn't deleted
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, 2c3dfb388d6f wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 3995bec58d7e wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, f4f8ca96169a wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 4d2e883408fc wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 9ac0428c11a5 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, cannot delete 915002d5049f because the container d60ce7e7c90b is using it
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, cd586d568f79 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, a4fed81b1473 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, bdca5cbdf09f wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 35b6c9262a1f wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, c818b2a6370e wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, b8f7b8b05a2e wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, cc5e209fc876 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, cannot delete abb3db6fb60c because the container 5730cf0fd858 is using it
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 10b9ea1e41f0 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 700a10b7b650 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 30d9ef2dff6e wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Error: Conflict, 89cda0b54389 wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Error: Conflict, 4e5299a2abcb wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Error: Conflict, 56fbae8a3a14 wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Error: Conflict, 257f990baceb wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Untagged: jpetazzo/nsenter:latest
Deleted: f495905a267097922d909bf498e6af89cfa13af16bbbba8a732c8888c9014541
Deleted: 52ee3273029ce1f373a3e1f1d77995016aa344d784817795be50c55a3cd73f55
Deleted: c1bd260973faa67aa1ba7682aea81a3238bb026202364e603ef5a78e4061ffc1
Deleted: 851b05e19fc2f228f614110fa44b9664fef334dd584f921ce8ddc702850fbfe5
Deleted: 731d521b86c71ed616df1046455976b6df9800c1c1efea48673ae0701d132c11
Deleted: b3073fbfdb5a39028f29eb846c368cc2a62f432804a0e8da356165d2cbd702d8
Deleted: d577e4cdab15cbdba7bf67e8443392788ba70f79ee943d2ed70e7f880d8adfb2
Deleted: 7199b299c30f513eba072ff26043c07014ebb7d680fcfb9e1e89804250c79255
Deleted: 1ab95ee9407f64147046a806a424fcc9dbfe7e38e3e41c44ac748e9c66be5504
Deleted: 66e54c5b5d94369d11861a6cc4d0b2ac75dd7e0cbbfdaa6734a0692e1242d4bf
Deleted: 5845be93387406193033bd0489ae2e9a9ef774f9f169dc884de84084865eb6a2
Deleted: b0b29173013c55c6009a38f843798d12cb133148b777fdd372db595ea09460f9
Deleted: f485dd2ac292809b7cb90ade149dfdc78cbed1fa9960529585014e521e4c2d96
Deleted: 7a37928bf6a187403404df5214322a5ff73ba09151324fadbef6671f610943b0
Deleted: badcb042f7c92d0847f4a8372cf11fc1af3ab4b3d0f7da4013d3605f83905101
Deleted: a841579d7409739248acad87cfe14f80692287d904b35a15211e02ac0327c7c4
Deleted: 743c1e45e688224fa627bf95950552f05a6443eaf2985d7cbc70d29ac64960be
Error: No such image: 52ee3273029c
2015/09/14 13:30:36 Error: failed to remove one or more images
Error: No such image: c1bd260973fa
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 851b05e19fc2
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 731d521b86c7
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: b3073fbfdb5a
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: d577e4cdab15
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 7199b299c30f
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 1ab95ee9407f
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 66e54c5b5d94
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 5845be933874
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: b0b29173013c
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: f485dd2ac292
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 7a37928bf6a1
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: badcb042f7c9
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: a841579d7409
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 743c1e45e688
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: Conflict, 6cfbb482a117 wasn't deleted
2015/09/14 13:30:37 Error: failed to remove one or more images
Untagged: busybox:latest
Deleted: 2d8e5b282c81244037eb15b2068e1c46319c1a42b80493acb128da24b2090739
Deleted: 9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a
Deleted: 6c991eb934609424f761d3d0a7c79f4f72b76db286aa02e617659ac116aa7758
Deleted: 511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
Error: No such image: 9f4e93171ec5
2015/09/14 13:30:41 Error: failed to remove one or more images
Error: No such image: 6c991eb93460
2015/09/14 13:30:41 Error: failed to remove one or more images
Error: No such image: 511136ea3c5a
2015/09/14 13:30:41 Error: failed to remove one or more images

Thanks

Mario

Can you help with this problem?

Provide an answer of your own, or ask mario for more information if necessary.

To post a message you must log in.