Problem with space on my Fuel server

Hi,

Most probably you have to get rid of old log files. It's save to remove them.
Please run "df -sh /var/*" to get sure.

Please note (!) you should clear the space ASAP as in such situation there is a risk of corruption of FSs in your containers.

For the future you can
1. provide own partition for logging (if possible)
2. set up more aggressive logrotate policies (see "weekly/daily", "rotate", "compress" logrotate's options)

Regards,

Thanks.

How to clean space?

Mario

In this moment i have 16 Gb free. When i launch the command dockerctl list -l i receive the keystone: Stopped and the Postgres and Nailgun not created.

Hi,

Could you please check the output of "du -sh /var/*" and "du -sh /var/lib/*" commands? The significant amount of space should be mostly taken by
/var/lib/docker - which is normal and should take around 16 GB
/var/log - which are the logfiles mentioned by Evgeny, and you can clean it in the way that Evgeny suggested

If some other directories take more than ~50 MB, please inform us here about this.

After cleaning some space, please try to do:

dockerctl buld
dockerctl start

Regards,
Michal

PS. @ekozhemyakin: you probably meant du instead of df, right?

Hi,

Please see here is described the recovery proccess for your case:
https://docs.mirantis.com/openstack/fuel/fuel-6.1/operations.html#fuel-master-and-docker-disk-space-troubleshooting

@Michal: "du" surely. Thanks for the correction.

Hi, when i execute the dockerctl build i receive this errors:

Error: No such image or container: fuel-core-6.0-nailgun
Error: No such image or container: fuel-core-6.0-postgresql

Thanks

Hi,

That means that base images are missing. Try to execute the following command:

bootstrap_admin_node.sh

If it will fail, let me know, then I will prepare other solution.

Cheers,
Michal

Hi Michal, after i execute the command "bootstrap_admin_node.sh" i receive the same errors.

If i execute dockerctl check i receive this messages:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
410a1e7daaf1 fuel/mcollective_6.0:latest /usr/local/bin/start 6 months ago Up 17 minutes fuel-core-6.0-mcollective
0a830c919b7e fuel/cobbler_6.0:latest /bin/sh -c /usr/loca 6 months ago Up 18 minutes 53/tcp, 192.168.245.11:53->53/udp, 127.0.0.1:53->53/udp, 67/tcp, 67/udp, 127.0.0.1:69->69/udp, 192.168.245.11:69->69/udp, 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp fuel-core-6.0-cobbler
ed1f069f7d0d fuel/nginx_6.0:latest /usr/local/bin/start 6 months ago Up 18 minutes 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp fuel-core-6.0-nginx
460540add433 fuel/ostf_6.0:latest /bin/sh -c /usr/loca 6 months ago Up 18 minutes 192.168.245.11:8777->8777/tcp, 127.0.0.1:8777->8777/tcp fuel-core-6.0-ostf
c75e805f391b fuel/rsyslog_6.0:latest /usr/local/bin/start 6 months ago Up 17 minutes 192.168.245.11:514->514/tcp, 127.0.0.1:514->514/tcp, 192.168.245.11:514->514/udp, 127.0.0.1:514->514/udp, 192.168.245.11:25150->25150/tcp, 127.0.0.1:25150->25150/tcp fuel-core-6.0-rsyslog
28e62586eb7a fuel/astute_6.0:latest /bin/sh -c /usr/loca 6 months ago Up 18 minutes fuel-core-6.0-astute
f088e37b78d9 fuel/rsync_6.0:latest /bin/sh -c /usr/loca 6 months ago Up 17 minutes 192.168.245.11:873->873/tcp, 127.0.0.1:873->873/tcp fuel-core-6.0-rsync
5be21fe1b9b3 fuel/rabbitmq_6.0:latest /usr/local/bin/start 6 months ago Up 18 minutes 192.168.245.11:4369->4369/tcp, 127.0.0.1:4369->4369/tcp, 192.168.245.11:5672->5672/tcp, 127.0.0.1:5672->5672/tcp, 192.168.245.11:15672->15672/tcp, 127.0.0.1:15672->15672/tcp, 192.168.245.11:61613->61613/tcp, 127.0.0.1:61613->61613/tcp fuel-core-6.0-rabbitmq

Thanks

Mario

Hi Michal, are there news about this problem?

Thanks

Hi,

Maybe try to do:

docker ps -aq | xargs -i docker rm -f {}
docker images -aq | xargs -i docker rmi -f {}

And then:

bootstrap_admin_node.sh

Cheers,
Michal

Sorry but i receive the same errors:

Error: No such image or container: postgres
Could not get docker ID for container postgres. Is it running?
Could not get docker ID for all. Is it running?
try number 83
return code is 1

And before i reive this errors:

Error: Conflict, 60231f81bfe1 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 1013fa291ef2 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Untagged: fuel/rsync_6.0:latest
Error: Conflict, a4810ad8da05 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 6bf667202a2d wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 362f11c4e6f1 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 82d8bf5b7319 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 80313f95bc28 wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, c6670015d36a wasn't deleted
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, cannot delete 96098ec431cb because the container 965adbc66925 is using it
2015/09/14 13:29:02 Error: failed to remove one or more images
Error: Conflict, 47992f26e8f0 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 252f6f9d6816 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, ee38b4d1bb40 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, fabf031f337c wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 3c41fd48ffa5 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 952d0365303d wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, cannot delete a4b579fd3e3b because the container fa3b7d73ec90 is using it
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, bb488ff8c87d wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 1073c8944675 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, d5f4b2da1855 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, d7399d704461 wasn't deleted
2015/09/14 13:29:03 Error: failed to remove one or more images
Error: Conflict, 26057cf625e7 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 6709af80a9a2 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, e91a67cf4784 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, cannot delete 3d07100014c1 because the container 65852e740ed1 is using it
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 35237eeaaa3f wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 40aae4d25eda wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, e7ed31f3e08a wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 3ecee02c8543 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 4e2d28860d97 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, e117ebcdf8c8 wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, 547c8c68a76e wasn't deleted
2015/09/14 13:29:04 Error: failed to remove one or more images
Error: Conflict, ade41f2aa2a2 wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, cc1a22b100e8 wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, 79ca48f7cce3 wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, 2d767baf3da6 wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, 7ebe67bcf12b wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Error: Conflict, c0a05a6ab02f wasn't deleted
2015/09/14 13:29:05 Error: failed to remove one or more images
Untagged: fuel/nailgun_6.0:latest
Deleted: 2dfbdc8c01526ab2e659f6fa18ec407eaabc36e29aacfc84a668cd3f27053b70
Deleted: 95c45df1e4bb79f4f3d12695bac0f97087d2bb2384e082ba6c0c6e55d12c60e8
Deleted: ade53f9727e772dae83806b343f2f8d54e844054b2e72fabf1b9ba197114ac7c
Deleted: 718546056f5e1b40ae46a3cb6fb69fe197fd60fe5d1cc66d2e5f5fcaa7d5f8d5
Deleted: 694ea6c8964f4d01e64f60d36c06d7b3c2a1c9a70d9be6aa9124f8088784ba30
Deleted: baf4543ca7d1532cefc9c3b4d20e6fd97cbd67f2875f34159b0413236717465c
Deleted: ed473ae9a77c074ba2c6276a7080aa6531fec40abdb32cc7979bc82d90e993ef
Deleted: 2df3374482ddaf57b1cb98c93040002f26f1000b10004c2a39d4e97ad22b011c
Deleted: 44d577c28426154d3f159cea9a38d5d9af9c35dfa69450fa7193beec8795d89f
Deleted: d7b94927fbad4bf0951aaa9285bdc37b890eea571e02d3ee9c1eb8123f5deccf
Deleted: 60d043dafa118397a11e4e8b99f3eed94e97f84b2e2cb5fdf51324576eb24a8e
Deleted: a8eae6bae18ac06d4777dfb9ec9c09bc754b96de6a003b4a8ff08e9d4925cda6
Error: No such image: 718546056f5e
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 95c45df1e4bb
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: ade53f9727e7
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: baf4543ca7d1
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 694ea6c8964f
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: ed473ae9a77c
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 44d577c28426
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 2df3374482dd
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: d7b94927fbad
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: 60d043dafa11
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: No such image: a8eae6bae18a
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, 03d2f86d1a14 wasn't deleted
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, 73287e15959a wasn't deleted
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, cannot delete 7fab7bd8a297 because the container 410a1e7daaf1 is using it
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, bcf870ec7125 wasn't deleted
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, 2dbebf6d4d75 wasn't deleted
2015/09/14 13:29:44 Error: failed to remove one or more images
Error: Conflict, 2c3dfb388d6f wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 3995bec58d7e wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, f4f8ca96169a wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 4d2e883408fc wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 9ac0428c11a5 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, cannot delete 915002d5049f because the container d60ce7e7c90b is using it
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, cd586d568f79 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, a4fed81b1473 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, bdca5cbdf09f wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 35b6c9262a1f wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, c818b2a6370e wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, b8f7b8b05a2e wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, cc5e209fc876 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, cannot delete abb3db6fb60c because the container 5730cf0fd858 is using it
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 10b9ea1e41f0 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 700a10b7b650 wasn't deleted
2015/09/14 13:29:45 Error: failed to remove one or more images
Error: Conflict, 30d9ef2dff6e wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Error: Conflict, 89cda0b54389 wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Error: Conflict, 4e5299a2abcb wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Error: Conflict, 56fbae8a3a14 wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Error: Conflict, 257f990baceb wasn't deleted
2015/09/14 13:29:46 Error: failed to remove one or more images
Untagged: jpetazzo/nsenter:latest
Deleted: f495905a267097922d909bf498e6af89cfa13af16bbbba8a732c8888c9014541
Deleted: 52ee3273029ce1f373a3e1f1d77995016aa344d784817795be50c55a3cd73f55
Deleted: c1bd260973faa67aa1ba7682aea81a3238bb026202364e603ef5a78e4061ffc1
Deleted: 851b05e19fc2f228f614110fa44b9664fef334dd584f921ce8ddc702850fbfe5
Deleted: 731d521b86c71ed616df1046455976b6df9800c1c1efea48673ae0701d132c11
Deleted: b3073fbfdb5a39028f29eb846c368cc2a62f432804a0e8da356165d2cbd702d8
Deleted: d577e4cdab15cbdba7bf67e8443392788ba70f79ee943d2ed70e7f880d8adfb2
Deleted: 7199b299c30f513eba072ff26043c07014ebb7d680fcfb9e1e89804250c79255
Deleted: 1ab95ee9407f64147046a806a424fcc9dbfe7e38e3e41c44ac748e9c66be5504
Deleted: 66e54c5b5d94369d11861a6cc4d0b2ac75dd7e0cbbfdaa6734a0692e1242d4bf
Deleted: 5845be93387406193033bd0489ae2e9a9ef774f9f169dc884de84084865eb6a2
Deleted: b0b29173013c55c6009a38f843798d12cb133148b777fdd372db595ea09460f9
Deleted: f485dd2ac292809b7cb90ade149dfdc78cbed1fa9960529585014e521e4c2d96
Deleted: 7a37928bf6a187403404df5214322a5ff73ba09151324fadbef6671f610943b0
Deleted: badcb042f7c92d0847f4a8372cf11fc1af3ab4b3d0f7da4013d3605f83905101
Deleted: a841579d7409739248acad87cfe14f80692287d904b35a15211e02ac0327c7c4
Deleted: 743c1e45e688224fa627bf95950552f05a6443eaf2985d7cbc70d29ac64960be
Error: No such image: 52ee3273029c
2015/09/14 13:30:36 Error: failed to remove one or more images
Error: No such image: c1bd260973fa
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 851b05e19fc2
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 731d521b86c7
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: b3073fbfdb5a
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: d577e4cdab15
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 7199b299c30f
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 1ab95ee9407f
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 66e54c5b5d94
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 5845be933874
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: b0b29173013c
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: f485dd2ac292
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 7a37928bf6a1
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: badcb042f7c9
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: a841579d7409
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: No such image: 743c1e45e688
2015/09/14 13:30:37 Error: failed to remove one or more images
Error: Conflict, 6cfbb482a117 wasn't deleted
2015/09/14 13:30:37 Error: failed to remove one or more images
Untagged: busybox:latest
Deleted: 2d8e5b282c81244037eb15b2068e1c46319c1a42b80493acb128da24b2090739
Deleted: 9f4e93171ec525221fa9013d0e21f8690cef68590664eb5249e0b324c5faf31a
Deleted: 6c991eb934609424f761d3d0a7c79f4f72b76db286aa02e617659ac116aa7758
Deleted: 511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
Error: No such image: 9f4e93171ec5
2015/09/14 13:30:41 Error: failed to remove one or more images
Error: No such image: 6c991eb93460
2015/09/14 13:30:41 Error: failed to remove one or more images
Error: No such image: 511136ea3c5a
2015/09/14 13:30:41 Error: failed to remove one or more images

Thanks

Mario