Fuel for OpenStack

Fuel harddisk size in a scale testbed

Asked by SongBeng

Hi,
I am attempting to use Fuel to deploy 90 compute nodes and one controller. The Fuel VM is allocated 200gig of harddisk space, and I am noticing after a few days that /var is at 100%. What is the recommended size for Fuel to support a large deployment? and is there a script I can run to clean up /var or what files can I safely delete from /var?

[root@fuel log]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.7G 1.3G 8.0G 14% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sda3 194M 28M 156M 16% /boot
/dev/mapper/os-var 186G 186G 20K 100% /var

thanks

Song

Question information

Language:
English Edit question
Status:
Answered
For:
Fuel for OpenStack Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Fabrizio Soppelsa (fsoppelsa) said : 		#1

Hi SongBeng,
Logs are growing in /var because events are raised in detail from deployed nodes and written in /var/log/remote
It's possible that after having deployed many nodes, this directory expands up to the /var size.
You can try forcing a logrotate ( `logrotate -f /etc/logrotate.conf` ), or stop the docker container for rsyslog, clean the logs, and restart rsyslog ( `dockerctl stop rsyslog; <CLEAN LOGS>; dockerctl start rsyslog` ).

Please be aware that a 100% full /var may lead to problems . We've addressed this issue with this bug https://bugs.launchpad.net/fuel/+bug/1371757

Best regards,
Fabrizio.

Revision history for this message
Miroslav Anashkin (manashkin) said : 		#2

Greetings SongBeng,

Could you please check what directories and what files have the greatest size on your Fuel master node and post the results here?

There is logrotate set up on Fuel master and recently we discovered logrotate misconfiguration for some services.
This issue addressed in this new bug https://bugs.launchpad.net/fuel/+bug/1378327

You may fix the configuration file names inside logrotate, mentioned in the bug - please try to change
20-fuel.conf to 20-fuel-docker.conf in /etc/cron.hourly/logrotate

Then run `dockerctl shell rsyslog` to get shell to rsyslog container and make the same change inside the rsyslog shell, so the changes take effect inside rsyslog container as well.

Can you help with this problem?

Provide an answer of your own, or ask SongBeng for more information if necessary.

To post a message you must log in.