Fuel for OpenStack

One of my controllers filled / with mysql-bin files, how do I recover from this?

Asked by Erik Swanson

I have a Fuel 6.0 (community edition) cluster with 3 controllers in an HA setup. (Ubuntu-based install.)

On one of the contollers, the root partition is now full because there are a lot of MySQL binary log files (/var/lib/mysql/mysql-bin.******) that nothing is cleaning up. Nova is unresponsive: all hypervisors appear as down and I cannot list instances, yet the VMs themselves are still running.

How do I recover from this without data loss, and how do I prevent this from happening again?

Question information

Language:
English Edit question
Status:
Solved
For:
Fuel for OpenStack Edit question
Assignee:
No assignee Edit question
Solved by:
Fabrizio Soppelsa
Solved:
Last query:
Last reply:
Revision history for this message
Best Fabrizio Soppelsa (fsoppelsa) said : 		#1

Greetings Erik,
we already had a conversation in IRC, but I will repost the solution here for history reasons.

On any controller:
1. Gain some space, pruning some unnecessary logs or things in /var and /tmp
2. Take a dump: `mysqldump --all-databases`
3. Login into MySQL: `mysql`
4. Clean the binary logs:
> flush logs;
> reset master;

It may take a while. After that, check that the services are recovered (like `nova list`).

Filed a blueprint to keep these logs reasonable in size https://blueprints.launchpad.net/fuel/+spec/clean-mysql-logs
As a recommendation, you can check the disk space on the controllers with Zabbix - which we ship with Fuel - or with a tool of your preference.

Revision history for this message
Erik Swanson (erik-swanson) said : 		#2

Thanks Fabrizio Soppelsa, that solved my question.

Revision history for this message
yexian (yexianwolail) said : 		#3

As a recommendation, you can check the disk space on the controllers with Zabbix - which we ship with Fuel - or with a tool of your preference.

HI ! Fabrizio Soppelsa (fsoppelsa)
Zabbix database size is very big, how to clean up the zabbix data？