OpenStack Object Storage (swift)

Rebalancing IO Overhead

Asked by Rob Janney

What is the best way to handle the IO overhead of rebalance operations? Are there best practices about when to rebalance & how to prevent the rebalancing IO overhead from degrading the performance of storing new data to Swift? I know that Swift should never be over 80% full, so I would assume expansion should be planned around 70% full so that the rebalance operation can slowly move data over to the new nodes without reaching 80%?

As an example, lets consider a system with 1000 2TB devices spread over 20 machines. Lets say that data has been evenly stored data over all 1000 devices, such that they are all about 70% full. Would it be reasonable to acquire 200 more devices (also 2TB) and rebalance? Or should the minimum growth size be a larger percentage of the whole to make rebalancing less problematic?

The documentation I have seen so far talks about 1GB nics for the internal storage network. Is this a bottleneck during rebalancing? Or are the 1 GB nics sufficient for this task?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
Florian Hines
Solved:
Last query:
Last reply:
Revision history for this message
Best Florian Hines (pandemicsyn) said : 		#1

Depending on how fast your growth rate is , provisioning and deploying more drive space at 70% of capacity is probably reasonable (its what we do).

How much storage to add is totally up to you and your growth rate. You could just add 1 server per zone per week, or 20 every 6 months..what ever keeps you at the utilization spot that you want and how much a head of the curve you want to get. We try to keep utilization around 70% so we're adding a full new Zone on a set schedule.

The network can certainly be a bottle neck during the rebalance (since replication/rsync is going move all those files around) as can the drive IO. This is especially the case on the new devices that your adding in since they're the ones on the receiving end of all that network traffic and file io. You'll essentially be having all your existing servers trying to hand off a % of their files to the new devices.

The easiest way to control how slow/fast replication happens is by tweaking the "max connections" in /etc/rsyncd.conf on the new nodes that you're adding to the ring. The first time its probably best to start slow and then slowly increase it until you know what a gives you a good balance on your environment.

A couple of tips:

Update and rebalance your account/container rings first (should be fairly fast). If you do object and container at the same time you'll probably see increased async pendings and container updates might get painful.

If you're adding new devices to existing zones rather than completely new zones add/rebalance a single zone at a time (you probably don't want multiple replica's in flight).

Start out by adding devices at lower weights. For example if your default device weight is 100, add the new devices in at 25 then let the system rebalance, then increase their weight to 50 rebalance, then 75, and finally at 100. It will take the same amount of time overall but its a bit safer.

You can keep an eye on progress using 'swift-stats-report -d' or by simply checking the utilization on the new disks.

Revision history for this message
Rob Janney (rob-janney) said : 		#2

Thanks Florian Hines, that solved my question.