OpenStack Object Storage (swift)

how to check object recovery status in disk replacement

Asked by Song Jie

When a disk is failed (discovered by swift-drive-audit or other disk monitor tools), the common case is perform disk replacement.

In failure window, hinted handoff is being selected. Once new disk is added (with same capacity, mount point), there is no need to trigger ring rebalance as Swift promise self-heal by its replicator that copy newest version back to the new disk (hinted handoff is removed afterwards).

The question is how to list all the affected replicas that laid in the bad disk, it should be a normal case that operator would like to know all the affected ones and track if self-heal have recovered all of them.

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
Song Jie
Solved:
Last query:
Last reply:
Revision history for this message
John Dickinson (notmyname) said : 		#1

Use swift-ring-builder. From it's usage string,

swift-ring-builder <builder_file> list_parts <search-value> [<search-value>] ..
    Returns a 2 column list of all the partitions that are assigned to any of
    the devices matching the search values given. The first column is the
    assigned partition number and the second column is the number of device
    matches for that partition. The list is ordered from most number of matches
    to least. If there are a lot of devices to match against, this command
    could take a while to run.

This is a tool that can let you find the partitions that were assigned to a particular volume (or set of volumes). With a list of partitions, you can choose to use the swift-*-replicator and give it the partitions to replicate right now.

Revision history for this message
Song Jie (nightelf1984) said : 		#2

Ok, I see. Thanks for the answer, it helps.

Should further checks is needed once partition is replicated? I mean partition folder contains all the objects that hash falls to it (vnode), if the replicator tool reported succeed then should I think all those objects are copied, I should take a closer look at the code.

Another consideration is for self heal the replaced disk may suffer heavy IO load if data in the disk is huge. This might leads to recovery slow or disk failure again (I am not sure if throttling existed in such case). So in such case, may be remove the bad disk from the ring and then add new disk with moderate rebalance in phased manner could be consider, anyway need to take a further look.