Multiple Server Swift Installation—rebalance

Asked by eugene lin on 2011-09-06

I use four Virtual Machines to set up swift in muitiple Sever.
one for proxy&auth node ,and others for storage node.

root@swift-proxy:/etc/swift# ./verify.sh
account.builder, build version 3
262144 partitions, 2 replicas, 1 zones, 3 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
             0 5 192.168.1.84 6002 sdb1 100.00 0 -100.00
             1 5 192.168.1.82 6002 sdb1 100.00 0 -100.00
             2 5 192.168.1.69 6002 sdb1 100.00 0 -100.00
container.builder, build version 3
262144 partitions, 2 replicas, 1 zones, 3 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
             0 5 192.168.1.84 6001 sdb1 100.00 0 -100.00
             1 5 192.168.1.82 6001 sdb1 100.00 0 -100.00
             2 5 192.168.1.69 6001 sdb1 100.00 0 -100.00
object.builder, build version 3
262144 partitions, 2 replicas, 1 zones, 3 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
             0 5 192.168.1.84 6000 sdb1 100.00 0 -100.00
             1 5 192.168.1.82 6000 sdb1 100.00 0 -100.00
             2 5 192.168.1.69 6000 sdb1 100.00 0 -100.00

but

root@swift-proxy:/etc/swift# ./rebalance.sh
Traceback (most recent call last):
  File "/usr/bin/swift-ring-builder", line 651, in <module>
    Commands.__dict__.get(command, Commands.unknown)()
  File "/usr/bin/swift-ring-builder", line 512, in rebalance
    parts, balance = builder.rebalance()
  File "/usr/lib/pymodules/python2.7/swift/common/ring/builder.py", line 243, in rebalance
    self._initial_balance()
  File "/usr/lib/pymodules/python2.7/swift/common/ring/builder.py", line 403, in _initial_balance
    while available_devs[index]['zone'] in other_zones:
IndexError: list index out of range
Traceback (most recent call last):
  File "/usr/bin/swift-ring-builder", line 651, in <module>
    Commands.__dict__.get(command, Commands.unknown)()
  File "/usr/bin/swift-ring-builder", line 512, in rebalance
    parts, balance = builder.rebalance()
  File "/usr/lib/pymodules/python2.7/swift/common/ring/builder.py", line 243, in rebalance
    self._initial_balance()
  File "/usr/lib/pymodules/python2.7/swift/common/ring/builder.py", line 403, in _initial_balance
    while available_devs[index]['zone'] in other_zones:
IndexError: list index out of range
Traceback (most recent call last):
  File "/usr/bin/swift-ring-builder", line 651, in <module>
    Commands.__dict__.get(command, Commands.unknown)()
  File "/usr/bin/swift-ring-builder", line 512, in rebalance
    parts, balance = builder.rebalance()
  File "/usr/lib/pymodules/python2.7/swift/common/ring/builder.py", line 243, in rebalance
    self._initial_balance()
  File "/usr/lib/pymodules/python2.7/swift/common/ring/builder.py", line 403, in _initial_balance
    while available_devs[index]['zone'] in other_zones:
IndexError: list index out of range

root@swift-proxy:/etc/swift# less rebalance.sh
#!/bin/bash
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance

anyone could tell me why???

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
eugene lin
Solved:
2011-09-06
Last query:
2011-09-06
Last reply:
2011-09-06
Phil Rogers (progers-n) said : #1

I have had this problem a few weeks ago

             0 5 192.168.1.84 6002 sdb1 100.00 0 -100.00
             1 5 192.168.1.82 6002 sdb1 100.00 0 -100.00
             2 5 192.168.1.69 6002 sdb1 100.00 0 -100.00

Try creating you nodes with incremental numbers
instead of 5 try 1 2 3
Assuming you have a test system delete you /etc/swift/account.builder , Container.builder and object.builder and then recreate

             0 1 192.168.1.84 6002 sdb1 100.00 0 -100.00
             1 2 192.168.1.82 6002 sdb1 100.00 0 -100.00
             2 3 192.168.1.69 6002 sdb1 100.00 0 -100.00

cd /etc/swift
swift-ring-builder account.builder create 18 3 1
swift-ring-builder container.builder create 18 3 1
swift-ring-builder object.builder create 18 3 1

export ZONE=1
export STORAGE_LOCAL_NET_IP=192.168.1.84
export WEIGHT=100
export DEVICE=sdb1
swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
export ZONE=2
export STORAGE_LOCAL_NET_IP= 192.168.1.82
export WEIGHT=100
export DEVICE=sdb1
swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
export ZONE=3
export STORAGE_LOCAL_NET_IP=192.168.1.69
export WEIGHT=100
export DEVICE=sdb1
swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT

then do
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance

this should then create the ring.gz files you will then need to then copy the .gz files and the swift.conf over to all storage nodes

Hope that helps

Cheers
Phil

eugene lin (linyouqing7) said : #2

Phil Rogers
Thank you very much !!!
"Try creating you nodes with incremental numbersinstead of 5 try 1 2 3"
This solved my problem

rudal (rudy-rudal) said : #3

Hi Phil, I am currently using 12.04 LTS ubuntu.
I tried to follow your instruction but I couldnt seem able to do what you have.

This is what I have:

root@proxy-node:/etc/swift# swift-ring-builder account.builder
account.builder, build version 2
262144 partitions, 3 replicas, 2 zones, 2 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
             0 1 10.188.188.101 6002 sdb1 100.00 0 -100.00
             1 2 10.188.188.102 6002 sdb1 100.00 0 -100.00
root@proxy-node:/etc/swift# swift-ring-builder container.builder
container.builder, build version 2
262144 partitions, 3 replicas, 2 zones, 2 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
             0 1 10.188.188.101 6001 sdb1 100.00 0 -100.00
             1 2 10.188.188.102 6001 sdb1 100.00 0 -100.00
root@proxy-node:/etc/swift# swift-ring-builder object.builder
object.builder, build version 2
262144 partitions, 3 replicas, 2 zones, 2 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id zone ip address port name weight partitions balance meta
             0 1 10.188.188.101 6000 sdb1 100.00 0 -100.00
             1 2 10.188.188.102 6000 sdb1 100.00 0 -100.00
root@proxy-node:/etc/swift#

Then I tried to rebalance them:

root@proxy-node:/etc/swift# swift-ring-builder object.builder rebalance
Traceback (most recent call last):
  File "/usr/bin/swift-ring-builder", line 690, in <module>
    Commands.__dict__.get(command, Commands.unknown)()
  File "/usr/bin/swift-ring-builder", line 528, in rebalance
    parts, balance = builder.rebalance()
  File "/usr/lib/python2.7/dist-packages/swift/common/ring/builder.py", line 254, in rebalance
    self._initial_balance()
  File "/usr/lib/python2.7/dist-packages/swift/common/ring/builder.py", line 412, in _initial_balance
    while available_devs[index]['zone'] in other_zones:
IndexError: list index out of range

Anything that I did wrong / missing?

Thank you

Samuel Merritt (torgomatic) said : #4

You're running a very old version of Swift. With that version, you need to have devices in at least 3 zones to use 3 replicas, otherwise you get crashes like this. Either get more devices, use fewer replicas, or upgrade Swift.