404 errors after rebalance object ring

There's a lot to unpack here. Without more info about your cluster, it's pretty hard to make specific recommendations, but I can stick to some general things. Please also check out https://ask.openstack.org/en/questions/ and feel free to ask Swift developers question in #openstack-swift on freenode IRC.

* In your particular case, it sounds like you are adding a huge percentage of capacity all at once. Generally, it's better to add in smaller increments so that the data movement associated with ring rebalancing won't impact normal cluster operation. The ring-builder tool will only rebalance enough ring partition as is safe at one time. This means it will normally take multiple rebalance steps to add a significant amount of new capacity.

* Yes, it's important to wait for a replication cycle to complete before doing another rebalance. This will prevent service disruption caused by the ring being updated faster than data is moved in the cluster (ie the 404s you were seeing). The min_part_hours setting on the ring should be matched to your replication cycle time to give you some protection from changing things too quickly. (ie you should monitor replication and update min_part_hours accordingly)

* Unfortunately, replication can be quite slow when dealing with small files. Some things you can do to help speed it up are using a separate replication network, increasing rsync max_connections settings, increasing replication concurrency, and using a servers-per-port object server configuration.

* Monitoring replication cycle time is very important, as well as monitoring handoff partitions (eg with swift-recon or swift-dispersion-report).

I don't envy the situation you're in, but it's absolutely possible to move forward and get from where you are today to a healthy Swift cluster with all the extra capacity you're trying to add.

Thanks a lot for answer.
So now I have situation while fill my cluster I have many PUT requests, but I have enought load capacity for increase replication speed, and need to move in this way.

I would replication run more fast, tried set more replication threads but it didn't very help me.
I have impression that swift 'afraid' to overload my HDDs despite I have enouth load iops capacity.
I Will be very grateful if you will help me find parameters which can increase replication speed.
I have 3 storage nodes, each has 25 HDD x 6T, and replica factor 3.

This is my object-server.conf
---------------
[DEFAULT]
bind_port = 6000
user = swift
swift_dir = /etc/swift
devices = /mnt/swift
mount_check = True
log_level = ERROR
conn_timeout = 5
container_update_timeout = 5
node_timeout = 5
max_clients = 4096

[pipeline:main]
pipeline = healthcheck recon object-server

[app:object-server]
use = egg:swift#object
replication_concurrency = 800
replication_one_per_device = True
replication_lock_timeout = 30

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
recon_lock_path = /var/lock

[object-replicator]
concurrency = 800
run_pause = 5
interval = 5
log_level = DEBUG
stats_interval = 10
rsync_io_timeout = 60

[object-reconstructor]

[object-updater]
concurrency = 200
interval = 20
slowdown = 0.008
log_level = DEBUG

[object-auditor]
interval = 300

[filter:xprofile]
use = egg:swift#xprofile

UPD:
I'm use separate replication network with 10G link
rsync max_connections and replication concurrency = 800
but don'r use servers-per-port object server

I think I need more agressive replication mode)

Can anybody help with this problem ? Is this issue unresolved by swift( ?

This question was expired because it remained in the 'Open' state without activity for the last 15 days.