Sqlite and .ts maintenance with deletes

Have you checked to make sure the replicators are running? They are what is responsible for reclimation.

They are running. Also, I've noticed that although the SQLite DB is large, there are no deleted objects older than a week, so data seems to be reclaimed there.

However, regarding the .ts nodes, their number is always increasing and I see nodes 2 months old which is the age of the system. Clearly they are not being reclaimed. Is there any scenario when this is supposed to happen?

Can I/should I delete them manually from the file system?

Thanks for the help.

How active is the cluster? It looks like the reclamation happens when the suffix hashes are re-hashed (which usually only happens when a change, like a new object PUT, happens in that suffix. If nothing has changed in that suffix dir, then it is likely it hasn't had a chance to crawl, and thus reclaim the .ts files.

I'd say the cluster is pretty active. We get 400k uploads per day and the average file size is 2MB.

As for the "suffix dirs", I read the http://docs.openstack.org/developer/swift/overview_replication.html documentation but could not find out for certain what they mean. However, I have a lot of uploads/deletes for a specific container, and it is here that the problem has been worse since I already have much more tombstones than regular files. Since this container/dir is changed so frequently, shouldn't it be crawled and .ts files cleaned? Don't these happen in the same "suffix"?

Thanks for the help.

How can I force activity on a certain suffix dir? Is it the suffix of the filename? Do I just have to upload an object to the same dir with a filename with the same suffix?

Thanks!

The suffix is the last 3 nybbles of the hash of the object name (that's MD5(object-name + per-cluster-suffix)). It's there so that there's not a ton of subdirectories in the partition directory. Think of a URL like "http://archive.ubuntu.com/ubuntu/pool/main/c/cairo/cairo_1.6.0.orig.tar.gz"; the "/c/cairo/" part is just there to spread the files around a bit. Same thing.

So, if you wanted to force activity on a certain suffix directory, you'd need to come up with an object name such that the last 3 nybbles of hash(objectname) are the suffix *and* the first P bits are the same as the partition number, where P is your cluster's part_power setting. That's computationally feasible (find at most 44 bits of collision), but it's not trivial.

Well, I suppose this is the best I can do. But it is rather painful doing this on multiple partitions....