What's the best way to backup?

I won't speculate as to the reasons behind your requirements, but I will offer my comments. I am assuming that this would be for a small swift deploy (since you mention pulling the changes into a temp system).

1) My first response is that the best practices are that you shouldn't try to make a backup of a swift cluster. Since swift systems can grow to enormous sizes, the only realistic backups involve a second, duplicate cluster. Generally costs (financial and otherwise) prohibit doing that. Swift is geared towards multi-petabyte clusters where backing up to tape is not even considered. The multiple replicas of the data that swift stores are designed to provide data durability.

2) No, there is no feature of swift that allows you to keep track of what has changed for today. There is no central metadata store like in MogileFS. However, you could walk the storage nodes to look for object timestamps or, more realistically, look at swift logs to see what has been PUT. Although fairly simple, these would be tools that you would need to write. However, dealing with logs from a multi-proxy setup can get difficult in some conditions (eg. an overwrite and a delete in close temporal proximity on two separate proxy servers).

3) Beware of scaling the central metadata db in MogileFS. It can be done, but it gets expensive and hard.

4) Perhaps an answer would be to use a filesystem that supports snapshots for the swift storage nodes. Take snapshots of each storage node and use those for backup. I don't really know how such a system would work, though.

5) Some piece of middleware that you write could log each operation that you need to track into your "to be archived on tape" storage. This is doable, but it gets hard at scale. It doesn't offer much benefit over looking at the proxy logs directly.

6) Perhaps the most realistic option to back up a swift cluster is to use a second swift cluster. Intercept each request and send it to each cluster. Of course the hard part comes when one cluster succeeds and the other errors. And it doesn't solve your "backup to tape" requirement.

Depending on your requirements (including total eventual size of your swift cluster) some of these options may be feasible, but they all generally break as your swift cluster gets larger.

Thanks for your reply John, you provided a lot of things to think about. I certainly would prefer swift over mogilefs due to the metadata db scaling issue and the general increasing momentum of swift. It seems like a better long term solution for us.

There's a little bit of information that I probably should have shared about our specific implementation goals that may help out other people and spur some more discussion.

The deployment is going to be in the low hundreds of TBs with various types of data stored there; disk backups, PDF documents and thumbnail/png images of pdfs. The backups and images of PDFs are not critical and life will go on if they're lost. Longer term we'll want a secondary datacenter site with multi-region replication running between them.

On the cluster there will be a critical set of data. These are the PDFS (mostly financial documents), growing at about 2 TB/month. That will need to be backed up since a single loss of data could cost the company hundreds of thousands of dollars in financial exposure. Using swift or mogilefs gives us the benefit of having replicated copies for almost all failure situations except for a true SHTF where either our software does something wrong or some unknown internal of swift goes crazy and blows up the entire cluster. That's the failure scenario we're trying to watch out for. Additionally we have a strict compliance team and outside auditors that would never approve data storage process where we're not actively backing up to tape offsite.

In our short term case, watching the proxy logs for writes to specific containers probably is the best idea presented since we can look for writes to specific containers. Thanks for that idea.

I was looking into some of the multi-region replicated code and was wondering if there wasn't a better solution, that would obviously require more work.

I believe you should be able to setup a process that understands the switft protocols and can accept only a push from the main cluster. This backup system would keep enough metadata around to keep synchronization working between the container on the main cluster and the backup system. The backup system would accept the incoming files into a backup directory and then when the backup process is run, the files are marked that they are now backed up and the directory is purged.

Backing up would then work as files are written to disk, container by container, and the DB index could be saved also if the entire backup db system goes down. When you need to restore data, you can find out which tape has the latest copy of the files, fetch that and have the backup system push the file back to the main cluster. Restoring all data would be a nightmare, but if you're having tapes delivered from your offsite vendor at 2am, something terrible probably already happened.

I haven't spent more than a half hour looking through the node to node communication or multi site replication so there's probably hundreds of problems with that working, but it was just a thought.

We are working on a container sync feature for swift that will allow you to sync a container from one cluster to another. This would allow you to have 2 clusters (with appropriate DR separation) that keep in sync.

The code branch for this feature is https://code.launchpad.net/~gholt/swift/consync