Can Swift serve up a 301/303 redirect?

Asked by Tim Newsome on 2013-02-28

Can I upload a redirect to Swift?

The problem I have is that I have a data store of a lot of files, many of which are identical. When a client requests a file, I would like to redirect that client to the canonical URL for that file (also stored in Swift). This allows clients to maximize their cache hit rate, and prevents them from storing multiple copies of the same thing in their cache.

As a quick example, I'd like to do something like:
curl –X PUT -i \
    -T JingleRocky.jpg \
    https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg
curl –X PUT -i \
    -T /dev/null \
    -H "X-Object-Meta-Redirect-303: https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg" \
    https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/dog1.jpg

Then when I GET https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/dog1.jpg, Swift would return an HTTP 303 return code, with a redirect to https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg.

Is this possible?
If not, how much work would you estimate this is to implement by an experienced python programmer?
Would such a change be considered for inclusion into the main project?

Thank you,
Tim

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
Samuel Merritt
Solved:
2013-02-28
Last query:
2013-02-28
Last reply:
2013-02-28
John Dickinson (notmyname) said : #1

It's an interesting thought, but I'd first be a little concerned about public swift clusters being abused by this. However, that's with about 10 seconds of thought and it may not be a major concern.

Instead of doing 3xx redirects, you can use the existing large-object manifest files for this. You can upload, eg, user_guide_latest.pdf, and have that reference a specific file on the back end (eg versions/user_guide_20130301.pdf). Then you can stage new versions and update the link very quickly.

Docs for large object support: http://docs.openstack.org/developer/swift/overview_large_objects.html

--John

On Feb 28, 2013, at 2:46 PM, Tim Newsome <email address hidden> wrote:

> New question #223106 on OpenStack Object Storage (swift):
> https://answers.launchpad.net/swift/+question/223106
>
> Can I upload a redirect to Swift?
>
> The problem I have is that I have a data store of a lot of files, many of which are identical. When a client requests a file, I would like to redirect that client to the canonical URL for that file (also stored in Swift). This allows clients to maximize their cache hit rate, and prevents them from storing multiple copies of the same thing in their cache.
>
> As a quick example, I'd like to do something like:
> curl –X PUT -i \
> -T JingleRocky.jpg \
> https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg
> curl –X PUT -i \
> -T /dev/null \
> -H "X-Object-Meta-Redirect-303: https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg" \
> https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/dog1.jpg
>
> Then when I GET https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/dog1.jpg, Swift would return an HTTP 303 return code, with a redirect to https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg.
>
> Is this possible?
> If not, how much work would you estimate this is to implement by an experienced python programmer?
> Would such a change be considered for inclusion into the main project?
>
> Thank you,
> Tim
>
> --
> You received this question notification because you are a member of
> Swift Core, which is an answer contact for OpenStack Object Storage
> (swift).

Tim Newsome (tnewsome) said : #2

I don't think that does what I want. I really do want a 3xx redirect. The issue is that I have clients fetching the same file through a lot of different URLs. So on the server I'll have (with simplified paths)
.../project1/baseFile
.../project2/baseFile
.../project3/baseFile
.../canonical/baseFile

I'll have clients getting the baseFile for project[123], and they're really all the same file. These clients are using a standard caching HTTP proxy. Without the redirect, the proxy will end up downloading and keeping 3 identical copies of baseFile in its cache. If project[123]/baseFile redirect to canonical, however, then the proxy will cache the fact that those are redirects (assuming 301), and only download and store the single canonical/baseFile.

I have the same problem in reverse. Many clients are adding files to this database. Each client can quickly check whether a canonical version exists. (This works by putting a hash of the file contents into the name of the canonical file.) Only if it does not, does the client need to upload the canonical version. Once that's done it's quick enough to add a 'redirect' in the appropriate project directory.

Currently I have this setup working with Apache on a single server, but I'd like to use Swift for the reliability and scaling features it provides.

I could try and fix all the clients to not need the redirects, but I don't have as much control over the clients as I do over the server. Another approach is to build something that is a front-end for Swift that does the what I want, but it feels like adding the functionality to Swift directly would not be much extra work, and provide a much better long-term solution.

Best Samuel Merritt (torgomatic) said : #3

One could write a custom Swift middleware to handle this. The middleware would pass the request on through to the proxy, and then if (a) the request is a GET for an object and (b) the response headers have X-Meta-Object-Redirect-303 in them, then return a redirect response instead.

Samuel Merritt (torgomatic) said : #4

Note that if you wanted to add this to Swift, you'd almost certainly be told to write it as a middleware, so "write some middleware" does not imply "keep it out of core Swift." The above suggestion makes no statements about the suitability of such a middleware for inclusion in core Swift, either positive or negative.

Tim Newsome (tnewsome) said : #5

Thanks Samuel Merritt, that solved my question.

Tim Newsome (tnewsome) said : #6

Thanks for pointing me towards middleware. That looks like the right solution. Disregard my question about getting it included in core swift. With a flexible system like this there's no need!