OpenStack Object Storage (Swift)

How many objects can be contained at one-container

Asked by Ted,Jung on 2011-07-01

To anyone

Let us suppose
Some service have 10mil users
Each user generates 10 objects everyday

If 10 mil users mapped to just one swift user
If The swift user have one container

Each day 100 mil objects are stored to just one container

In this case, what is the problem
Is this right thing or not?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
John Dickinson
Solved:
2011-07-03
Last query:
2011-07-03
Last reply:
2011-07-01
Best John Dickinson (notmyname) said : #1

The concern is that a single container db that is handling all of these objects. Each update (new object) then has to update the container. These updates get slower as the container gets larger (the fixed cost of the btree update for each object PUT gets larger). Object reads, however, are unaffected (because they don't even touch the container db).

We recommend that you shard large containers and aim for about 10 million objects in each if you are running account and containers servers on standard hard drives. At large scale, running your account and container servers on boxes with fast IO can also improve things dramatically. For example, in internal tests, we've seen containers with a billion rows still handle 400 updates a second when they have fast IO (read: SSDs).

--John

On Jun 30, 2011, at 11:05 PM, Ted,Jung wrote:

> New question #163356 on OpenStack Object Storage (swift):
> https://answers.launchpad.net/swift/+question/163356
>
> To anyone
>
> Let us suppose
> Some service have 10mil users
> Each user generates 10 objects everyday
>
> If 10 mil users mapped to just one swift user
> If The swift user have one container
>
> Each day 100 mil objects are stored to just one container
>
> In this case, what is the problem
> Is this right thing or not?
>
> --
> You received this question notification because you are a member of
> Swift Core, which is an answer contact for OpenStack Object Storage
> (swift).
> This email may include confidential information. If you received it in error, please delete it.

Ted,Jung (chungjl) said : #2

Thanks John Dickinson, that solved my question.

Ted,Jung (chungjl) said : #3

John Thanks for your answer

One more issue is about SSD
How to map SSD to ring partition like Hard Disk Drive
Can you show me detail description

I agree with you that SSD can show more efficient performance and more fast IO than HDD
how to setup and configure the SSD
Does configuration the same as HDD configuration?

thanks John

regards
/Ted

John Dickinson (notmyname) said : #4

Yes, the setup for an SSD should be the same as for an HDD. When building the rings, though, if you want to have IOPS for accounts and containers, just only use the SSDs for those rings and the HDDs for the object ring.

On Jul 5, 2011, at 12:55 AM, Ted,Jung wrote:

> Question #163356 on OpenStack Object Storage (swift) changed:
> https://answers.launchpad.net/swift/+question/163356
>
> Ted,Jung posted a new comment:
> John Thanks for your answer
>
> One more issue is about SSD
> How to map SSD to ring partition like Hard Disk Drive
> Can you show me detail description
>
> I agree with you that SSD can show more efficient performance and more fast IO than HDD
> how to setup and configure the SSD
> Does configuration the same as HDD configuration?
>
> thanks John
>
> regards
> /Ted
>
> --
> You received this question notification because you are a member of
> Swift Core, which is an answer contact for OpenStack Object Storage
> (swift).
> This email may include confidential information. If you received it in error, please delete it.