1Gb for internal network is enough for multi-petabyte deployments?

Hi all,

we are in the early design phase of a multi-petabyte Swift deployment and there are some networking recommendations that make us think twice if our design is correct.

We are going to use 100TB servers as storage nodes and making a quick calculation, if a server fails, on a 1Gb connection it would take more than 10 days to fully replicate one of this failed storage nodes.

We think a 10GbE connection or even a 40Gb Infiniband based solution makes more sense... but reading the docs, forums and some presentations in the web, it seems that nobody is using >1Gb connections to connect the storage nodes each other. 10GbE is recommended in the external network, but not in the internal network.

May be our 100TB per server approach is too aggressive, but I think KT and even Rackspace have to be around 70TB per storage node, which make us think they had addressed our concerns before.

So 1Gb is really enough?