OpenStack Object Storage (swift)

Get object failed with 503 error

Asked by kalrey on 2013-06-27

Hi,guys!
I've got a 503 error code in swift log while getting objects by Multiple client.I trace the log trying to figure out the problem of the issue.but there are some information make me confused.I pickued up relational part in proxy log.

>>
 Jun 26 11:57:34 station215 proxy-server ERROR with Object server 192.168.52.180:6020/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187)
Jun 26 11:57:35 station215 proxy-server ERROR with Object server 192.168.52.181:6010/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187)
Jun 26 11:57:36 station215 proxy-server ERROR with Object server 192.168.52.186:6030/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187)
Jun 26 11:57:37 station215 proxy-server ERROR with Object server 192.168.52.182:6040/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187)
Jun 26 11:57:38 station215 proxy-server ERROR with Object server 192.168.52.47:6010/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187)
Jun 26 11:57:39 station215 proxy-server ERROR with Object server 192.168.52.186:6040/device re: Trying to GET /v1/AUTH_mwliaotest/linux/67344: ConnectionTimeout (1.0s) (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187)
Jun 26 11:57:39 station215 proxy-server Object GET returning 503 for [] (txn: tx9c2b55167f244dd785487-0051ca66ad) (client_ip: 192.168.52.187)
Jun 26 11:57:39 station215 proxy-server 192.168.52.187 192.168.52.187 26/Jun/2013/03/57/39 GET /v1/AUTH_mwliaotest/linux/67344 HTTP/1.0 503 - Jakarta%20Commons-HttpClient/3.1 mwliaotest%2CAUTH_tkda034cde76e746da8ad31303856d2cbd - 118 - tx9c2b55167f244dd785487-0051ca66ad - 6.3996 -

As far as I know,Proxy-Server will look up the location of object server in the ring.gz file.But,the proxy-server route the GET request to all of 6 object-servers.I config the replacas_count with 3.Why did the proxy-server route request to the other 3 object-server?

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
Samuel Merritt
Solved:
2013-06-28
Last query:
2013-06-28
Last reply:
2013-06-27
Kun Huang (academicgareth) said : #1

Because your didn't get a response from obj server with 200 or 404.

Proxy should get enough 200 or enough 404 from obj server and convert it into 200 or 404 to user (you). All of your nodes are timeout, so proxy request all nodes then.

kalrey (249524324-q) said : #2

I do not really understand.Is it useful to connect the other object server ?There ara no replica stored in those object.Why did the proxy do this?I appreciate.

Best Samuel Merritt (torgomatic) said : #3

Well, here's an example for you. Let's say your primary nodes are d0, d1, and d2, with handoff nodes d3, d4, and d5.

On object PUT, the proxy will normally write 3 replicas to d0, d1, and d2, as you said. However, if d0 is full, then the object server for d0 will return a 507 response, and the proxy will use a handoff, so the object gets written to nodes d1, d2, d3.

Similarly, if a machine is down, then more handoffs will be used. Let's say d0 and d1 are full, and d2's machine has a broken network port. Now the object PUT will get written to d3, d4, and d5: all handoffs.

The proxy looks to some handoffs in order to give read-after-write consistency for new objects. Otherwise, if an object were written to only handoff nodes, then a subsequent GET wouldn't be able to find it.

The number of handoffs to look at is bounded, though; the proxy won't just look for everything in the cluster. The setting "request_node_count" in the proxy config controls how many handoffs will be asked, and the default is twice your replica count.

kalrey (249524324-q) said : #4

Thanks Samuel Merritt, that solved my question.