unable to copy volume from one cluster to another via euca-create-volume --snapshot -z

Asked by Cladmin

Issue originally raised here:
http://ubuntuforums.org/showthread.php?p=10623367&posted=1#post10623367

I'm using Ubuntu 10.4.2 updated to the latest binaries, with Eucalyptus v1.6.

I have a cloud consisting of 5 servers in all: 1 clc/walrus, 1 cc/sc + 1 nc (cluster1-01), 1
cc/sc + 1 nc (cluster1-02).

ivacau@host0002:~$ sudo euca_conf --list-scs
registered storage controllers:
cluster1-02 192.168.121.5
cluster1-01 192.168.121.1
ivacau@host0002:~$ sudo euca_conf --list-clusters
registered clusters:
cluster1-02 192.168.121.5
cluster1-01 192.168.121.1
ivacau@host0002:~$ sudo euca_conf --list-nodes
registered nodes:
192.168.127.1 cluster1-01
192.168.128.1 cluster1-02

I am able to start instances in both clusters (availability zones) no problem. I'm also able to create volumes, associate volume to instance, create snapshot from a volume and subsequently create a volume from the snapshot, all within one of the two clusters.

ivacau@host0002:~$ euca-describe-volumes
VOLUME vol-5F880657 1 snap-5F170646 cluster1-02 available 2011-03-25T08:10:26.986Z
VOLUME vol-5F970659 1 cluster1-02 available 2011-03-25T07:45:44.709Z
VOLUME vol-5F40064C 1 snap-58DE0616 cluster1-01 available 2011-03-25T04:49:04.251Z
VOLUME vol-5F0B064A 1 cluster1-01 available 2011-03-25T03:00:16.941Z
VOLUME vol-6000065F 1 cluster1-01 available 2011-03-25T04:11:33.592Z

However when I attempt to create a volume (say in cluster1-02) from a snapshot that originally resided in the other cluster (say cluster1-01), I get the following error:

ivacau@host0002:~$ euca-create-volume --snapshot snap-58DE0616 -z cluster1-02
Volume: Error while communicating with Storage Controller: CreateStorageVolume:Internal Error.

The same error is encountered if I invert the source and destination availability zone.

On the clc/walrus server, I can see the following exceptions in /var/log/eucalyptus/cloud-error.log:

23:56:06 [NioResponseHandler:Hashed wheel timer #1] ERROR Caught exception in asynchronous response handler.
org.jboss.netty.handler.timeout.ReadTimeoutExcepti on
at org.jboss.netty.handler.timeout.ReadTimeoutHandler .<clinit>(ReadTimeoutHandler.java:59)
at com.eucalyptus.ws.util.ChannelUtil.addPipelineMoni tors(ChannelUtil.java:108)
at com.eucalyptus.cluster.handlers.AbstractClusterMes sageDispatcher.getPipeline(AbstractClusterMessageD ispatcher.java:109)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioB ootstrap.java:177)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioB ootstrap.java:163)
at com.eucalyptus.cluster.handlers.AbstractClusterMes sageDispatcher.write(AbstractClusterMessageDispatc her.java:129)
at com.eucalyptus.cluster.handlers.ClusterCertificate Handler.trigger(ClusterCertificateHandler.java:32)
at com.eucalyptus.cluster.handlers.ClusterCertificate Handler.fireEvent(ClusterCertificateHandler.java:5 7)
at com.eucalyptus.event.ReentrantListenerRegistry.fir eEvent(ReentrantListenerRegistry.java:80)
at com.eucalyptus.event.ReentrantListenerRegistry.fir eEvent(ReentrantListenerRegistry.java:65)
at com.eucalyptus.event.ListenerRegistry.fireEvent(Li stenerRegistry.java:69)
at com.eucalyptus.cluster.ClusterBootstrapper.load(Cl usterBootstrapper.java:140)
at com.eucalyptus.bootstrap.SystemBootstrapper.load(S ystemBootstrapper.java:175)
23:56:06 [RemoteDispatcher:Volume.10] ERROR com.eucalyptus.util.EucalyptusClusterException: Exception in NIO request.
com.eucalyptus.util.EucalyptusClusterException: Exception in NIO request.
at com.eucalyptus.ws.handlers.NioResponseHandler.getR esponse(NioResponseHandler.java:115)
at com.eucalyptus.ws.client.NioClient.send(NioClient. java:113)
at com.eucalyptus.ws.client.RemoteDispatcher.send(Rem oteDispatcher.java:110)
at com.eucalyptus.ws.client.ServiceDispatcher.send(Se rviceDispatcher.java:80)
at edu.ucsb.eucalyptus.cloud.ws.VolumeManager.CreateV olume(VolumeManager.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mule.model.resolvers.AbstractEntryPointResolve r.invokeMethod(AbstractEntryPointResolver.java:147 )
at org.mule.model.resolvers.ReflectionEntryPointResol ver.invoke(ReflectionEntryPointResolver.java:178)
at org.mule.model.resolvers.DefaultEntryPointResolver Set.invoke(DefaultEntryPointResolverSet.java:50)
at org.mule.component.DefaultLifecycleAdapter.interce pt(DefaultLifecycleAdapter.java:202)
at org.mule.component.AbstractJavaComponent.invokeCom ponentInstance(AbstractJavaComponent.java:82)
at org.mule.component.AbstractJavaComponent.doOnCall( AbstractJavaComponent.java:73)
at org.mule.component.AbstractComponent.onCall(Abstra ctComponent.java:87)
at org.mule.model.seda.SedaService$ComponentStageWork er.run(SedaService.java:533)
at org.mule.work.WorkerContext.run(WorkerContext.java :310)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1061)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor$Worker.run(ThreadPoolExecutor.java :575)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.jboss.netty.handler.timeout.ReadTimeoutExcepti on
at org.jboss.netty.handler.timeout.ReadTimeoutHandler .<clinit>(ReadTimeoutHandler.java:59)
at com.eucalyptus.ws.util.ChannelUtil.addPipelineMoni tors(ChannelUtil.java:108)
at com.eucalyptus.cluster.handlers.AbstractClusterMes sageDispatcher.getPipeline(AbstractClusterMessageD ispatcher.java:109)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioB ootstrap.java:177)
at com.eucalyptus.ws.client.NioBootstrap.connect(NioB ootstrap.java:163)
at com.eucalyptus.cluster.handlers.AbstractClusterMes sageDispatcher.write(AbstractClusterMessageDispatc her.java:129)
at com.eucalyptus.cluster.handlers.ClusterCertificate Handler.trigger(ClusterCertificateHandler.java:32)
at com.eucalyptus.cluster.handlers.ClusterCertificate Handler.fireEvent(ClusterCertificateHandler.java:5 7)
at com.eucalyptus.event.ReentrantListenerRegistry.fir eEvent(ReentrantListenerRegistry.java:80)
at com.eucalyptus.event.ReentrantListenerRegistry.fir eEvent(ReentrantListenerRegistry.java:65)
at com.eucalyptus.event.ListenerRegistry.fireEvent(Li stenerRegistry.java:69)
at com.eucalyptus.cluster.ClusterBootstrapper.load(Cl usterBootstrapper.java:140)
at com.eucalyptus.bootstrap.SystemBootstrapper.load(S ystemBootstrapper.java:175)
23:56:06 [DefaultServiceExceptionStrategy:Volume.10] ERROR
************************************************** ******************************
Message : Component that caused exception is: Volume. Message payload is of type: CreateVolumeType
Type : org.mule.api.service.ServiceException
Code : MULE_ERROR--2
Payload : <?xml version="1.0" encoding="UTF-8"?>
<euca:CreateVolume xmlns:euca="http://msgs.eucalyptus.ucsb.edu">
<euca:BlockVolumeMessage>
<euca:correlationId>c5b404ad-f63c-4038-8c21-5fe0f282d81f</euca:correlationId>
<euca:userId>admin</euca:userId>
<euca:effectiveUserId>eucalyptus</euca:effectiveUserId>
</euca:BlockVolumeMessage>
<euca:snapshotId>snap-58DE0616</euca:snapshotId>
<euca:availabilityZone>cluster1-02</euca:availabilityZone>
</euca:CreateVolume>
JavaDoc : http://mule.mulesource.org/docs/apid...Exception.html
************************************************** ******************************
Exception stack is:
1. Error while communicating with Storage Controller: CreateStorageVolume:Internal Error. (com.eucalyptus.util.EucalyptusCloudException)
edu.ucsb.eucalyptus.cloud.ws.VolumeManager:182 (null)
2. Component that caused exception is: Volume. Message payload is of type: CreateVolumeType (org.mule.api.service.ServiceException)
org.mule.component.DefaultLifecycleAdapter:214 (http://mule.mulesource.org/docs/apid...Exception.html)
************************************************** ******************************
Root Exception stack trace:
com.eucalyptus.util.EucalyptusCloudException: Error while communicating with Storage Controller: CreateStorageVolume:Internal Error.
at edu.ucsb.eucalyptus.cloud.ws.VolumeManager.CreateV olume(VolumeManager.java:182)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mule.model.resolvers.AbstractEntryPointResolve r.invokeMethod(AbstractEntryPointResolver.java:147 )
at org.mule.model.resolvers.ReflectionEntryPointResol ver.invoke(ReflectionEntryPointResolver.java:178)
at org.mule.model.resolvers.DefaultEntryPointResolver Set.invoke(DefaultEntryPointResolverSet.java:50)
at org.mule.component.DefaultLifecycleAdapter.interce pt(DefaultLifecycleAdapter.java:202)
at org.mule.component.AbstractJavaComponent.invokeCom ponentInstance(AbstractJavaComponent.java:82)
at org.mule.component.AbstractJavaComponent.doOnCall( AbstractJavaComponent.java:73)
at org.mule.component.AbstractComponent.onCall(Abstra ctComponent.java:87)
at org.mule.model.seda.SedaService$ComponentStageWork er.run(SedaService.java:533)
at org.mule.work.WorkerContext.run(WorkerContext.java :310)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1061)
at edu.emory.mathcs.backport.java.util.concurrent.Thr eadPoolExecutor$Worker.run(ThreadPoolExecutor.java :575)
at java.lang.Thread.run(Thread.java:636)

************************************************** ******************************

23:56:06 [DefaultServiceExceptionStrategy:Volume.10] ERROR Message being processed is: org.mule.transport.DefaultMessageAdapter/org.mule.transport.DefaultMessageAdapter@105a9{id= acd6fb00-593a-11e0-80e4-e3ba4516fe20, payload=edu.ucsb.eucalyptus.msgs.CreateVolumeType, correlationId=acd6fb00-593a-11e0-80e4-e3ba4516fe20, correlationGroup=-1, correlationSeq=-1, encoding=UTF-8, exceptionPayload=org.mule.message.DefaultException Payload@11aaec3}

on the example given, the 'source' storage controller for cluster1-01 (i.e. containing the snapshot) has the no relevant exceptions in /var/log/eucalyptus/cloud-error.log at the time the euca-create-volume command is given.

The 'destination' storage controller for cluster1-02 (i.e. where the volume is being attempted to be stored) has the following exceptions in /var/log/eucalyptus/cloud-error.log:

23:55:38 [DefaultServiceExceptionStrategy:New I/O server worker #1-29] ERROR
************************************************** ******************************
Message : Component that caused exception is: StorageInternal. Message payload is of type: CreateStorageVolumeType
Type : org.mule.api.service.ServiceException
Code : MULE_ERROR--2
Payload : <?xml version="1.0" encoding="UTF-8"?>
<euca:CreateStorageVolume xmlns:euca="http://msgs.eucalyptus.ucsb.edu">
<euca:StorageRequest>
<euca:correlationId>7f31b339-10eb-4e1e-8d93-286ade1bdb0f</euca:correlationId>
<euca:userId>admin</euca:userId>
<euca:effectiveUserId>eucalyptus</euca:effectiveUserId>
</euca:StorageRequest>
<euca:volumeId>vol-5F8E065B</euca:volumeId>
<euca:snapshotId>snap-58DE0616</euca:snapshotId>
</euca:CreateStorageVolume>
JavaDoc : http://mule.mulesource.org/docs/apid...Exception.html
************************************************** ******************************
Exception stack is:
1. The specified entity was not found (edu.ucsb.eucalyptus.cloud.NoSuchEntityException)
edu.ucsb.eucalyptus.cloud.ws.BlockStorage:544 (null)
2. Component that caused exception is: StorageInternal. Message payload is of type: CreateStorageVolumeType (org.mule.api.service.ServiceException)
org.mule.component.DefaultLifecycleAdapter:214 (http://mule.mulesource.org/docs/apid...Exception.html)
************************************************** ******************************
Root Exception stack trace:
edu.ucsb.eucalyptus.cloud.NoSuchEntityException: The specified entity was not found
at edu.ucsb.eucalyptus.cloud.ws.BlockStorage.CreateSt orageVolume(BlockStorage.java:544)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.mule.model.resolvers.AbstractEntryPointResolve r.invokeMethod(AbstractEntryPointResolver.java:147 )
at org.mule.model.resolvers.ReflectionEntryPointResol ver.invoke(ReflectionEntryPointResolver.java:178)
at org.mule.model.resolvers.DefaultEntryPointResolver Set.invoke(DefaultEntryPointResolverSet.java:50)
at org.mule.component.DefaultLifecycleAdapter.interce pt(DefaultLifecycleAdapter.java:202)
at org.mule.component.AbstractJavaComponent.invokeCom ponentInstance(AbstractJavaComponent.java:82)
at org.mule.component.AbstractJavaComponent.doOnCall( AbstractJavaComponent.java:73)
at org.mule.component.AbstractComponent.onCall(Abstra ctComponent.java:87)
at org.mule.model.seda.SedaService.doSend(SedaService .java:234)
at org.mule.service.AbstractService.sendEvent(Abstrac tService.java:510)
at org.mule.DefaultMuleSession.sendEvent(DefaultMuleS ession.java:351)
at org.mule.routing.inbound.DefaultInboundRouterColle ction.send(DefaultInboundRouterCollection.java:196 )
at org.mule.routing.inbound.DefaultInboundRouterColle ction.route(DefaultInboundRouterCollection.java:16 4)
at org.mule.transport.AbstractMessageReceiver$Default InternalMessageListener.onMessage(AbstractMessageR eceiver.java:604)
at org.mule.transport.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:346)
at org.mule.transport.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:269)
at org.mule.transport.AbstractMessageReceiver.routeMe ssage(AbstractMessageReceiver.java:262)
at com.eucalyptus.ws.handlers.ServiceSinkHandler.hand leUpstream(ServiceSinkHandler.java:203)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.handlers.MessageStackHandler.han dleUpstream(MessageStackHandler.java:133)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at com.eucalyptus.ws.server.NioServerHandler.messageR eceived(NioServerHandler.java:119)
at org.jboss.netty.channel.SimpleChannelUpstreamHandl er.handleUpstream(SimpleChannelUpstreamHandler.jav a:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.handler.stream.ChunkedWriteHandler .handleUpstream(ChunkedWriteHandler.java:114)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:385)
at org.jboss.netty.handler.codec.replay.ReplayingDeco der.unfoldAndfireMessageReceived(ReplayingDecoder. java:459)
at org.jboss.netty.handler.codec.replay.ReplayingDeco der.callDecode(ReplayingDecoder.java:443)
at org.jboss.netty.handler.codec.replay.ReplayingDeco der.messageReceived(ReplayingDecoder.java:381)
at org.jboss.netty.channel.SimpleChannelUpstreamHandl er.handleUpstream(SimpleChannelUpstreamHandler.jav a:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.handler.timeout.ReadTimeoutHandler .messageReceived(ReadTimeoutHandler.java:146)
at org.jboss.netty.channel.SimpleChannelUpstreamHandl er.handleUpstream(SimpleChannelUpstreamHandler.jav a:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.handler.timeout.IdleStateHandler.m essageReceived(IdleStateHandler.java:214)
at org.jboss.netty.channel.SimpleChannelUpstreamHandl er.handleUpstream(SimpleChannelUpstreamHandler.jav a:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline$Def aultChannelHandlerContext.sendUpstream(DefaultChan nelPipeline.java:803)
at org.jboss.netty.channel.SimpleChannelHandler.messa geReceived(SimpleChannelHandler.java:159)
at com.eucalyptus.ws.handlers.ChannelStateMonitor.mes sageReceived(ChannelStateMonitor.java:61)
at org.jboss.netty.channel.SimpleChannelHandler.handl eUpstream(SimpleChannelHandler.java:105)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:567)
at org.jboss.netty.channel.DefaultChannelPipeline.sen dUpstream(DefaultChannelPipeline.java:562)
at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:342)
at org.jboss.netty.channel.Channels.fireMessageReceiv ed(Channels.java:329)
at org.jboss.netty.channel.socket.nio.NioWorker.read( NioWorker.java:330)
at org.jboss.netty.channel.socket.nio.NioWorker.proce ssSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(N ioWorker.java:203)
at org.jboss.netty.util.ThreadRenamingRunnable.run(Th readRenamingRunnable.java:110)
at org.jboss.netty.util.internal.IoWorkerRunnable.run (IoWorkerRunnable.java:53)
at org.jboss.netty.handler.execution.MemoryAwareThrea dPoolExecutor$MemoryAwareRunnable.run(MemoryAwareT hreadPoolExecutor.java:502)
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

************************************************** ******************************

One strange thing I've noticed: in the cloud-output.log of the host on which the destination storage controller is located, a huge quantity of lines similar to the following appear, at about the time that the snapshots are generated. Does this indicate a crash of some kind?

10:01:33 DEBUG content | Wire.java:84 >> "[0x1f][0x8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0xed][0xc1][0x1][0x1][0x0][0x0][0x0][0x82] [0xff][0xaf]nH@[0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0]|[0x18]a[0x85][0xf8]J[0x0][0x90][0x1][0x0]"

Also, following Graziano's description of the process, I've located the two copies of the snapshot, which are created sequentially first on the sc (/var/lib/eucalyptus/volumes) and then on walrus (/var/lib/eucalyptus/bukkits). I was surprised to note that the size of the snapshot file is different on the sc and walrus, using both 'ls -l' and 'du'. I would have expected them to be identical (unless only a compressed version is stored on walrus). Is it possible that the snapshot copy to walrus is incomplete for some reason, even if euca-describe-snapshots reports 100% completion?

Can anyone help to resolve the problem?

Question information

Language:
English Edit question
Status:
Open
For:
Eucalyptus Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Cladmin (cladmin) said :
#1

I've also seen the below lines in cloud-output.log of the destination storage controller at the end of the hex code dumped there:

10:02:52 DEBUG content | Wire.java:70 >> "[\r][\n]"
10:02:52 DEBUG content | Wire.java:84 >> "0"
10:02:52 DEBUG content | Wire.java:70 >> "[\r][\n]"
10:02:52 DEBUG content | Wire.java:70 >> "[\r][\n]"
10:02:52 DEBUG header | Wire.java:70 << "HTTP/1.1 200 OK[\r][\n]"
10:02:52 DEBUG header | Wire.java:70 << "HTTP/1.1 200 OK[\r][\n]"
10:02:52 DEBUG header | Wire.java:70 << "Content-Length: 247[\r][\n]"
10:02:52 DEBUG header | Wire.java:70 << "Content-Type: application/xml[\r][\n]"
10:02:52 DEBUG header | Wire.java:70 << "[\r][\n]"
10:02:52 DEBUG content | Wire.java:84 << "<StoreSnapshotResponse xmlns="http://s3.amazonaws.com/doc/2006-03-01/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><ETag>79589333dce395892af0e319ab22835a</ETag><LastModified>2011-03-31T23:02:52.000Z</LastModified></StoreSnapshotResponse>"

This also indicates a problem with snapshot generation, but on the sc not the walrus. However it could be the sc failing to complete the copy of the snapshot to walrus, if the sc is in control here.

Revision history for this message
Cladmin (cladmin) said :
#2

Can I correct slightly the information just provided. Both the hex dump and the xml data found in the cloud-output.log file are generated in the source sc, not the destination sc - i.e. where the snapshot is first generated. Furthermore, this data gets generated during snapshot establishment, and at the stage when the snapshot is being copied to walrus, i.e. after the snapshot file has been created on the sc. So at this stage, I would not even have started attempting to create a new volume on a 2nd cluster from the snapshot in the 1st. It would stand to reason that if the snapshot does not exist correctly in walrus, then it's impossible to create a clone on the other cluster as I'm trying to do.

Can you help with this problem?

Provide an answer of your own, or ask Cladmin for more information if necessary.

To post a message you must log in.