savanna 0.3 excuting pig script encounter error

Asked by gws

script:
    A = load '$INPUT' as (line);
    B = foreach A generate flatten(TOKENIZE(line)) as word;
   C = group B by word;
   D = foreach C generate group, COUNT(B);
   store D into '$OUTPUT';

there is no any exceptions ocurr in jobtacker,namenode,tasktracker or datanode logs. the only exceptions which i can find in /opt/oozie/logs/oozie.log is as follows:

2014-02-28 06:12:49,347 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time = 2014-02-28T06:11Z
2014-02-28 06:12:49,350 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:13:09,664 INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:13:09,687 INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:13:49,351 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:13:49,352 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time = 2014-02-28T06:12Z
2014-02-28 06:13:49,355 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time = 2014-02-28T06:12Z
2014-02-28 06:13:49,358 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:14:09,689 INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:14:09,712 INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:14:49,358 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:14:49,359 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time = 2014-02-28T06:13Z
2014-02-28 06:14:49,362 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time = 2014-02-28T06:13Z
2014-02-28 06:14:49,366 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:15:05,358 WARN LiteWorkflowAppService:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://v-ndhoo-v-nn-001:8020/user/hadoop/worldcount/4496286e-d5e0-42fe-8842-800bace75f54/lib] does not exist
2014-02-28 06:15:05,499 INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@:start:] Start action [0000004-140227105121167-oozie-hado-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-02-28 06:15:05,499 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@:start:] [***0000004-140227105121167-oozie-hado-W@:start:***]Action status=DONE
2014-02-28 06:15:05,500 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@:start:] [***0000004-140227105121167-oozie-hado-W@:start:***]Action updated in DB!
2014-02-28 06:15:05,602 INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] Start action [0000004-140227105121167-oozie-hado-W@job-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-02-28 06:15:06,029 WARN PigActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] credentials is null for the action
2014-02-28 06:15:06,475 WARN JobClient:746 - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2014-02-28 06:15:06,575 WARN JobClient:870 - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
2014-02-28 06:15:06,820 INFO PigActionExecutor:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] checking action, external ID [job_201402271128_0005] status [RUNNING]
2014-02-28 06:15:06,825 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] [***0000004-140227105121167-oozie-hado-W@job-node***]Action status=RUNNING
2014-02-28 06:15:06,826 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] [***0000004-140227105121167-oozie-hado-W@job-node***]Action updated in DB!
2014-02-28 06:15:09,712 INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:15:09,728 INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:15:19,450 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] callback for action [0000004-140227105121167-oozie-hado-W@job-node]
2014-02-28 06:15:19,531 INFO PigActionExecutor:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] action completed, external ID [job_201402271128_0005]

------------------------------------------------------------------------------------------------------------------------

2014-02-28 06:15:19,558 WARN PigActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
-------------------------------------------------------------------------------------------------------------------------

2014-02-28 06:15:19,659 INFO ActionEndXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] ERROR is considered as FAILED for SLA
2014-02-28 06:15:19,762 INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@fail] Start action [0000004-140227105121167-oozie-hado-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-02-28 06:15:19,767 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@fail] [***0000004-140227105121167-oozie-hado-W@fail***]Action status=DONE
2014-02-28 06:15:19,767 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@fail] [***0000004-140227105121167-oozie-hado-W@fail***]Action updated in DB!
2014-02-28 06:15:19,855 WARN CoordActionUpdateXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
2014-02-28 06:15:49,366 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]

Question information

Language:
English Edit question
Status:
Solved
For:
Sahara Edit question
Assignee:
No assignee Edit question
Solved by:
gws
Solved:
Last query:
Last reply:
Revision history for this message
gws (gzsgwsgwl) said :
#1

open debug, this is more detail log output:

2014-02-28 07:48:44,688 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8021 from hadoop got value #147
2014-02-28 07:48:44,688 DEBUG RPC:233 - Call: getJobStatus 1
2014-02-28 07:48:44,690 DEBUG Client:813 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8021 from hadoop sending #148
2014-02-28 07:48:44,692 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8021 from hadoop got value #148
2014-02-28 07:48:44,692 DEBUG Counters:178 - Creating group oozie.launcher with nothing
2014-02-28 07:48:44,692 DEBUG Counters:178 - Creating group org.apache.hadoop.mapred.FileInputFormat$Counter with bundle
2014-02-28 07:48:44,692 DEBUG Counters:178 - Creating group org.apache.hadoop.mapred.JobInProgress$Counter with bundle
2014-02-28 07:48:44,692 DEBUG Counters:178 - Creating group org.apache.hadoop.mapred.FileOutputFormat$Counter with bundle
2014-02-28 07:48:44,693 DEBUG Counters:178 - Creating group FileSystemCounters with nothing
2014-02-28 07:48:44,693 DEBUG Counters:178 - Creating group org.apache.hadoop.mapred.Task$Counter with bundle
2014-02-28 07:48:44,694 DEBUG RPC:233 - Call: getJobCounters 4
2014-02-28 07:48:44,703 DEBUG HadoopAccessorService:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Checking if filesystem hdfs is supported
2014-02-28 07:48:44,703 DEBUG UserGroupInformation:1212 - PriviledgedAction as:hadoop via hadoop from:org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
2014-02-28 07:48:44,704 DEBUG Client:244 - The ping interval is60000ms.
2014-02-28 07:48:44,704 DEBUG Client:287 - Use SIMPLE authentication for protocol ClientProtocol
2014-02-28 07:48:44,704 DEBUG Client:577 - Connecting to v-ndhoo-v-nn-001/10.12.1.4:8020
2014-02-28 07:48:44,705 DEBUG Client:813 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop sending #149
2014-02-28 07:48:44,705 DEBUG Client:786 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop: starting, having connections 2
2014-02-28 07:48:44,708 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop got value #149
2014-02-28 07:48:44,708 DEBUG RPC:233 - Call: getFileInfo 5
2014-02-28 07:48:44,708 DEBUG Client:813 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop sending #150
2014-02-28 07:48:44,710 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop got value #150
2014-02-28 07:48:44,710 DEBUG RPC:233 - Call: getBlockLocations 2
2014-02-28 07:48:44,710 DEBUG DFSClient:2285 - Connecting to /10.12.1.7:50010
2014-02-28 07:48:44,712 DEBUG FSInputChecker:1700 - DFSClient readChunk got seqno 0 offsetInBlock 0 lastPacketInBlock true packetLen 133
2014-02-28 07:48:44,713 WARN PigActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
2014-02-28 07:48:44,726 DEBUG HadoopAccessorService:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Checking if filesystem hdfs is supported
2014-02-28 07:48:44,726 DEBUG UserGroupInformation:1212 - PriviledgedAction as:hadoop via hadoop from:org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
2014-02-28 07:48:44,726 DEBUG Client:813 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop sending #151
2014-02-28 07:48:44,728 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop got value #151
2014-02-28 07:48:44,728 DEBUG RPC:233 - Call: getFileInfo 2
2014-02-28 07:48:44,739 DEBUG ActionCheckXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] ENDED ActionCheckXCommand for wf actionId=0000000-140228074649480-oozie-hado-W@job-node, jobId=0000000-140228074649480-oozie-hado-W
2014-02-28 07:48:44,739 DEBUG ActionCheckXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Queuing [1] commands with delay [0]ms
2014-02-28 07:48:44,740 DEBUG ActionCheckXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Released lock for [0000000-140228074649480-oozie-hado-W] in [action.check]
2014-02-28 07:48:44,740 DEBUG ActionEndXCommand:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [0000000-140228074649480-oozie-hado-W] in [action.end]
2014-02-28 07:48:44,746 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Execute command [action.end] key [0000000-140228074649480-oozie-hado-W]
2014-02-28 07:48:44,747 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] STARTED ActionEndXCommand for action 0000000-140228074649480-oozie-hado-W@job-node
2014-02-28 07:48:44,750 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] End, name [job-node] type [pig] status[DONE] external status [FAILED/KILLED] signal value [null]
2014-02-28 07:48:44,756 DEBUG HadoopAccessorService:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Checking if filesystem hdfs is supported
2014-02-28 07:48:44,757 DEBUG UserGroupInformation:1212 - PriviledgedAction as:hadoop via hadoop from:org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
2014-02-28 07:48:44,763 DEBUG HadoopAccessorService:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Checking if filesystem hdfs is supported
2014-02-28 07:48:44,763 DEBUG UserGroupInformation:1212 - PriviledgedAction as:hadoop via hadoop from:org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
2014-02-28 07:48:44,763 DEBUG Client:813 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop sending #152
2014-02-28 07:48:44,764 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop got value #152
2014-02-28 07:48:44,765 DEBUG RPC:233 - Call: getFileInfo 2
2014-02-28 07:48:44,765 DEBUG Client:813 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop sending #153
2014-02-28 07:48:44,767 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop got value #153
2014-02-28 07:48:44,767 DEBUG RPC:233 - Call: delete 2
2014-02-28 07:48:44,775 INFO ActionEndXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] ERROR is considered as FAILED for SLA
2014-02-28 07:48:44,776 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Queuing commands for action=0000000-140228074649480-oozie-hado-W@job-node, status=ERROR, Set pending=true
2014-02-28 07:48:44,803 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] ENDED ActionEndXCommand for action 0000000-140228074649480-oozie-hado-W@job-node
2014-02-28 07:48:44,810 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Queuing [1] commands with delay [0]ms
2014-02-28 07:48:44,811 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Released lock for [0000000-140228074649480-oozie-hado-W] in [action.end]
2014-02-28 07:48:44,811 DEBUG SignalXCommand:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [0000000-140228074649480-oozie-hado-W] in [signal]
2014-02-28 07:48:44,817 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Execute command [signal] key [0000000-140228074649480-oozie-hado-W]
2014-02-28 07:48:44,817 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] STARTED SignalCommand for jobid=0000000-140228074649480-oozie-hado-W, actionId=0000000-140228074649480-oozie-hado-W@job-node
2014-02-28 07:48:44,818 DEBUG LiteWorkflowInstance:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Signaling job execution path [/] signal value [ERROR]
2014-02-28 07:48:44,819 DEBUG LiteWorkflowInstance:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Exiting node [job-node] with transition[/#fail]
2014-02-28 07:48:44,819 DEBUG LiteWorkflowInstance:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Signaling job execution path [/] signal value [::synch::]
2014-02-28 07:48:44,819 DEBUG LiteWorkflowStoreService:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Creating action for node [fail]
2014-02-28 07:48:44,821 DEBUG LiteWorkflowStoreService:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Setting action for cred: 'null', name: 'fail'
2014-02-28 07:48:44,822 DEBUG LiteWorkflowStoreService:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Setting action for userRetryMax: '0', userRetryInterval: '10', name: 'fail'
2014-02-28 07:48:44,829 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] SignalXCommand: Name: fail, Id: 0000000-140228074649480-oozie-hado-W@fail, Authcode:null
2014-02-28 07:48:44,846 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Updated the workflow status to 0000000-140228074649480-oozie-hado-W status =RUNNING
2014-02-28 07:48:44,847 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] ENDED SignalCommand for jobid=0000000-140228074649480-oozie-hado-W, actionId=0000000-140228074649480-oozie-hado-W@job-node
2014-02-28 07:48:44,847 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Queuing [2] commands with delay [0]ms
2014-02-28 07:48:44,847 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Released lock for [0000000-140228074649480-oozie-hado-W] in [signal]
2014-02-28 07:48:44,848 DEBUG NotificationXCommand:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Execute command [action.notification] key [null]
2014-02-28 07:48:44,853 DEBUG ActionStartXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@job-node] Acquired lock for [0000000-140228074649480-oozie-hado-W] in [action.start]
2014-02-28 07:48:44,870 DEBUG ActionStartXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Execute command [action.start] key [0000000-140228074649480-oozie-hado-W]
2014-02-28 07:48:44,871 DEBUG ActionStartXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] STARTED ActionStartXCommand for wf actionId=0000000-140228074649480-oozie-hado-W@fail
2014-02-28 07:48:44,879 INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Start action [0000000-140228074649480-oozie-hado-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-02-28 07:48:44,879 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] [***0000000-140228074649480-oozie-hado-W@fail***]Action status=DONE
2014-02-28 07:48:44,880 WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] [***0000000-140228074649480-oozie-hado-W@fail***]Action updated in DB!
2014-02-28 07:48:44,896 DEBUG ActionStartXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] ENDED ActionStartXCommand for wf actionId=0000000-140228074649480-oozie-hado-W@fail, jobId=0000000-140228074649480-oozie-hado-W
2014-02-28 07:48:44,897 DEBUG ActionStartXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Queuing [1] commands with delay [0]ms
2014-02-28 07:48:44,897 DEBUG ActionStartXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Released lock for [0000000-140228074649480-oozie-hado-W] in [action.start]
2014-02-28 07:48:44,898 DEBUG ActionEndXCommand:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [0000000-140228074649480-oozie-hado-W] in [action.end]
2014-02-28 07:48:44,908 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Execute command [action.end] key [0000000-140228074649480-oozie-hado-W]
2014-02-28 07:48:44,908 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] STARTED ActionEndXCommand for action 0000000-140228074649480-oozie-hado-W@fail
2014-02-28 07:48:44,914 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] End, name [fail] type [:KILL:] status[DONE] external status [OK] signal value [null]
2014-02-28 07:48:44,921 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Queuing commands for action=0000000-140228074649480-oozie-hado-W@fail, status=OK, Set pending=true
2014-02-28 07:48:44,949 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] ENDED ActionEndXCommand for action 0000000-140228074649480-oozie-hado-W@fail
2014-02-28 07:48:44,955 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Queuing [1] commands with delay [0]ms
2014-02-28 07:48:44,955 DEBUG ActionEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Released lock for [0000000-140228074649480-oozie-hado-W] in [action.end]
2014-02-28 07:48:44,955 DEBUG SignalXCommand:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [0000000-140228074649480-oozie-hado-W] in [signal]
2014-02-28 07:48:44,970 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Execute command [signal] key [0000000-140228074649480-oozie-hado-W]
2014-02-28 07:48:44,971 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] STARTED SignalCommand for jobid=0000000-140228074649480-oozie-hado-W, actionId=0000000-140228074649480-oozie-hado-W@fail
2014-02-28 07:48:44,973 DEBUG LiteWorkflowInstance:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Signaling job execution path [/] signal value [OK]
2014-02-28 07:48:44,974 DEBUG LiteWorkflowInstance:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Completing job, kill node [fail]
2014-02-28 07:48:45,037 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Try to resolve KillNode message for jobid [0000000-140228074649480-oozie-hado-W], actionId [0000000-140228074649480-oozie-hado-W@fail], before resolve [Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}]], after resolve [Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]]
2014-02-28 07:48:45,073 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Updated the workflow status to 0000000-140228074649480-oozie-hado-W status =KILLED
2014-02-28 07:48:45,125 WARN CoordActionUpdateXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
2014-02-28 07:48:45,129 DEBUG WfEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[-] Execute command [wf_end] key [0000000-140228074649480-oozie-hado-W]
2014-02-28 07:48:45,130 DEBUG WfEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[-] STARTED WFEndXCommand 0000000-140228074649480-oozie-hado-W
2014-02-28 07:48:45,162 DEBUG HadoopAccessorService:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[-] Checking if filesystem hdfs is supported
2014-02-28 07:48:45,165 DEBUG UserGroupInformation:1212 - PriviledgedAction as:hadoop via hadoop from:org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
2014-02-28 07:48:45,166 DEBUG WfEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[-] WF tmp dir :hdfs://v-ndhoo-v-nn-001:8020/user/hadoop/oozie-hado/0000000-140228074649480-oozie-hado-W
2014-02-28 07:48:45,166 DEBUG Client:813 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop sending #154
2014-02-28 07:48:45,167 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop got value #154
2014-02-28 07:48:45,172 DEBUG RPC:233 - Call: getFileInfo 6
2014-02-28 07:48:45,173 DEBUG Client:813 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop sending #155
2014-02-28 07:48:45,175 DEBUG Client:848 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop got value #155
2014-02-28 07:48:45,180 DEBUG RPC:233 - Call: delete 7
2014-02-28 07:48:45,181 DEBUG WfEndXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[-] ENDED WFEndXCommand 0000000-140228074649480-oozie-hado-W
2014-02-28 07:48:45,181 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] ENDED SignalCommand for jobid=0000000-140228074649480-oozie-hado-W, actionId=0000000-140228074649480-oozie-hado-W@fail
2014-02-28 07:48:45,181 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Queuing [2] commands with delay [0]ms
2014-02-28 07:48:45,181 DEBUG SignalXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Released lock for [0000000-140228074649480-oozie-hado-W] in [signal]
2014-02-28 07:48:45,182 DEBUG NotificationXCommand:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Execute command [action.notification] key [null]
2014-02-28 07:48:45,182 DEBUG NotificationXCommand:545 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000000-140228074649480-oozie-hado-W] ACTION[0000000-140228074649480-oozie-hado-W@fail] Execute command [job.notification] key [null]
2014-02-28 07:48:54,690 DEBUG Client:920 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8021 from hadoop: closed
2014-02-28 07:48:54,691 DEBUG Client:796 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8021 from hadoop: stopped, remaining connections 1
2014-02-28 07:48:55,174 DEBUG Client:920 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop: closed
2014-02-28 07:48:55,174 DEBUG Client:796 - IPC Client (47) connection to v-ndhoo-v-nn-001/10.12.1.4:8020 from hadoop: stopped, remaining connections 0
2014-02-28 07:49:03,501 DEBUG DagEngine:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Oozie DagEngine is using XCommands.
2014-02-28 07:49:03,503 DEBUG JobXCommand:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Execute command [job.info] key [0000000-140228074649480-oozie-hado-W]
2014-02-28 07:49:04,562 DEBUG ActionCheckerService$ActionCheckRunnable:545 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] QUEUING [] for potential checking
2014-02-28 07:49:04,606 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 07:49:04,606 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time = 2014-02-28T07:48Z
2014-02-28 07:49:04,609 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time = 2014-02-28T07:48Z
2014-02-28 07:49:04,613 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 07:49:04,779 INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 07:49:04,792 INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]

Revision history for this message
Andrew Lazarev (alazarev) said :
#2

I see the following line in your logs

2014-02-28 06:15:05,358 WARN LiteWorkflowAppService:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://v-ndhoo-v-nn-001:8020/user/hadoop/worldcount/4496286e-d5e0-42fe-8842-800bace75f54/lib] does not exist

'lib' folder usually contains all libraries attached to the job. Most probably you forgot to add .jar file with classes you use in pig script (second tab on UI during Job creation).

Revision history for this message
Sergey Lukjanov (slukjanov) said :
#3
Revision history for this message
gws (gzsgwsgwl) said :
#4

thanks,maybe there is some wrong configuration indeed,but no other jar need to be included.i will go on to find the reason for this problem,and then i will show the actual reason.

Revision history for this message
gws (gzsgwsgwl) said :
#5

find the reason which occurred mainly because of my wrong swift configuration。
after i apply savanna's 1240144 and 1240511 two bugs, then change swift://demo/input to swift://demo.savanna/input, and every things start to work well.

i am very sorry to bother you all just because my carefulless.
thank all of you !

Revision history for this message
Sergey Lukjanov (slukjanov) said :
#6

Could you please repost your answers to ask.openstack.org?

Revision history for this message
gws (gzsgwsgwl) said :
#7

hi,i have posted the answer to ask.openstack.org!