স্থানীয় মোডে শূকর টিউটোরিয়াল আউটআফমিউরিওর প্রদান করে


1

আমি শুয়োরের সাথে নতুন এবং আমি অ্যাপাচি ওয়েবসাইটে পিগ টিউটোরিয়াল থেকে শিখতে চেষ্টা করি। আমি হ্যাডোপ ০.০.১ এবং পিগভার্সিয়ান ০.০১.১ নিয়ে কাজ করছি। টিউটোরিয়ালে, দু'টি উদাহরণের (যা স্ক্রিপ্ট 1-লোকাল.পিগ এবং স্ক্রিপ্ট 2-লোকাল.পিগ নামে পরিচিত) পিগ চেষ্টা করার প্রস্তাব দেওয়া হয়েছে। তবে যখন আমি নিম্নলিখিত সিএলআই দিয়ে প্রথম উদাহরণটি চালানোর চেষ্টা করি:

pig -x local script1-local.pig

আমি নিম্নলিখিত ত্রুটিটি পেয়েছি: (এটি জাভা হ্যাপ আকারের সাথে আচরণ করবে বলে মনে হচ্ছে ...)

2013-07-16 15:26:59,033 [main] INFO  org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
2013-07-16 15:26:59,033 [main] INFO  org.apache.pig.Main - Logging error messages to: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:26:59,246 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2013-07-16 15:26:59,332 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2013-07-16 15:27:00,287 [main] WARN  org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 3 time(s).
2013-07-16 15:27:00,287 [main] WARN  org.apache.pig.PigServer - Encountered Warning USING_OVERLOADED_FUNCTION 3 time(s).
2013-07-16 15:27:00,300 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,ORDER_BY,DISTINCT,FILTER
2013-07-16 15:27:00,446 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-07-16 15:27:00,469 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2013-07-16 15:27:00,492 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 5
2013-07-16 15:27:00,493 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 5
2013-07-16 15:27:00,518 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-07-16 15:27:00,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-07-16 15:27:00,536 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2013-07-16 15:27:00,537 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=208348
2013-07-16 15:27:00,537 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2013-07-16 15:27:00,557 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported or needed in local mode. Setting key [pig.schematuple.local.dir] with code temp directory: /tmp/1373988420563-0
2013-07-16 15:27:00,564 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting identity combiner class.
2013-07-16 15:27:00,637 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-07-16 15:27:00,648 [JobControl] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-07-16 15:27:00,652 [JobControl] WARN  org.apache.hadoop.mapred.JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
****file:/root/pigtmp/excite-small.log
2013-07-16 15:27:00,681 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-07-16 15:27:00,681 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2013-07-16 15:27:00,688 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2013-07-16 15:27:00,922 [Thread-3] INFO  org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2013-07-16 15:27:00,925 [Thread-3] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b4b0967
2013-07-16 15:27:00,936 [Thread-3] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/root/pigtmp/excite-small.log:0+208348
2013-07-16 15:27:00,942 [Thread-3] INFO  org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
2013-07-16 15:27:01,059 [Thread-3] WARN  org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:674)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases clean1,clean2,houred,ngramed1,raw
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: raw[28,6],clean1[31,9],clean2[-1,-1],houred[39,9],ngramed1[42,11] C:  R: 
2013-07-16 15:27:01,145 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-07-16 15:27:01,148 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2013-07-16 15:27:01,148 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local_0001 has failed! Stop running all dependent jobs
2013-07-16 15:27:01,149 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-07-16 15:27:01,149 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-07-16 15:27:01,149 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
2013-07-16 15:27:01,150 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
1.0.1   0.11.1  root    2013-07-16 15:27:00 2013-07-16 15:27:01 GROUP_BY,ORDER_BY,DISTINCT,FILTER

Failed!

Failed Jobs:
JobId   Alias   Feature Message Outputs
job_local_0001  clean1,clean2,houred,ngramed1,raw   DISTINCT    Message: Job failed! Error - NA 

Input(s):
Failed to read data from "file:///root/pigtmp/excite-small.log"

Output(s):

Job DAG:
job_local_0001  ->  null,
null    ->  null,
null    ->  null,
null    ->  null,
null


2013-07-16 15:27:01,150 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2013-07-16 15:27:01,152 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,153 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,154 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,154 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,155 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log

কেউ কি একই সমস্যা সম্মুখীন হিসাবে?

ধন্যবাদ.

উত্তর:


1

এই উত্তরে যেমন উল্লেখ করা হয়েছে , আপনি সর্বাধিক স্মৃতি বরাদ্দ স্থান বাড়ানোর চেষ্টা করতে পারেন:

export HADOOP_CLIENT_OPTS="-Xmx1024m"

1
সুপারউজারে আপনাকে স্বাগতম। দয়া করে কেবল লিঙ্কযুক্ত উত্তর সরবরাহ করবেন না। লিঙ্কযুক্ত পৃষ্ঠাগুলি পরিবর্তন বা স্থানান্তর করতে পারে এবং এটি আপনার উত্তর ভবিষ্যতের দর্শকদের কাছে অকেজো করে দেবে। পরিবর্তে, লিঙ্কটিতে থাকা উত্তরে যুক্ত করুন বা, এটি খুব বড় হলে, লিঙ্কটির সাথে একটি প্রাকস বা সংক্ষিপ্ত সংস্করণ সরবরাহ করুন।
টগ
আমাদের সাইট ব্যবহার করে, আপনি স্বীকার করেছেন যে আপনি আমাদের কুকি নীতি এবং গোপনীয়তা নীতিটি পড়েছেন এবং বুঝতে পেরেছেন ।
Licensed under cc by-sa 3.0 with attribution required.