0
点赞
收藏
分享

微信扫一扫

踩坑:Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer

Ichjns 2022-04-29 阅读 215

今天在做MapReduce的时候,为了做个降序排列,所以定义了一个自定义序列,然后运行的时候就出现了这个问题,差了好多教程,最后发现错在了一个比较离谱的地方,记录一下,顺带协商其他几种可能。(1-3来源网络,4亲身经历)
先放个报错信息以供参考

2022-04-29 01:04:34,361 WARN  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(418)) - Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer
java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
        at org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:144)
        at org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:130)
        at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:65)
        at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:903)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1019)
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:409)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:83)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:711)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodException: model.classes.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
        ... 16 more
2022-04-29 01:04:34,364 INFO  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(486)) - map task executor complete.
2022-04-29 01:04:34,370 WARN  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1220408697_0001
java.lang.Exception: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:424)
        at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:83)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:711)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: model.classes.<init>()
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
        at org.apache.hadoop.io.WritableComparator.newKey(WritableComparator.java:144)
        at org.apache.hadoop.io.WritableComparator.<init>(WritableComparator.java:130)
        at org.apache.hadoop.io.WritableComparator.get(WritableComparator.java:65)
        at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:903)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1019)
        at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:409)
        ... 10 more
Caused by: java.lang.NoSuchMethodException: model.classes.<init>()
        at java.lang.Class.getConstructor0(Class.java:3082)
        at java.lang.Class.getDeclaredConstructor(Class.java:2178)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
        ... 16 more

情况1

包导入错了,这是网上最多情况的报错原因,比如Text应该要导入org.apache.hadoop.io.Text之类的,仔细检查一下,看看有没有。

情况2

自定义序列的时候没有没有完全继承WritableComparable, 这种情况的话,只要继承一下WritableComparable就ok了, WritableComparable已经继承了WritableComparable。比如我的自定义序列:

public class classes implements WritableComparable<classes>{...}

情况3

如果是自定义序列的话,再MapReduce操作中是要求其可迭代的,也就是说需要重写一个compareTo方法,比如我的

public int compareTo(classes o) {
        //降序
       return (int) (-this.avgsorce+o.getAvgsorce());
    }

如果没写的话也可能出现报错

情况4

最后就是我的问题了,很简单,在自定义序列的时候,我只定义了一个有参的构造器,总所周知,在Java里如果已经定义了构造器,系统就不再有默认构造器了,这就导致我定义的这个自定义序列没有无参构造器,好巧不巧,MapReduce排序的时候就偏偏要用到这个无参构造器,得嘞,这可不就报错了,到此我定义了一个无参构造器,问题解决

public classes(){

    }

这坑有点深,找了我好久,再加上现在是北京时间1:21,半夜啦!!忍不住发一下吐槽一下(这句纯吐槽,不想看可以略过哈哈哈)

转载请注明出处,谢谢!

举报

相关推荐

0 条评论