0
点赞
收藏
分享

微信扫一扫

Hadoop入门小例子WordCount


参考自书籍《Hadoop+Spark 大数据巨量分析与机器学习》

1 编写测试程序例子

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;



import java.io.IOException;

import java.util.StringTokenizer;



public class WordCount {

public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

Configuration conf = new Configuration();

Job job = Job.getInstance(conf, "word count");

job.setJarByClass(WordCount.class);

job.setMapperClass(TokenizerMapper.class);

job.setCombinerClass(IntSumReducer.class);

job.setReducerClass(IntSumReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

FileInputFormat.addInputPath(job, new Path(args[0]));

FileOutputFormat.setOutputPath(job, new Path(args[1]));

System.exit(job.waitForCompletion(true) ? 0 : 1);

}



private static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {

private final static IntWritable one = new IntWritable();

private Text word = new Text();



@Override

public void map(Object key, Text value, Context context) throws IOException, InterruptedException {

StringTokenizer itr = new StringTokenizer(value.toString());

while (itr.hasMoreTokens()) {

word.set(itr.nextToken());

context.write(word, one);

}

}

}



private static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

private IntWritable result = new IntWritable();



public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {

int sum = 0;

for (IntWritable val : values) {

sum += val.get();

}

result.set(sum);

context.write(key, result);

}

}

}

 

2 设置hadoop的依赖

$ vim ~/.bashrc

在最后面添加如下行并保存

export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath):$CLASSPATH

$ source ~/.bashrc //使配置生效

 

3 程序打包

$ mkdir ~/wordcount  //先创建目录

$ rz //将本地的WordCount.java上传至当前目录

$ javac WordCount.java //编译程序

$ jar cf wc.jar WordCount*.class //打成jar包

$ ll //可看到如下图片

Hadoop入门小例子WordCount_apache

 

4 创建测试文本文件

$ mkdir -p ~/wordcount/input

$ cp ~/hadoop/LICENSE.txt ~/wordcount/input

//在hdfs创建目录

$ hdfs dfs -mkdir -p /user/hduser/wordcount/input

$ cd ~/wordcount/input

//上传文本文件到HDFS

$ hdfs dfs -copyFromLocal LICENSE.txt /user/hduser/wordcount/input

$ hdfs dfs -ls /user/hduser/wordcount/input //hdfs查看当前目录

Hadoop入门小例子WordCount_apache_02

 

5 运行WordCount.java

$ cd ~/wordcount

//运行程序,格式为“hadoop jar wc.jar [输入文件][输出目录]”,例子如下:

$ hadoop jar wc.jar WordCount /user/hduser/wordcount/input/LICENSE.txt /user/hduser/wordcount/output

//执行成功后,可见下面截图

$ hdfs dfs -ls /user/hduser/wordcount/output //有个success文件说明成功了,part-r-00000是运行结果文件

Hadoop入门小例子WordCount_Text_03

 

6 查看HDFS中的输出文件内容

$ hdfs dfs -cat /user/hduser/wordcount/output/part-r-00000|more

 

举报

相关推荐

0 条评论