Flink 本地环境搭建

本地环境

Java > 1.8、mac os

参考

​

启动

bin/start-cluster.sh 

访问界面

​​http://127.0.0.1:8081/#/overview​​

示例：词语统计

package com.jihitee.myflink.flinklearn;

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.util.Collector;

public class SocketWindowWordCountJava {
    public static void main(String[] args) throws Exception {
        ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

        DataSet<String> text = env.readTextFile("/path/flinklearn/in.txt");

        DataSet<Tuple2<String, Integer>> counts =
                // split up the lines in pairs (2-tuples) containing: (word,1)
                text.flatMap(new Tokenizer())
                        // group by the tuple field "0" and sum up tuple field "1"
                        .groupBy(0)
                        .sum(1);

        String outputPath = "/path/out.txt";
        counts.writeAsCsv(outputPath, "\n", " ");
        env.execute("myflink");
    }

    public static class Tokenizer implements FlatMapFunction<String, Tuple2<String, Integer>> {
        @Override
        public void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
            String[] tokens = value.split(" ");
            // emit the pairs
            for (String token : tokens) {
                if (token.length() > 0) {
                    out.collect(new Tuple2<String, Integer>(token, 1));
                }
            }
        }
    }
}

运行

./bin/flink run -c com.jihitee.myflink.flinklearn.SocketWindowWordCountJava  /youpath/flinklearn-1.0-SNAPSHOT.jar

问题

​

Flink本地模式报错:Exception in thread main java.lang.NoClassDefFoundError

解决:

修改pom.xml中的依赖项，屏蔽掉 <scope>provided</scope>

<dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-scala_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
<!--        <scope>provided</scope>-->
        </dependency>