0
点赞
收藏
分享

微信扫一扫

【FLINK】 No Executor found. Please make sure to export the HADOOP_CLASSPATH

爱做梦的夏夏 2022-04-13 阅读 107
java后端

背景:

为了更好的使用flink sql&table,系统决定从flink 从1.10 升级到 1.11,然后在测试flink example的命令时出现了这个bug

(不懂怎么安装的直接参考官网链接:https://ci.apache.org/projects/flink/flink-docs-release-1.11/try-flink/local_installation.html)

java.lang.IllegalStateException: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath.
 For more information refer to the "Deployment & Operations" section of the official 
 Apache Flink documentation. at org.apache.flink.yarn.cli.FallbackYarnSessionCli.isActive(FallbackYarnSessionCli.java:59) at 
 org.apache.flink.client.cli.CliFrontend.validateAndGetActiveCommandLine(CliFrontend.java:1090) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:218) at 
 org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992) at 
 org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) at 
 org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)

原因:

产生这个问题的原因这个,具体的意思就是从Flink 1.11开始,flink-shaded-hadoop-2-uberFlink项目不再正式支持使用发行版。建议用户通过提供Hadoop依赖关系HADOOP_CLASSPATH(请参见上文),官方(https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/hadoop.html)针对这个问题有详细的说明见如下:

Hadoop Integration

  • Providing Hadoop classes
  • Running a job locally
  • Using?[这里是代码006]?jar for resolving dependency conflicts (legacy)

Providing Hadoop classes

In order to use Hadoop features (e.g., YARN, HDFS) it is necessary to provide Flink with the required Hadoop classes, as these are not bundled by default.

The recommended approach is adding the Hadoop classpath to Flink through theHADOOP_CLASSPATHenvironment variable.

Flink will use the environment variableHADOOP_CLASSPATHto augment the classpath that is used when starting Flink components such as the Client, JobManager, or TaskManager. Most Hadoop distributions and cloud environments will not set this variable by default so if the Hadoop classpath should be picked up by Flink the environment variable must be exported on all machines that are running Flink components.

When running on YARN, this is usually not a problem because the components running inside YARN will be started with the Hadoop classpaths, but it can happen that the Hadoop dependencies must be in the classpath when submitting a job to YARN. For this, it’s usually enough to run

export HADOOP_CLASSPATH=`hadoop classpath`

in the shell. Note thathadoopis the hadoop binary and thatclasspathis an argument that will make it print the configured Hadoop classpath. The classpath returned byhadoop classpathalso includes the Hadoop configuration directories.

If you are manually assembling theHADOOP_CLASSPATHvariable, we recommend adding the Hadoop configuration directories as well.

Running a job locally

To run a job locally as one JVM process using the mini cluster, the required hadoop dependencies have to be explicitly added to the classpath of the started JVM process.

To run an application using Maven (also from IDE as a Maven project), the required Hadoop dependencies can be added as provided to the pom.xml, e.g.:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.8.3</version>
    <scope>provided</scope>
</dependency>

This way it should work both in local and cluster mode where the provided dependencies are added elsewhere as described before.

To run or debug an application in IntelliJ Idea the provided dependencies can be included to the class path in the “Run|Edit Configurations” window.

Usingflink-shaded-hadoop-2-uberjar for resolving dependency conflicts (legacy)

**Warning:**Starting from Flink 1.11, usingflink-shaded-hadoop-2-uberreleases is not officially supported by the Flink project anymore. Users are advised to provide Hadoop dependencies throughHADOOP_CLASSPATH(see above).

The Flink project used to (until Flink 1.10) release Hadoop distributions for specific versions, that relocate or exclude several dependencies to reduce the risk of dependency clashes. These can be found in theAdditional Componentssection of the download page. For these versions it is sufficient to download the correspondingPre-bundled Hadoopcomponent and putting it into the/libdirectory of the Flink distribution.

If the used Hadoop version is not listed on the download page (possibly due to being a Vendor-specific version), then it is necessary to buildflink-shadedagainst this version. You can find the source code for this project in theAdditional Componentssection of the download page.

NoteIf you want to buildflink-shadedagainst a vendor specific Hadoop version, you first have to configure the vendor-specific maven repository in your local maven setup as describedhere.

Run the following command to build and installflink-shadedagainst your desired Hadoop version (e.g., for version2.6.5-custom):

mvn clean install -Dhadoop.version=2.6.5-custom

After this step is complete, put theflink-shaded-hadoop-2-uberjar into the/libdirectory of the Flink distribution.

解决思路:

参考下面链接:https://issues.apache.org/jira/browse/FLINK-19126

注意啦注意啦,在这里我做了个ZZ的骚操作!!!!

在这里别自己想太多,一开始我理解这个命令是 export HADOOP_CLASSPATH=hadoop的文件路径,就这样运行了

export HADOOP_CLASSPATH=/usr/local/hadoop

发现一点作用的都没起,其实直接运行这个命令就可以了不需要改任何东西,马上就不报错了

 export HADOOP_CLASSPATH=`hadoop classpath`

后续:

其实hadoop classpath 命令就是把有关于hadoop的环境全部加载出来

hadoop classpath

举报

相关推荐

0 条评论