0
点赞
收藏
分享

微信扫一扫

CDH6.3.2搭建HIVE TO TEZ步骤

何晓杰Dev 2022-04-14 阅读 82
hivehadooptez

|引言

参考：CDH6.3.2 hive on tez搭建过程_我不是橙子的博客-CSDN博客

CDH6.3.2版本搭建Tez执行引擎过程 - 虎啸千峰 - 博客园

hive on tez集成完整采坑指南(含tez-ui及安全环境)_匆匆z2的博客-CSDN博客_hive on tez

下载过程中若是感觉慢的话可以参考这个操作步骤：

https://blog.csdn.net/ly8951677/article/details/124151146

环境：

服务器：CentOS7.9.2009

jdk：1.8.0_181

CDH：6.3.2

TEZ：0.9.2

1. 下载

1.1 下载TEZ

官网：Apache Tez – Welcome to Apache TEZ®

采用Apache下载路径寻找：Welcome to The Apache Software Foundation!

选择编译版或者源码版下载：Index of /tez/0.9.2

直接下载：https://dlcdn.apache.org/tez/0.9.2/apache-tez-0.9.2-src.tar.gz

 

 

 

 

 

 

Index of /tez/0.9.2icon-default.png?t=M276https://dlcdn.apache.org/tez/0.9.2/

 

1.2 下载Maven

官网地址：Maven – Welcome to Apache Maven

 

1.3 下载安装protobuf

注：源码编译过程中hadoop中需要使用protobuf，需要下载protobuf-2.5.0

下载地址：Releases · protocolbuffers/protobuf · GitHub

 

2. 安装

2.1安装maven

  1. 解压

    [root@node02 ~]# cd /data1/software/tez/
[root@node02 tez]# tar -zxvf apache-maven-3.8.5-bin.tar.gz -C /data1/program/

  2. 配置/etc/profile

    [root@node02 tez]# vim /etc/profile
#在最后一行输入以下内容
#maven
export MAVEN_HOME=/data1/program/apache-maven-3.8.5
export PATH=$PATH:$MAVEN_HOME/bin

  3. 使配置生效并校验

    [root@node02 tez]# source /etc/profile
[root@node02 tez]# mvn --version
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /data1/program/apache-maven-3.8.5
Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_181-cloudera/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-862.el7.x86_64", arch: "amd64", family: "unix"
##备注，若是maven有新版本，安装后不退出当前窗口还是会显示旧版本信息。退出后重新登录后显示正常。

2.2 安装protobuf（我为什么是2.5.0版本，因为是前人这么操作。）

  1. 解压

    [root@node02 tez]# tar -zxvf protobuf-2.5.0.tar.gz -C /data1/program/

  2. 配置并生效

    
[root@node02 tez]# vim /etc/profile
#在最后一行输入以下内容
#protobuf-2.5.0
export POF_HOME=/data1/program/protobuf-2.5.0
export PATH=$PATH:$POF_HOME/bin
[root@node02 tez]# source /etc/profile

  3. 编译安装

    [root@node02 tez]# cd /data1/program/protobuf-2.5.0/
[root@node02 protobuf-2.5.0]# ./configure
[root@node02 protobuf-2.5.0]# make && make install
##注：如果在编译的过程中报错缺少c或者c++，使用yum方式安装后重试即可

  4. 验证

    [root@node02 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
注：当configure校验不通过的时候,缺少哪些包就安装,一般需要安装gcc

2.3 tez编译0.9.2版本

  1. 解压

    [root@node02 protobuf-2.5.0]# cd /data1/software/tez/
[root@node02 tez]# tar -zxvf apache-tez-0.9.2-src.tar.gz

  2. 修改配置文件pom.xml

    ##进行第二步配置文件命令
[root@node02 tez]# cd /data1/software/tez/apache-tez-0.9.2-src
[root@node02 apache-tez-0.9.2-src]# vim pom.xml
<!--以下是修改不走 -->
<!--第一处:这里才是修改，后续2,3,4都是增加-->
<hadoop.version>3.0.0-cdh6.3.2</hadoop.version>
<!--第二处：repositorys标签内最后增加-->
<repository>
  <id>cloudera</id>
  <url><https://repository.cloudera.com/artifactory/cloudera-repos/></url>
  <name>Cloudera Repositories</name>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
</repository>
<!--第三处:pluginRepositories标签内最后增加-->
<pluginRepository>
  <id>cloudera</id>
  <name>Cloudera Repositories</name>
  <url><https://repository.cloudera.com/artifactory/cloudera-repos/></url>
</pluginRepository>
<!--第四处:dependencyManagement标签里的dependencies标签内最后一行。不过这里有个1.9版本，我暂时不理，有兴趣可以验证下需不需要加这个。成功可以留言告知下-->
<dependency>
  <groupId>com.sun.jersey</groupId>
  <artifactId>jersey-client</artifactId>
  <version>1.19</version>
</dependency>
<!--第五处:-->
<!--<module>tez-ext-service-tests</module>
<module>tez-ui</module>-->
<!--注:将这两个注释掉,如果有需要可以不用注释，赶时间的建议最好注释，要不maven编译很耗时。-->

  3. maven编译

    
[root@node02 apache-tez-0.9.2-src]# mvn clean package -Dmaven.javadoc.skip=true -Dmaven.test.skip=true
##注:这样会跳过test编译,很快就编译完成
##经过一段时间等待，所有输出选项均为SUCCESS则表示为编译通过

##Maven编译后的压缩包在项目的tez-dist/target目录下，没有编译成功是没有target这个目录的
##查看是否成功
[root@node02 apache-tez-0.9.2-src]# cd /data1/software/tez/apache-tez-0.9.2-src/tez-dist/target
[root@node02 apache-tez-0.9.2-src]# ll
drwxr-xr-x 2 root root        6 Apr  8 19:41 archive-tmp
drwxr-xr-x 3 root root     4096 Apr  8 19:41 tez-0.9.2
drwxr-xr-x 3 root root     4096 Apr  8 19:41 tez-0.9.2-minimal
-rw-r--r-- 1 root root 13421050 Apr  8 19:41 tez-0.9.2-minimal.tar.gz
-rw-r--r-- 1 root root 68025360 Apr  8 19:41 tez-0.9.2.tar.gz

3.配置

  1. 上传到hdfs路径

    [root@node02 target]# su hdfs
bash-4.2$ hdfs dfs -mkdir /tez
bash-4.2$ tez-0.9.2.tar.gz /tez

  2. 配置客户端（每台集群机子都需要配置，切记，弄好了也可以scp过去）

    1. 在CDH里的lib目录生成tez目录

      [root@node02 target]# mkdir -p /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/tez/conf

    2. 生成tez_site.xml文件

      [root@node02 target]# vim /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/tez/conf/tez-site.xml
##一定要加${fs.defaultFS}，要不就找不到，这个就是第一步上传到hdfs的压缩包路径
<configuration>
 <property>
  <name>tez.lib.uris</name>
  <value>${fs.defaultFS}/tez/tez-0.9.2.tar.gz</value>
 </property>
 <property>
  <name>tez.use.cluster.hadoop-libs</name>
  <value>false</value>
 </property>
 <property>
  <name>tez.am.launch.env</name>
  <value>LD_LIBRARY_PATH=${PARCELS_ROOT}/CDH/lib/hadoop/lib/native</value>
 </property>
 <property>
  <name>tez.task.launch.env</name>
  <value>LD_LIBRARY_PATH=${PARCELS_ROOT}/CDH/lib/hadoop/lib/native</value>
 </property>
</configuration>

    3. 拷贝jar包和lib目录

      [root@node02 target]# cd tez-0.9.1-minimal
[root@node02 tez-0.9.1-minimal]# cp ./*.jar /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/tez/
[root@node02 tez-0.9.1-minimal]# cp -r ./lib /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/tez/

    4. 避免kryo的错误

      [root@node02 tez-0.9.1-minimal]# cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/auxlib
[root@node02 auxlib]# mv hive-exec-2.1.1-cdh6.3.2-core.jar hive-exec-2.1.1-cdh6.3.2-core.jar.bak
[root@node02 auxlib]# mv hive-exec-core.jar hive-exec-core.jar.bak
##备注，要不就会报异常。所以一定要处理

  3. 配置HADOOP_CLASSPATH路径（这个在cdh上面配置就好）

    HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/tez/conf:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/tez/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/tez/lib/*

    hive-env.sh 的 Gateway 客户端环境高级配置代码段（安全阀）

  4.  

     

     

4.启动

hive> set hive.execution.engine=tez;
hive> set hive.tez.container.size=1024;
## 第一次查询的时候要加载jar包，所以比较慢。耐心点就好。示例：
hive> select count(1) from cwqejg_test2 ;

 

 

5. 异常

5.1 HiveSplitGenerator异常

  • 异常信息

org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator

 

  • 解决方案

    hive> set hive.tez.container.size=1024;

5.2 kryo版本不一致异常

  • 异常信息

    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1649661932037_0034_3_00, diagnostics=[Vertex vertex_1649661932037_0034_3_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
        at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)
        at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4120)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:206)
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2931)
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2878)
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2860)
        at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
        at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1956)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:205)
        at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2317)
        at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2303)
        at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)
        at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
        ... 25 more
Caused by: java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/Serializer
        at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:404)
        at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:317)
        at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:131)
        ... 30 more
Caused by: java.lang.ClassNotFoundException: com.esotericsoftware.kryo.Serializer
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 33 more

  • 解决方案

    [root@node01 ~]# cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/auxlib
[root@node01 auxlib]# mv hive-exec-2.1.1-cdh6.3.2-core.jar hive-exec-2.1.1-cdh6.3.2-core.jar.bak
[root@node01 auxlib]# mv hive-exec-core.jar hive-exec-core.jar.bak

5.2.1 kyro修复后异常

  • 异常信息

    22/04/13 10:22:22 ERROR exec.Task: Failed to execute tez graph.
java.io.FileNotFoundException: File file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/auxlib/hive-exec-2.1.1-cdh6.3.2-core.jar does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
        at org.apache.hadoop.hive.ql.exec.tez.DagUtils.checkPreExisting(DagUtils.java:974)
        at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:991)
        at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:902)
        at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:845)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:466)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:296)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:157)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:227)

  • 解决方案

    退出客户端，重新进入就好。

举报

相关推荐

大数据学习之CDH6.3.2搭建

pythonpdf前端

cdh6.1.1的hive升级到CDH6.3.2的hive

javahiveapacheHive大数据

CDH6.3.2开启kerberos认证

cdhLinux系统/运维

CDH6.3.2集成Kerberos

redis数据库缓存

CDH6.3.2集群部署(离线版本)

hadoopmysqlhive

CDH6.3.2 的pyspark读取excel表格数据写入hive中的问题汇总

excelhivehadoop

flink1.14.5使用CDH6.3.2的yarn提交作业

flink大数据

运维实战100：CDH5.16.2升级至CDH6.3.2

运维大数据cdh集群运维clouderahadoop系统/运维

【cdh】hive执行SQL提示缺少3.0.0-cdh6.3.2-mr-framework.tar.gz文件

gpt深度学习人工智能

Hive整合Tez

hive大数据jarhadoopHive
0 条评论