0
点赞
收藏
分享

微信扫一扫

oozie-5.2.1 build on hadoop3.2.2

腾讯优测 2022-04-23 阅读 26
hadoop

# Build oozie-5.2.1 基于docker

容器内网页下载:

    wget 下载网址

临时在容器中配置环境变量

    ln -s maven-apache-3.2.1 maven3

    export MAVEN_HOME=/root/maven3

    export PATH=$PATH:$MAVEN_HOME/bin

oozie解压包下执行:下载各类jar包

    bin/mkdistro.sh -DskipTests -Dhadoop-3 -Dhadoop.version=3.2.2 -Dhive.version=3.1.2 -Dhbase=2.4.6

问题一:Apache Oozie Fluent Job API

    Blocked mirror for repositories

解决:

    1.在 home 目录下的 .m2/ 目录创建 settings.xml 文件,编辑文件

    ll /root/.m2/

    cd ~/.m2

    2.通过命令 mvn help:effective-settings 检查当前settings

问题二:Apache Oozie Core

     Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project

    oozie-core: Compilation failure

    [ERROR] /root/oozie-5.2.1/core/src/main/java/org/apache/oozie/dependency/HCatURIHandler.java:[50,37] 找不到符号

解决:高版本中代码所处类变更,两种解决方法,一种修改高版本中代码,二添加低版本jar到core的pom.xml文件中

    mvn仓库中查找jar

    1.hive-shims (common)

        1.2.2

    2.commons-httpclient

        3.1

        添加标签<scope>test</scope>

    3.hive-metastore

        1.2.2

        添加标签<scope>test</scope>

    4./root/oozie-5.1/core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java

        更改xxx.java为xxx.java.bak(bak备份)

    5.apache oozie Tools

        问题报错:/root/oozie-5.2.1/tools/src/test/java/org/apache/oozie/tools/TestECPolicyDisabler.java:[50,71] org.apache.oozie.tools.TestECPolicyDisabler.MockDistributedFileSystem中的getErasureCodingPolicy(org.apache.hadoop.fs.Path)无法覆盖org.apache.hadoop.hdfs.DistributedFileSystem中的getErasureCodingPolicy(org.apache.hadoop.fs.Path)

    解决:更改xxx.java为xxx.java.bak(bak备份)

全部通过成功后会在/root/oozie-5.2.1/distro/target生成oozie-5.2.1-distro.tar.gz压缩包,将其解压到一个目录中

    tar zxvf oozie-5.2.1-distro.tar.gz -C /root/oozie08

    If using the ExtJS library copy the ZIP file to the libext/ directory. If hadoop and hcatalog libraries are not already included in the war, add the corresponding libraries to libext/ directory.

    1)在其解压的bin级目录下创建libext目录,oozie-sharelib-5.2.1.tar.gz解压,复制share/lib/hcatalog下所有jar包到libext中

    2)复制mysql-connect.jar到libext中,下载ext-2.2.zip到libext中

    3)cp $HADOOP_HOME/share/hadoop/*/*.jar

        cp $HADOOP_HOME/share/hadoop/*/lib/*.jar

# 部署 连接mysql

    1)bin/oozie-setup.sh sharelib create -fs hdfs://hadoop-master:9000 -locallib oozie-sharelib-5.2.1.tar.gz

    报错:

        Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object jar包不存在

    解决:

        将lib中的guave低版本jar删除并替换为高版本jar包

        再将mysql-connector-jar复制到lib下

    2)初始化数据库,先将oozie-site.xml文件编辑好并覆盖

        bin/ooziedb.sh create -sqlfile oozie.sql -run

    3)启动oozie

        bin/oozied.sh start

    4)查看状态

        bin/oozie admin -oozie http://localhost:11000/oozie -status

finish完成

举报

相关推荐

0 条评论