Giter VIP home page Giter VIP logo

Comments (14)

LG-1 avatar LG-1 commented on June 23, 2024 2

@nkxujun 确实。
我大概遇到了两个问题，都是关于oozie安装成功后跑job不成功的问题，弄了差不多两天。
Q1：JA017: Could not lookup launched hadoop Job ID
Q2：Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Unknown protocol，org.apache.oozie.action.ActionExecutorException: JA009: Unknown protocol: org.apache.hadoop.yarn.api.ApplicationClientProtocolPB

1、oozie安装参考：https://segmentfault.com/a/1190000002738484
2、我安装成功后便出现Q1，解决主要参考了：https://stackoverflow.com/search?q=org.apache.oozie.action.ActionExecutorException%3A+JA017%3A+Could+not+lookup+launched+hadoop+Job+ID+
3、之后又出现Q2，解决主要受启发于：https://kb.informatica.com/solution/23/Pages/51/303398.aspx
4、配置文件的修改过程可以参考：https://stackoverflow.com/questions/43426691/oozie-workflow-failed-due-to-error-ja017?noredirect=1
5、由于解决过程比较纠结，所以至今我依然没有锁定具体是那一个因子造成成了上述问题，只是知道大体范围就在上面的修改空间。下面我会附上我跑起来OK的配置文件内容，希望能有参考作用。

${HADOOP_HOME}/etc/hadoop/yarn-site.xml:


<configuration>

<!-- Site specific YARN configuration properties -->
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
</property>
<property>
	<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
	<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

<property>
	<name>yarn.resourcemanager.resource-tracker.address</name>
	<value>master:8025</value>
</property>
<property>
	<name>yarn.resourcemanager.scheduler.address</name>
	<value>master:8030</value>
</property>
<property>
	<name>yarn.resourcemanager.address</name>
	<value>master:8033</value>
</property>

</configuration>

${HADOOP_HOME}/etc/hadoop/mapred-site.xml:

<configuration>
    <property>  
        <name>mapreduce.framework.name</name>  
        <value>yarn</value>  
    </property>  

<property>
    <name>mapreduce.job.queuename</name>
    <value>default</value>
</property>

<property>
    <name>mapreduce.tasktracker.http.address</name>
    <value>0.0.0.0:50060</value>
</property>
<property>
    <name>mapreduce.tasktracker.report.address</name>
    <value>127.0.0.1:0</value>
</property>
    <property>  
        <name>mapreduce.jobtracker.http.address</name>  
        <value>master:50030</value>  
    </property>  
    <property>  
        <name>mapreduce.jobhistory.address</name>  
        <value>master:10020</value>  
    </property>  
    <property>  
        <name>mapreduce.jobhistory.webapp.address</name>  
        <value>master:19888</value>  
    </property>

备注：以上，master是我的主机名，请自行修改，另外一个比较关键的是端口号的配置，具体配置什么端口号一般来说应该是固定的，但是也请注意查看自己机器具体启用的是哪个端口。

from easyml.

sinllychen avatar sinllychen commented on June 23, 2024

@LG-1 您好，您可以到tomcat下的logs目录查看一下相关日志，然后把错误日志给我们截个图吗？

from easyml.

LG-1 avatar LG-1 commented on June 23, 2024

@sinllychen 您好，目前还没有配置tomcat服务

from easyml.

LG-1 avatar LG-1 commented on June 23, 2024

@sinllychen 对不起，我再去后台看了一下，HDFS目录是有的，workflow.xml文件也写进去了。如下图。
可为什么还是会报路径不存在呢？
oozie-capture

from easyml.

LG-1 avatar LG-1 commented on June 23, 2024

@sinllychen 具体错误返回日志

`[mkdirs]hdfs://master:9000/EML/oozie/APP-PATH-111
E0504 : E0504: App directory [/EML/oozie/APP-PATH-111] does not exist
	at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:508)
	at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:591)
	at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:561)
	at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:479)
	at org.apache.oozie.client.OozieClient.run(OozieClient.java:655)
	at eml.studio.server.util.OozieUtil.submit(OozieUtil.java:48)
	at eml.studio.server.oozie.instance.OozieInstance.exec(OozieInstance.java:56)
	at eml.studio.server.rpc.JobServiceImpl.submit(JobServiceImpl.java:435)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)`

from easyml.

sinllychen avatar sinllychen commented on June 23, 2024

@LG-1 你的日志上显示hdfs地址是master:9000这个master是你自己配的吗？我们默认是hadoop-master

from easyml.

sinllychen avatar sinllychen commented on June 23, 2024

@LG-1顺带问您一下你是在ubuntu下安装的docker，然后安装我们的集群吗？然后通过window系统访问吗？如果是这样的话，你的ubuntu需要配置host: 本机ip hadoop-master和mysql。并且你的windows系统上也需要配置host: ubuntu的ip hadoop-master和mysql

from easyml.

LG-1 avatar LG-1 commented on June 23, 2024

@sinllychen 是的，整个后台我移植到自己搭建的环境了。联通性是没有问题的，我上传的数据已成功上传到HDFS。但job依然不能创建，提示路径找不到，而后台实际是有这个路径及配置文件的。
oozie-capture

from easyml.

LG-1 avatar LG-1 commented on June 23, 2024

@sinllychen 您好，我没有安装docker，我是在ubuntu下安装虚拟机然后搭建的Hadoop+Spark集群。目前webapplication也放在集群中的master机器上的，所以用户的访问权限应该是没有问题的。
mysql目前也装在master机器上。数据交互应该也没有问题(因为上传数据upload data已经成功将源信息写入表中，数据文件存入HDFS中)。

from easyml.

LG-1 avatar LG-1 commented on June 23, 2024

@sinllychen 在后台单独运行job也是没问题的。
oozie-capture1

oozie-capture2

from easyml.

sinllychen avatar sinllychen commented on June 23, 2024

@LG-1 您好，您能点击到oozie web console中的任意一个failed任务，查看一下失败的日志吗？还有，您的hadoop的job history server启动了吗？

from easyml.

LG-1 avatar LG-1 commented on June 23, 2024

@sinllychen 您好，创建job出错的问题已经解决了。谢谢。
总结一下原因：应该不是由于EML系统造成的。
应该是由于oozie本身配置的一些问题，依据报错信息，经过一些努力，在修改了yarn-site.xml、mapred-site.xml等配置文件后，从命令行可以成功创建job并不报错的情况下，在EML创建job也就不会报错了。
再次感谢。

from easyml.

nkxujun avatar nkxujun commented on June 23, 2024

@LG-1 如果方便请总结一下对配置文件做了哪一些修改，这样遇到同样问题的人可以快点找到方案，谢谢！

from easyml.

LG-1 avatar LG-1 commented on June 23, 2024

补充两点：
[1]，如果出现Unknown protocol: org.apache.hadoop.yarn.api.ApplicationClientProtocolPB。极有可能是因为yarn-site.xml中yarn.resourcemanager.address配置的端口号与job.properties中的jobTracker不一致。两者需要一致。
[2]，如果出现java.lang.NoSuchFieldError: HADOOP_CLASSPATH。极有可能是因为安装的hadoop版本与oozie支持的版本不一致。请参考：https://stackoverflow.com/questions/41205447/oozie-example-map-reduce-job-fails-with-java-lang-nosuchfielderror-hadoop-class。亲测有效。

from easyml.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.