azkaban / azkaban-plugins Goto Github PK
View Code? Open in Web Editor NEWPlugins for Azkaban.
Home Page: https://azkaban.github.io
License: Apache License 2.0
Plugins for Azkaban.
Home Page: https://azkaban.github.io
License: Apache License 2.0
The idea behind this change is to make it easier to search for those long running reportal job from command line.
This fix will add two JVM args and they look something like this:
-Dreportal.user.name=hreportal -Dreportal.execid=72
Example, if I try to go to /hdfs/user/bob, I see the "Permission denied" error message. However, if I try to go to /hdfs/user/bob/foo, I see a 500 page.
We display one cell per row instead of one row per row.
Internal JIRA: HADOOP-4463
There is no need to add -Dreportal.execid=${azkaban.flow.execid} here - https://github.com/azkaban/azkaban-plugins/blob/master/plugins/reportal/src/azkaban/viewer/reportal/ReportalTypeManager.java#L59 - because Azkaban's JobRunner already inserts it here - https://github.com/azkaban/azkaban/blob/master/azkaban-execserver/src/main/java/azkaban/execapp/JobRunner.java#L536-539
Counterpart to azkaban/azkaban#249
Other job types already do this.
In log-data.js, the job_id regexes expect job_<12 digits>_<4+ digits>
. However, on Hadoop 2, instead of <12 digits> of the form YYYYMMDDHHMM, the job_id appears appears to contain the milliseconds since epoch, which is variable in length and is currently 13 digits.
Also, in Hadoop 2, the job URL printed out in the logs does not contain job_
(which the current url regex is looking for) but instead looks something like http://<host>:<port>/proxy/application_<milliseconds_since_epoch>_<counter>
.
We should fix the regexes so that the job summary plugin will find the job ids and URLs.
Tracked by internal JIRA HADOOP-6977.
When running Azkaban on a grid with Hadoop security enabled, viewing a file in one's user directory results in a permission denied error.
I have root-caused this to the fact that the Parquet file viewer does not use the FileSystem
object passed in from the HdfsBrowserServlet
, which is properly set up to doAs the current user logged rather than the azkaban
user. As a result, the Parquet file viewer ends up trying to view the file as azkaban
, throwing the AccessControlException
. Currently, AvroParquetReader
does not have an API that lets one pass in a FileSystem
object. The fix for now is to remove the catch AccessControlException
block from the Parquet file viewer.
Tracked by internal ticket: HADOOP-5350
As mentioned in #114, the ParquetFileViewer tries to view every file as azkaban
, meaning it can only view files owned by azkaban
, with the group set to azkaban
, or world-readable. This means users will not be able to view their Parquet files through the HDFSViewer unless their Parquet files are world-readable.
The BinaryJSON file viewer seems to be unstable. At times, it fails to display a file at all and at other times, it dumps binary junk.
Tracked by internal JIRA: HADOOP-4478
Previous versions were compiled with 1.7
Since we added spaces to the HDFS viewer breadcrumb, it is no longer possible to just copy and paste the path.
Either:
The blue loading bar remains forever.
Today while checking out this project through subversion I got a strange error.
svn: E204899: Unable to make directories
This is because of below directory.
azkaban-plugins-2.1/plugins/jobtype/jobtypes/hive-0.8.1/hive-0.8.1/aux
After some research on google I found that on Windows platform a directory/file can't be created with name "aux".
Reference - http://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words.
Update HadoopSecurityManager_H_2_0 security manager to work with Hadoop 2.3.x.
The code that did this got deleted in this commit - 906b938 - but it's actually necessary, so we should add it back.
Every path in viewer.external.classpath will be prefixed with plugin's dir.
When a MapredLocalTask executes, it launches "hadoop jar ...", but the jar it specifies is sometimes the wrong jar. It sometimes erroneously uses hive-common instead of hive-exec.
The problem is with https://github.com/azkaban/azkaban-plugins/blob/master/plugins/jobtype/src/azkaban/jobtype/hiveutils/HiveQueryExecutorModule.java#L36. We should use new HiveConf(SessionState.class)
instead.
Tracked by internal JIRA HADOOP-5465.
This is to prevent scheduled reports with repeating from hogging Azkaban server. One use case we saw was there were many running instances of the same report because this report takes a while to complete and it is configured to repeat every minute.
The Hive plugin works with no user.to.proxy or proxy.user set anywhere. It probably defaults to the user that runs the azkaban executor daemon.
If Pig/hadoopJava did this, Azkaban security would be very easy for small shops (like us) who do not use Hadoop security features.
Internal JIRA: HADOOP-4646
Due to the CSS change for azkaban/azkaban#157, the sidebar and graph views for the Pig Visualizer graph no longer displays correctly.
Here - https://github.com/azkaban/azkaban-plugins/blob/master/plugins/reportal/src/azkaban/viewer/reportal/ReportalTypeManager.java#L58-59 - -Dreportal.user.name should be set to ${reportal.execution.user}, not userName. We want the property to show who executed the report, not who last edited the report.
The problem is that HadoopSecureHiveWrapper uses an interface method that has been removed: https://github.com/azkaban/azkaban-plugins/blob/master/plugins/jobtype/src/azkaban/jobtype/HadoopSecureHiveWrapper.java#L155
When a report (flow) is in the PREPARING state, the startTime is -1, and the Reportal UI renders this as "1969-12-31 15:59:59", which confuses and alarms users. Instead, we should leave the start time blank or display a "Not Started" message.
Tracked by internal JIRA HADOOP-6960.
Tracked by internal JIRA HADOOP-4744.
hive-0.8.1/aux/lib should have the antlr runtime. I used version 3.0.1 and that worked. I don't know what is latest.
antlr-runtime-3.0.1.jar
The jobtype and hadoopsecuritymanager jars are not being copied into the directory for the new Hive job type, which results in a ClassNotFound exception when the executor server starts.
This is the root cause for azkaban/azkaban#173
We reduced the limit for the lines of file displayed from 1000 to 100. The limit should be increased or made adjustable.
Tracked by internal JIRA: HADOOP-4479
The HDFS tab shows the local file system, not the Hadoop File System. Where would I change "file:///" to "hdfs://" ?
Companion to azkaban/azkaban#152
Similiar to azkaban core issue #224, the included links should be passed on downstream. This only affects Hadoop specific jobtypes
Companion to azkaban/azkaban#167
For ETL distcp use cases which need to copy files from multiple clusters, the Azkaban security manager needs to retrieve the HDFS tokens from other cluster before the file copying process can start.
when i clone the repos, it always throw
fatal: cannot create directory at 'plugins/jobtype/jobtypes/hive-0.8.1/hive-0.8.
1/aux': Invalid argument
it seems aux is not a valid name in windows environment?
thx
Due to azkaban/azkaban#255, package names of some classes have been change. For example, azkaban.webapp.AzkabanServer
is now azkaban.server.WebServer
.
The new Azkaban 2.6.1 JARs need to be imported so that the plugins can compile against the JARs with the new packages to be compatible with the new Azkaban core.
Display contents of file and schema in separate tabs.
Tracked by internal JIRA ticket: HADOOP-4413
The problem is that ReportalHiveRunner uses an interface method that has been removed:
https://github.com/azkaban/azkaban-plugins/blob/master/plugins/reportal/src/azkaban/jobtype/ReportalHiveRunner.java#L105
The HadoopSecurityManager_H_1_0 class expects to find a property 'user.to.proxy'. I have placed this in every configuration file and .job file, and nothing has worked.
Which file should this property be in?
Here is the full log section for this attempt at running the java-wc job. HadoopSecurityManager_H_1_0 is clearly trying to pass in a 'user.to.proxy' property which is not there.
2013/08/02 03:08:05.255 +0000 INFO [pig-upload] [Azkaban] Need to proxy. Getting tokens.
2013/08/02 03:08:05.255 +0000 INFO [pig-upload] [Azkaban] Getting hadoop tokens for apxqueue
2013/08/02 03:08:05.255 +0000 INFO [HadoopSecurityManager] [Azkaban] proxy user apxqueue not exist. Creating new proxy user
2013/08/02 03:08:05.258 +0000 INFO [pig-upload] [Azkaban] Getting DFS token from 10.176.235.204:8020hdfs://ip-10-176-235-204.us-west-1.compute.internal:8020
java.lang.NullPointerException
at org.apache.hadoop.security.SecurityUtil.setTokenService(SecurityUtil.java:246)
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:408)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:571)
at azkaban.security.HadoopSecurityManager_H_1_0$2.getToken(HadoopSecurityManager_H_1_0.java:268)
at azkaban.security.HadoopSecurityManager_H_1_0$2.run(HadoopSecurityManager_H_1_0.java:258)
at azkaban.security.HadoopSecurityManager_H_1_0$2.run(HadoopSecurityManager_H_1_0.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at azkaban.security.HadoopSecurityManager_H_1_0.prefetchToken(HadoopSecurityManager_H_1_0.java:253)
at azkaban.jobtype.HadoopPigJob.getHadoopTokens(HadoopPigJob.java:166)
at azkaban.jobtype.HadoopPigJob.run(HadoopPigJob.java:102)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:379)
at azkaban.execapp.JobRunner.run(JobRunner.java:280)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
azkaban.security.commons.HadoopSecurityManagerException: Failed to get hadoop tokens! nullnull
at azkaban.security.HadoopSecurityManager_H_1_0.prefetchToken(HadoopSecurityManager_H_1_0.java:318)
at azkaban.jobtype.HadoopPigJob.getHadoopTokens(HadoopPigJob.java:166)
at azkaban.jobtype.HadoopPigJob.run(HadoopPigJob.java:102)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:379)
at azkaban.execapp.JobRunner.run(JobRunner.java:280)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013/08/02 03:08:05.261 +0000 ERROR [pig-upload] [Azkaban] Job run failed!
2013/08/02 03:08:05.261 +0000 ERROR [pig-upload] [Azkaban] Failed to get hadoop tokens! nullnullnull
2013/08/02 03:08:05.261 +0000 INFO [pig-upload] [Azkaban] Finishing job pig-upload at 1375412885261
2013/08/02 03:08:05.267 +0000 INFO [wordcount-java] [Azkaban] Job Finished pig-upload with status FAILED
2013/08/02 03:08:05.278 +0000 INFO [wordcount-java] [Azkaban] Killing wordcount-java due to prior errors.
2013/08/02 03:08:05.289 +0000 INFO [wordcount-java] [Azkaban] Finishing up flow. Awaiting Termination
2013/08/02 03:08:05.289 +0000 INFO [wordcount-java] [Azkaban] Setting flow status to Failed.
2013/08/02 03:08:05.289 +0000 INFO [wordcount-java] [Azkaban] Flow is set to FAILED
2013/08/02 03:08:05.289 +0000 INFO [wordcount-java] [Azkaban] Setting end time for flow 8 to 1375412885289
2013/08/02 03:08:05.305 +0000 INFO [FlowRunnerManager] [Azkaban] Flow 8 is finished. Adding it to recently finished flows list.
2013/08/02 03:10:01.532 +0000 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished
2013/08/02 03:10:01.532 +0000 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 8 from recently finished flows list.
I am filing this issue because I encountered the same issue described at https://groups.google.com/d/msg/azkaban-dev/S9G9Lqmfm1Q/7pV0P7Re820J but there does not seem to be a bug report for yet.
The problem is that the CamusJob run() method signaturerun(String[] args)
is not supported by the hadoopJavaJob plugin. This results in the following error:
14-08-2014 14:32:44 PDT consume_kafka ERROR - Caused by: java.lang.IllegalArgumentException: Can not create a Path from a null string
14-08-2014 14:32:44 PDT consume_kafka ERROR - at org.apache.hadoop.fs.Path.checkPathArg(Path.java:87)
14-08-2014 14:32:44 PDT consume_kafka ERROR - at org.apache.hadoop.fs.Path.(Path.java:99)
14-08-2014 14:32:44 PDT consume_kafka ERROR - at com.linkedin.camus.etl.kafka.mapred.EtlMultiOutputFormat.getDestinationPath(EtlMultiOutputFormat.java:113)
14-08-2014 14:32:44 PDT consume_kafka ERROR - at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:181)
run()
is invoked by default but it has not received the main.args to load the Camus specific properties, hence when run()
is called it is missing key settings. It seems that the proper behaviour would be to call the Camus run(String[] args)
method in order to have Camus properly initialize.
I am happy to give this a shot if I can get some pointers on how / what to adjust in the hadoopJavaJob plugin.
This issue derives from azkaban/azkaban#247 and azkaban/azkaban#248
I am changing the MailCreator interface API, so I update ReportalMailCreator (https://github.com/azkaban/azkaban-plugins/blob/master/plugins/reportal/src/azkaban/viewer/reportal/ReportalMailCreator.java) to conform to the new interface API.
Sometimes after firing off a run, I realize there is a bug in my code. I would like to kill the running report to avoid wasting resources. Currently, there is no way for me to kill the report from the Reportal UI. We should add this functionality.
This is especially critical now that Reportal only allows you to have one RUNNING execution of each report at a time. Sometimes jobs have OOM errors and hang, causing the flow to remain in the RUNNING state forever. In such a scenario, I will never be able to run my report again, since I have no way of killing the currently RUNNING execution.
Alternatively, we should roll back this commit, or make a new commit so that concurrent execution can be enabled/disabled by a property in the Azkaban conf file.
Tracked by internal JIRA HADOOP-6976.
May be useful to create a Pig 0.12.0 job type since Pig 0.12 is now out.
Tracked by internal JIRA ticket: HADOOP-4414
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.