azkaban / azkaban-plugins Goto Github PK

The idea behind this change is to make it easier to search for those long running reportal job from command line.

This fix will add two JVM args and they look something like this:

-Dreportal.user.name=hreportal -Dreportal.execid=72

• Add 20px margin to bottommost box
• 25% width for Job Type key cell
• Display placeholder when no stats are available

The HDFS tab shows the local file system, not the Hadoop File System. Where would I change "file:///" to "hdfs://" ?

Similiar to azkaban core issue #224, the included links should be passed on downstream. This only affects Hadoop specific jobtypes

Internal JIRA: HADOOP-4646

when i clone the repos, it always throw

 fatal: cannot create directory at 'plugins/jobtype/jobtypes/hive-0.8.1/hive-0.8.
1/aux': Invalid argument

it seems aux is not a valid name in windows environment?

thx

Other job types already do this.

Sometimes after firing off a run, I realize there is a bug in my code. I would like to kill the running report to avoid wasting resources. Currently, there is no way for me to kill the report from the Reportal UI. We should add this functionality.

This is especially critical now that Reportal only allows you to have one RUNNING execution of each report at a time. Sometimes jobs have OOM errors and hang, causing the flow to remain in the RUNNING state forever. In such a scenario, I will never be able to run my report again, since I have no way of killing the currently RUNNING execution.

Alternatively, we should roll back this commit, or make a new commit so that concurrent execution can be enabled/disabled by a property in the Azkaban conf file.

Tracked by internal JIRA HADOOP-6976.

Today while checking out this project through subversion I got a strange error.
svn: E204899: Unable to make directories
This is because of below directory.
azkaban-plugins-2.1/plugins/jobtype/jobtypes/hive-0.8.1/hive-0.8.1/aux
After some research on google I found that on Windows platform a directory/file can't be created with name "aux".
Reference - http://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words.

The jobtype and hadoopsecuritymanager jars are not being copied into the directory for the new Hive job type, which results in a ClassNotFound exception when the executor server starts.

This is the root cause for azkaban/azkaban#173

Previous versions were compiled with 1.7

As mentioned in #114, the ParquetFileViewer tries to view every file as azkaban, meaning it can only view files owned by azkaban, with the group set to azkaban, or world-readable. This means users will not be able to view their Parquet files through the HDFSViewer unless their Parquet files are world-readable.

The code that did this got deleted in this commit - 906b938 - but it's actually necessary, so we should add it back.

• Display placeholder if Pig Visualizer has nothing to visualize.
• Fix Auto Pan Zoom and Reset Pan Zoom buttons

When a report (flow) is in the PREPARING state, the startTime is -1, and the Reportal UI renders this as "1969-12-31 15:59:59", which confuses and alarms users. Instead, we should leave the start time blank or display a "Not Started" message.

Tracked by internal JIRA HADOOP-6960.

Here - https://github.com/azkaban/azkaban-plugins/blob/master/plugins/reportal/src/azkaban/viewer/reportal/ReportalTypeManager.java#L58-59 - -Dreportal.user.name should be set to ${reportal.execution.user}, not userName. We want the property to show who executed the report, not who last edited the report.

Update HadoopSecurityManager_H_2_0 security manager to work with Hadoop 2.3.x.

In log-data.js, the job_id regexes expect job_<12 digits>_<4+ digits>. However, on Hadoop 2, instead of <12 digits> of the form YYYYMMDDHHMM, the job_id appears appears to contain the milliseconds since epoch, which is variable in length and is currently 13 digits.

Also, in Hadoop 2, the job URL printed out in the logs does not contain job_ (which the current url regex is looking for) but instead looks something like http://<host>:<port>/proxy/application_<milliseconds_since_epoch>_<counter>.

We should fix the regexes so that the job summary plugin will find the job ids and URLs.

Tracked by internal JIRA HADOOP-6977.

We display one cell per row instead of one row per row.
Internal JIRA: HADOOP-4463

The problem is that ReportalHiveRunner uses an interface method that has been removed:
https://github.com/azkaban/azkaban-plugins/blob/master/plugins/reportal/src/azkaban/jobtype/ReportalHiveRunner.java#L105

May be useful to create a Pig 0.12.0 job type since Pig 0.12 is now out.

Tracked by internal JIRA ticket: HADOOP-4414

hive-0.8.1/aux/lib should have the antlr runtime. I used version 3.0.1 and that worked. I don't know what is latest.
antlr-runtime-3.0.1.jar

We reduced the limit for the lines of file displayed from 1000 to 100. The limit should be increased or made adjustable.

Tracked by internal JIRA: HADOOP-4479

The Hive plugin works with no user.to.proxy or proxy.user set anywhere. It probably defaults to the user that runs the azkaban executor daemon.

If Pig/hadoopJava did this, Azkaban security would be very easy for small shops (like us) who do not use Hadoop security features.

Tracked by internal JIRA HADOOP-4744.

Companion to azkaban/azkaban#152

Due to the CSS change for azkaban/azkaban#157, the sidebar and graph views for the Pig Visualizer graph no longer displays correctly.

Since we added spaces to the HDFS viewer breadcrumb, it is no longer possible to just copy and paste the path.

Either:

• Remove the spaces
• Create a textbox when clicking on the header or a button

This issue derives from azkaban/azkaban#247 and azkaban/azkaban#248

I am changing the MailCreator interface API, so I update ReportalMailCreator (https://github.com/azkaban/azkaban-plugins/blob/master/plugins/reportal/src/azkaban/viewer/reportal/ReportalMailCreator.java) to conform to the new interface API.

Counterpart to azkaban/azkaban#249

The blue loading bar remains forever.

The BinaryJSON file viewer seems to be unstable. At times, it fails to display a file at all and at other times, it dumps binary junk.

Tracked by internal JIRA: HADOOP-4478

Display contents of file and schema in separate tabs.

Tracked by internal JIRA ticket: HADOOP-4413

When a MapredLocalTask executes, it launches "hadoop jar ...", but the jar it specifies is sometimes the wrong jar. It sometimes erroneously uses hive-common instead of hive-exec.

The problem is with https://github.com/azkaban/azkaban-plugins/blob/master/plugins/jobtype/src/azkaban/jobtype/hiveutils/HiveQueryExecutorModule.java#L36. We should use new HiveConf(SessionState.class) instead.

Tracked by internal JIRA HADOOP-5465.

For ETL distcp use cases which need to copy files from multiple clusters, the Azkaban security manager needs to retrieve the HDFS tokens from other cluster before the file copying process can start.

When running Azkaban on a grid with Hadoop security enabled, viewing a file in one's user directory results in a permission denied error.

I have root-caused this to the fact that the Parquet file viewer does not use the FileSystem object passed in from the HdfsBrowserServlet, which is properly set up to doAs the current user logged rather than the azkaban user. As a result, the Parquet file viewer ends up trying to view the file as azkaban, throwing the AccessControlException. Currently, AvroParquetReader does not have an API that lets one pass in a FileSystem object. The fix for now is to remove the catch AccessControlException block from the Parquet file viewer.

Tracked by internal ticket: HADOOP-5350

The HadoopSecurityManager_H_1_0 class expects to find a property 'user.to.proxy'. I have placed this in every configuration file and .job file, and nothing has worked.

Which file should this property be in?

Here is the full log section for this attempt at running the java-wc job. HadoopSecurityManager_H_1_0 is clearly trying to pass in a 'user.to.proxy' property which is not there.

2013/08/02 03:08:05.255 +0000 INFO [pig-upload] [Azkaban] Need to proxy. Getting tokens.
2013/08/02 03:08:05.255 +0000 INFO [pig-upload] [Azkaban] Getting hadoop tokens for apxqueue
2013/08/02 03:08:05.255 +0000 INFO [HadoopSecurityManager] [Azkaban] proxy user apxqueue not exist. Creating new proxy user
2013/08/02 03:08:05.258 +0000 INFO [pig-upload] [Azkaban] Getting DFS token from 10.176.235.204:8020hdfs://ip-10-176-235-204.us-west-1.compute.internal:8020
java.lang.NullPointerException
at org.apache.hadoop.security.SecurityUtil.setTokenService(SecurityUtil.java:246)
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:408)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:571)
at azkaban.security.HadoopSecurityManager_H_1_0$2.getToken(HadoopSecurityManager_H_1_0.java:268)
at azkaban.security.HadoopSecurityManager_H_1_0$2.run(HadoopSecurityManager_H_1_0.java:258)
at azkaban.security.HadoopSecurityManager_H_1_0$2.run(HadoopSecurityManager_H_1_0.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1122)
at azkaban.security.HadoopSecurityManager_H_1_0.prefetchToken(HadoopSecurityManager_H_1_0.java:253)
at azkaban.jobtype.HadoopPigJob.getHadoopTokens(HadoopPigJob.java:166)
at azkaban.jobtype.HadoopPigJob.run(HadoopPigJob.java:102)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:379)
at azkaban.execapp.JobRunner.run(JobRunner.java:280)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
azkaban.security.commons.HadoopSecurityManagerException: Failed to get hadoop tokens! nullnull
at azkaban.security.HadoopSecurityManager_H_1_0.prefetchToken(HadoopSecurityManager_H_1_0.java:318)
at azkaban.jobtype.HadoopPigJob.getHadoopTokens(HadoopPigJob.java:166)
at azkaban.jobtype.HadoopPigJob.run(HadoopPigJob.java:102)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:379)
at azkaban.execapp.JobRunner.run(JobRunner.java:280)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013/08/02 03:08:05.261 +0000 ERROR [pig-upload] [Azkaban] Job run failed!
2013/08/02 03:08:05.261 +0000 ERROR [pig-upload] [Azkaban] Failed to get hadoop tokens! nullnullnull
2013/08/02 03:08:05.261 +0000 INFO [pig-upload] [Azkaban] Finishing job pig-upload at 1375412885261
2013/08/02 03:08:05.267 +0000 INFO [wordcount-java] [Azkaban] Job Finished pig-upload with status FAILED
2013/08/02 03:08:05.278 +0000 INFO [wordcount-java] [Azkaban] Killing wordcount-java due to prior errors.
2013/08/02 03:08:05.289 +0000 INFO [wordcount-java] [Azkaban] Finishing up flow. Awaiting Termination
2013/08/02 03:08:05.289 +0000 INFO [wordcount-java] [Azkaban] Setting flow status to Failed.
2013/08/02 03:08:05.289 +0000 INFO [wordcount-java] [Azkaban] Flow is set to FAILED
2013/08/02 03:08:05.289 +0000 INFO [wordcount-java] [Azkaban] Setting end time for flow 8 to 1375412885289
2013/08/02 03:08:05.305 +0000 INFO [FlowRunnerManager] [Azkaban] Flow 8 is finished. Adding it to recently finished flows list.
2013/08/02 03:10:01.532 +0000 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished
2013/08/02 03:10:01.532 +0000 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 8 from recently finished flows list.

I am filing this issue because I encountered the same issue described at https://groups.google.com/d/msg/azkaban-dev/S9G9Lqmfm1Q/7pV0P7Re820J but there does not seem to be a bug report for yet.

The problem is that the CamusJob run() method signaturerun(String[] args) is not supported by the hadoopJavaJob plugin. This results in the following error:

14-08-2014 14:32:44 PDT consume_kafka ERROR - Caused by: java.lang.IllegalArgumentException: Can not create a Path from a null string
14-08-2014 14:32:44 PDT consume_kafka ERROR -   at org.apache.hadoop.fs.Path.checkPathArg(Path.java:87)
14-08-2014 14:32:44 PDT consume_kafka ERROR -   at org.apache.hadoop.fs.Path.(Path.java:99)
14-08-2014 14:32:44 PDT consume_kafka ERROR -   at com.linkedin.camus.etl.kafka.mapred.EtlMultiOutputFormat.getDestinationPath(EtlMultiOutputFormat.java:113)
14-08-2014 14:32:44 PDT consume_kafka ERROR -   at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:181)

run() is invoked by default but it has not received the main.args to load the Camus specific properties, hence when run() is called it is missing key settings. It seems that the proper behaviour would be to call the Camus run(String[] args) method in order to have Camus properly initialize.

I am happy to give this a shot if I can get some pointers on how / what to adjust in the hadoopJavaJob plugin.

This is to prevent scheduled reports with repeating from hogging Azkaban server. One use case we saw was there were many running instances of the same report because this report takes a while to complete and it is configured to repeat every minute.

Every path in viewer.external.classpath will be prefixed with plugin's dir.

Due to azkaban/azkaban#255, package names of some classes have been change. For example, azkaban.webapp.AzkabanServer is now azkaban.server.WebServer.

The new Azkaban 2.6.1 JARs need to be imported so that the plugins can compile against the JARs with the new packages to be compatible with the new Azkaban core.

There is no need to add -Dreportal.execid=${azkaban.flow.execid} here - https://github.com/azkaban/azkaban-plugins/blob/master/plugins/reportal/src/azkaban/viewer/reportal/ReportalTypeManager.java#L59 - because Azkaban's JobRunner already inserts it here - https://github.com/azkaban/azkaban/blob/master/azkaban-execserver/src/main/java/azkaban/execapp/JobRunner.java#L536-539

Example, if I try to go to /hdfs/user/bob, I see the "Permission denied" error message. However, if I try to go to /hdfs/user/bob/foo, I see a 500 page.

The problem is that HadoopSecureHiveWrapper uses an interface method that has been removed: https://github.com/azkaban/azkaban-plugins/blob/master/plugins/jobtype/src/azkaban/jobtype/HadoopSecureHiveWrapper.java#L155

Companion to azkaban/azkaban#167