hramatskiu / shim-config-loader-start Goto Github PK

2017-12-22 12:48:42,389 57127 INFO [com.epam.shim.configurator.Krb5Configurator] (Thread-4:) Start download krb5 from cluster! 2017-12-22 12:48:45,354 60092 INFO [com.epam.loader.common.util.FileCommonUtil] (Thread-4:) Save - /home/devuser/dist/config-test/kerberos/krb5.conf 2017-12-22 12:48:45,356 60094 INFO [com.epam.shim.configurator.Krb5Configurator] (Thread-4:) Successfully copied to + /home/devuser/dist/config-test/kerberos/krb5.conf 2017-12-22 12:48:45,357 60095 INFO [com.epam.shim.configurator.Krb5Configurator] (Thread-4:) Start copy krb5 to JAVA_HOME! 2017-12-22 12:48:45,413 60151 INFO [com.epam.loader.common.util.FileCommonUtil] (Thread-4:) Save - /etc/krb5.conf 2017-12-22 12:48:45,452 60190 INFO [com.epam.shim.configurator.util.CopyDriversUtil] (Thread-4:) Impala Simba driver already exists in the shim folder 2017-12-22 12:48:45,506 60244 INFO [com.epam.shim.configurator.util.CopyDriversUtil] (Thread-4:) MySQL driver already exists in the destination folder 2017-12-22 12:48:45,507 60245 INFO [com.epam.shim.configurator.util.CopyDriversUtil] (Thread-4:) MySQL driver already exists in the destination folder
Where krb5.conf exactly copied?

In Linux when configure mapr client check-box is checked mapred-site.xml & yarn-site.xml are copied to
mapr client & shim. Actually we need these xml files only in one place mapr client

property "linux.classpath" is set to value "/home/mapr/Pentaho/design-tools/data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations/mapr60/,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/,/opt/mapr/lib/kvstore.jar,/opt/mapr/lib/libprotodefs.jar,/opt/mapr/lib/baseutils.jar,/opt/mapr/lib/maprutil.jar,/opt/mapr/lib/json-1.8.jar,/opt/mapr/lib/flexjson-2.1.jar,/opt/mapr/lib,/home/mapr/Pentaho/design-tools/data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations/mapr60" in file "/home/mapr/Pentaho/design-tools/data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations/mapr60/config.properties"

hbase-site.xml - should be copied only in shim folder
hive-site.xml - should be copied only in shim folder

first in line =)

Paths in hadoop classpath for Windows should be started with "file:///C" and NOT with "file:///c"

<property> <name>hadoop.spoofed.user.uid</name> <value>5000</value> </property> <property> <name>hadoop.spoofed.user.gid</name> <value>5000</value> </property> <property> <name>hadoop.spoofed.user.username</name> <value>mapr</value> </property>
This is not needed in linux

On main form "Kerberos" pair of fields exist. For EMR clusters used hadoop user, but later for 'pentaho.authentication.default.kerberos.principal' value is populating as "devuser@.....". It will be better to use the very same user as for kerberos authentication during config loading, and for kerberos principal in config.properties file.
Attachment: https://i.imgur.com/1CcdYLD.png

Create README file with instructions for using this tool.

Here is the list:

<property>
  <name>mapreduce.app-submission.cross-platform</name>
  <value>true</value>
</property>

--------------- The above property is necessary to run mapred apps on windows

<property>
<name>mapreduce.map.memory.mb</name>
<value>4000</value>
</property>

--------------- This property is usually necessary coz we often face the lack of resources

<property>

  <name>fs.default.name</name>
  <value>maprfs:///</value>
 </property>

It's necessary to apply this if we want "green" cluster test =)

Linux.classpath for mapr6.0 misses /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop, PMR jobs are failed for that