franksam007 / sebc Goto Github PK

Shell 100.00%

sebc's Introduction

Services Enablement Boot Camp

This repository contains materials for Cloudera's Services Enablement Boot Camp and Cloudera FCE's internal boot camp.

If you've received this repo as a ZIP file attachment, use the instructions in the email to create your class repo. You can make your GitHub repo Private if you wish, but it is not a free option. For that reason, most students choose to make their GitHub repository public.

NOTE: DO NOT add files to your repository through the GitHub browser interface.

Use GitHub only to receive your lab work by pushing from your local copy. Treat your GitHub repo like a backup to what is on your laptop. Doing so will save a lot of headaches around maintaining the repo, as we will explain in class. The only changes you should make directly to your GitHub copy are Issues and Milestones, which are described in README.md.

Add your instructors as Collaborators to your GitHub repo. Because we have 25 students and 4 people to facilitate labs, we'll give you all the GitHub names you need at the beginning of class. The lead instructor's GitHub name is mfernest.

Adding your instructors as Collaborators will let them create pull requests on your work. This is a way of editing your submissions without changing them directly. When you review instructor edits, you can choose to reject them. Perhaps you would like to correct problems a different way or ignore them for a while. The pull request stands as a record of that interaction.

We use GitHub's Issues feature to establish a workflow around your lab submissions.

For each lab section, such as Installation or Storage, you will use an Issue to track your progress. You will also use GitHub labels to mark the current state of each lab (e.g., submitted, stuck, review). Your instructors will use labels to evaluate your work once you have marked it for review (such as complete or 'incomplete`).

Finally, we will use GitHub milestones to separate your lab work from your challenge work.

In a large class, it may take some time for an instructor to help you with a difficult problem. Use the Issue to describe the problem and show what diagnostic work you've tried to isolate your problem.

You can include error messages or stack traces as Issue comments, or take a screenshot to show your cluster's current condition. It is quite possible these initial steps will help you solve the problem yourself. Otherwise, they will show an instructor what you have tried so far.

Before you start any labs, make the following changes to your GitHub repo:

Add collaborators under Settings -> Collaborators.
Enable Issues under Settings -> Options. Click the Features box and enable Issues.
Click the Issues tab and the Milestones button
- Create two milestones: Labs and Challenges
- Set the deadlines for Thursday and Friday of the current week.
Click the Labels button and change your labels as follows:
- Change bug to stuck
- Change duplicate to started
- Change enhancement to didNotSubmit
- Change help wanted to complete
- Change invalid to review
- Change wontfix to incomplete; set the label color to #fbca04.
- Leave the question Issue as it is

One of the instructors will open an issue on your repo to acknowledge your invitation to collaborate. They will also review your repo for all the settings described here, and point out anything that isn't setup correctly. This issue will be the only one you're asked to close. All other issues are to be closed by an instructor to show no further review is needed.

sebc's People

Contributors

Watchers

Forkers

happyfreeangel

sebc's Issues

Install CM

Integrate hue with sentry

Edit /etc/hue/conf/hue.ini (on CDH)

[libsentry]
  # Hostname or IP of server.
  hostname=localhost
 
  # Port the sentry service is running on.
  port=8038
 
  # Sentry configuration directory, where sentry-site.xml is located.
  sentry_conf_dir=/etc/sentry/conf

Create a group(for example "hue") in HUE, put user that manage security to this group, and put the user in OS group(same as group in hue)
In Sentry, edit /etc/sentry/conf/sentry-site.xml, and put the group in sentry.service.admin.group, or edit with CM.


  sentry.service.admin.group
  hive,impala,hue

In CM, HUE>Configuration>Sentry Service, enable!

Challenge Setup

Error when run hue useradmin_sync_with_unix

Script reports Error:

Error: Password not present
Traceback (most recent call last):
  File "./hue", line 12, in 
    load_entry_point('desktop==3.9.0', 'console_scripts', 'hue')()
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/desktop/core/src/desktop/manage_entry.py", line 59, in entry
    execute_from_command_line(sys.argv)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 261, in fetch_command
    commands = get_commands()
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/management/__init__.py", line 107, in get_commands
    apps = settings.INSTALLED_APPS
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 49, in _setup
    self._wrapped = Settings(settings_module)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/conf/__init__.py", line 128, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/utils/importlib.py", line 40, in import_module
    __import__(name)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/desktop/core/src/desktop/settings.py", line 323, in 
    "PASSWORD" : desktop.conf.get_database_password(),
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/desktop/core/src/desktop/conf.py", line 1353, in get_database_password
    password = DATABASE.PASSWORD_SCRIPT.get()
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 147, in get
    return self.config.get_value(data, present=present, prefix=self.prefix, coerce_type=True)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 263, in get_value
    return self._coerce_type(raw_val, prefix)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 283, in _coerce_type
    return self.type(raw)
  File "/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/hue/desktop/core/src/desktop/lib/conf.py", line 714, in coerce_password_from_script
    raise subprocess.CalledProcessError(p.returncode, script)
subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/720-hue-HUE_SERVER/altscript.sh sec-5-password' returned non-zero exit status 1

Export some env variables to bypass:
export HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=1 export HUE_DATABASE_PASSWORD=123456
Script still reports

Error: Password not present

But it can finish.

Install CDH

Storage labs

Resource Labs

Test HDFS

Integrate Linux accounts with Hue

Installation Lab

Installation lab finished.
Concerning the local repo(the bonus lab), I created a local repo, but I forgot to take the snapshot of the very step and I can't go back after I finish the installation. I took a snapshot of web page of the repo instead, I'm not sure if it is OK.

Install

Enterprise Lab

Security Lab

Client Lab

Welcome to SEBC

It looks like you haven't set up your labels or milestones yet. Please refer to the README.md and README_GitHub.md files for complete instructions.

Once you have finished configuring these elements and @godiswc has accepted your invitation to collaborate, you may close this issue.

HDFS stopped after Kerberos is enabled

I have enabled Kerberos on the cluster, then I started cm services and zookeeper, and they work fine.
When I started HDFS(with HA), all service roles started, but they reported 0 storage capacity, and after a few seconds datanodes went down, and other hdfs services just went down too.
I have checked the datanode log, it said:

org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/[email protected] (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for hdfs/[email protected] to ip-172-31-33-107.us-west-2.compute.internal/172.31.33.107:8022
2017-05-10 13:44:48,783 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: ip-172-31-33-107.us-west-2.compute.internal/172.31.33.107:8022
2017-05-10 13:44:50,017 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/[email protected] (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
2017-05-10 13:44:53,788 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/[email protected] (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
2017-05-10 13:44:54,499 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/[email protected] (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
2017-05-10 13:44:54,733 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/[email protected] (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
2017-05-10 13:44:55,181 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs/[email protected] (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
......

It seems the datanodes can not make connection with the namenode, but I do not know where to look.
I have push a segment of log file and Kerberos configuration file(kdc.conf and krb5.conf) into /SEBC/security/lab, would you please help me out?