Giter VIP home page Giter VIP logo

hdinsighttutorial's Introduction

HDInsightTutorial Using Azure PowerShell

https://github.com/munkhuush92/HDInsightTutorial ##1. Creating HDInsight cluster using Powershell

Prerequisites:

  • Azure subscription or free trial account
    • 30 day free trial

    • Install PowerShell If you have installed Azure PowerShell version 0.9x, you must uninstall it before installing a newer version.

      • Step 1. Make sure run Windows PowerShell as an administrator. Install the latest Azure PowerShell from the PowerShell Gallery using an elevated Windows PowerShell or PowerShell Integrated Scripting Environment (ISE) prompt:
      Install-Module AzureRM
      

      To install and replace over the old version, please enter "A"

      Then, if it is successfully started, it should show this progress:

      progress

      • Step 2. Connect to an Azure account

        • To login Azure account from Azure Shell, type in the shell:
         Login-AzureRmAccount
        
        • After that, it will pop up the browser for you to enter your credentials loginpopup
        • If logged successfully, then it will show your info loginsuccess

##2. To create an HDInsight cluster by using Azure PowerShell, you must complete the following procedures: 1. Create an Azure resource group. 2. Create an Azure Storage account. 3. Create an Azure Blob container. 4. Create an HDInsight cluster.

* Create an Azure resource group:
	* Step 1. Create a variable for group name. 
	```
	$resGroupName = "yourdesiredgroupname"
	```
	
 * Step 2. Using groupname variable to create resource group. Location can be anywhere.

		```
		New-AzureRmResourceGroup -Name $resGroupName -Location "South Central US"
		```
		If created successfully, then it should show the result

		![resourcegroupsuccess](/images/resgroupnamesuccess.png)
		
  * I created a multiple of variable so that I can use it later process. (Short Cut)
	
	Make sure $token is all in lowercase because we will use it for storage name and storage name must be all lower-case
		![resourcegroupsuccess2](/images/rescreated.png)
	
		
* Create an Azure Storage account

	Type the following in PowerShell to create a storage
	
	```
	# Create an Azure Storage account and container used as the default storage
	New-AzureRmStorageAccount `
		-ResourceGroupName $resourceGroupName `
		-StorageAccountName $defaultStorageAccountName `
	-Location $location `
		-Type Standard_LRS
	$defaultStorageAccountKey = (Get-AzureRmStorageAccountKey -Name $defaultStorageAccountName -ResourceGroupName $resourceGroupName)[0].Value
	$destContext = New-AzureStorageContext -StorageAccountName $defaultStorageAccountName -StorageAccountKey $defaultStorageAccountKey
	```
	
If created successfully, it should display below

 ![storagesuccess](/images/storagesuccess.png)
* Create an Azure Blob container. Type in Shell
	```
	New-AzureStorageContainer -Name $defaultStorageContainerName -Context $destContext
```
	
* Create an HDInsight cluster:+
	* Step 1. Creating the credentials.
```
$credentials = Get-Credential -Message "Enter Cluster user credentials" -UserName "admin"
$sshCredentials = Get-Credential -Message "Enter SSH user credentials"
```
 * Step 2. The location of the HDInsight cluster must be in the same data center as the Storage account.

			```
			$location = Get-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -StorageAccountName $defaultStorageAccountName | %{$_.Location}
			```
  * Step 3. Creating the HDInsight Cluster. Please be patient, It can take up to 20 minutes to create a cluster.

```
New-AzureRmHDInsightCluster `
	-ClusterName $clusterName `
	-ResourceGroupName $resourceGroupName `
	-HttpCredential $credentials `
	-Location $location `
	-DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
	-DefaultStorageAccountKey $defaultStorageAccountKey `
-DefaultStorageContainer $defaultStorageContainerName  `
-ClusterSizeInNodes $clusterNodes `
	-ClusterType Hadoop `
-OSType Linux `
	-Version "3.4" `
-SshCredential $sshCredentials
```
If the cluster created successfully, then it will display info of HDInsight Cluster like below
![storagesuccess](/images/clustersuccess.png)

hdinsighttutorial's People

Contributors

munkhuush92 avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.