Comments (13)
This had consistently failed for me, whether I provide a storageKey, or used azureAuthenticate(sc) to get a key. Valid values are placed in the "" for each param. I followed your steps, and added a setAzureContext() step, ran that, then ran the azureListStorageBlobs(), and it then worked.
> sc <- createAzureContext(tenantID = "", clientID = "", authKey = "")
> azureAuthenticate(sc)
[1] "Authentication Suceeded : Key Obtained"
> azureListStorageBlobs(sc, resourceGroup=" ", storageAccount="", container=" ")
Fetching Storage Key..
Fetching Storage Key..
Error in azureListStorageBlobs(sc, resourceGroup = "", :
Error: No storageKey provided: Use storageKey argument or set in AzureContext
OR
> sc <- createAzureContext(tenantID = "", clientID = "", authKey = "")
> azureListStorageBlobs(sc, resourceGroup=" ", storageAccount="", storageKey=”” container=" ")
Fetching Storage Key..
Fetching Storage Key..
Error in azureListStorageBlobs(sc, resourceGroup = "", :
Error: No storageKey provided: Use storageKey argument or set in AzureContext
from azuresmr.
Thanks for raising the issue.
I'm struggling to repo it. Is this also Intermittent for you?
Can you check the storageKey value in Azure context before and after azureListStorageBlobs using str(sc). I'm also interested in any functions you are calling before running azureListStorageBlobs that maybe setting the key to NULL.
IF you are running createAzureContext followed directly by azureListStorageBlobs, then it maybe something environmental. Try running
azureSAGetKey(sc, resourceGroup="YYY",storageAccount="XXXX", verbose=TRUE)
Thanks Alan
from azuresmr.
When I run createAzureContext(), and then run azureAuthenticate(sc), it says a Key is obtained, but running str(sc), there is no entry for storageKey; so I guess its NULL?
I can see a storageKey if I run the setAzureContext() with the storageKey explicitly stated as a param. str(sc) verifies it is there.
So, creating a context, authenticating, and attempting azureListStorageBlobs() does not work. Creating a context, and specifying a storageKey in the azureListStorageBlobs() does not work. This happens every time. It isn't until I run setAzureContext() with a storageKey param that this will work.
I'm not running anything else in-between these commands, so I don't think anything could be 'nullifying' the storageKey.
from azuresmr.
@chrislill Chris I have tired your code and it worked for me. Can you try azureSAGetKey
to see whether you can get a valid storage key? Guess problem was from there.
@Alanwe Function of azureSAGetKey
does not have an error handle after POST - the request may fail but the process will continue and that is why a NULL is returned!
@kjohnson4 In createAzureContext()
there are no option to set service-specific parameters like storageKey
. You need to specify these parameters either in setAzureContext
or the actual functions, i.e., azureBlobLS
or aureListStorageBlobs
.
from azuresmr.
@yueguoguo See the second example in my first post, where I tried providing a key in azureListStorageBlob, and it still failed. Just tried it again, and it fails with same message.
from azuresmr.
@kjohnson4 I was wondering if your Storage Account was created in classic mode. At the moment it only supports the newer API. I think Chris is right that the error handling in in azureSAGetKey needs tightening.
from azuresmr.
I'll have to check if the SA was created in classic, but all our resources are provisioned via Resource manager now, and resources that were in classic have been moved.
I can successfully perform the following operations; would I be able to do these if this was classic storage?:
azureBlobLS(sc, directory="mydir", storageAccount="mystorage", container="mycontainer")
rawBlob <- azureGetBlob(sc, blob = "somefileblob", type = "raw")
txtBlob <- azureGetBlob(sc, blob = "somefileblob", type = "text")
from azuresmr.
@kjohnson4 No if it were classic I don't think they would work either.
One point of note relating to @yueguoguo comment you should never have to actually provide a storageKey as the API should fetch the key automatically.
This example may help, it would be good to know if it works for you.
sc <- createAzureContext(tenantID=MYTID,clientID=MYCID,authKey=MYKEY)
azureCreateResourceGroup(sc,"myResourceGroup","northeurope") # Create Resource Group
azureCreateStorageAccount(sc,storageAccount="myazuresmrteststorage",resourceGroup = "myResourceGroup")
# Wait for the storage account to provision
azureListSA(sc,resourceGroup = "myResourceGroup")
azureCreateStorageContainer(sc,"myteststorage")
library(RCurl)
flightfile <- getURL("https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/244708/Home_Office_Air_Travel_Data_2011.csv")
azurePutBlob(sc,contents=flightfile,blob = "flightinfo/flightfile.csv")
azureBlobCD(sc,"/flightinfo")
azureBlobLS(sc)
azureDeleteBlob(sc,blob="flightfile.csv")
azureDeleteResourceGroup(sc,resourceGroup = "myResourceGroup")
But I would love to get to the bottom of your issue.
Thanks Alan
from azuresmr.
I skipped the first 3 lines (createResourceGroup and CreateStorageAccount) because I dont have permissions to create RGs, and due to AzureSMR has only been setup for authorization on the one SA that I've been using currently ( with Read&Write access).
However, all the other commands, after your "Wait for the storage account to provision" line worked just fine! I was able to verify the blob using AzureStorageExplorer.
I'll continue commenting on other issue opened for my continued issues trying to use azurePutBlob.
from azuresmr.
@kjohnson4 Yea your second use was correct. The cause might be invalid account, key, or even typos. As azureSAGetKey
function does not handle error, we might not have been aware of errors when there are. :)
To create resource group, you need to add access at subscription level. It can be done exactly the same as what you did with access control under a resource group.
- Go to Azure portal.
- Find subscription on the left pane. If there is no, click More services.
- Add access of your Active Directory Application to the subscription (all the same as resource group).
Hope helps!
from azuresmr.
Thanks for steps to setup permissions at subscription level, however our IT team does not intend to provide this service that level of access; my use-case only requires blob storage read/write. The creation of resources falls outside this process and something I don't intend to do with this package.
from azuresmr.
Thanks for the feedback, The Azure Systems Management API is indeed different from the Azure Storage API and shouldn't need any AD access. I will take your use case and look to ensure the storage functions can work effectively without having to access the Systems Management API.
from azuresmr.
@chrislill Can you please test again using the latest version (v0.2.4)?
All of the functions now perform error checking code when Azure responds, so if an error happens at least AzureSMR
should tell you what the problem is.
from azuresmr.
Related Issues (20)
- Undefined index_resource in azureDataConsumption()
- install_github failing HOT 1
- Issue authenticating with tutorial HOT 5
- Can't read/write to blob storage HOT 2
- azureCreateStorageAccount fails while storage account does not exist HOT 1
- license HOT 1
- Extracting an excel file with multiple tabs from Azure to R HOT 1
- Read .gz file from Data Lake HOT 1
- R 3.5 Support
- download more than 5000 blobs HOT 1
- Writing dataframe to blob with AzurePutbBlob HOT 5
- Error trying to write to Blob
- azureDataLake functions don't work HOT 2
- AzurePutBlob should take contenttype as an argument
- DeviceCode authentication doesn't work HOT 2
- resourceGroup should be optional when providing a storage key
- Use HTTPS when communicating with Storage API
- Storing raw objects does not work due to invalid authorisation signature HOT 1
- Support ADLS as alternative to storage account in HDI HOT 1
- Authentication Rmd outdated
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azuresmr.