A fully-featured AWS Athena database driver (+ athenareader https://github.com/uber/athenadriver/tree/master/athenareader)
Home Page: https://uber.github.io/athenadriver
License: MIT License
<catalogName>.<database>.<table>
I found it becomes a little hard to connect from one eks cluster to another account's athena through STS role assumption.
Static credentials may become expired after a priod.
I think it's better to give a closure in driver configuration like 'func() *session.Session' and call it before use ACCESS_KEY and SECRET to get session. This can support all aws sdk's credential types.
I would like to override the PoolInterval in constants.go
If athena takes 1 second to finish we will be waiting for 3 seconds
May be connection can take PollInterval as an argument and use that if provided or default to 3 seconds as it is now
Thanks in advance
Hi!
I have a use case that would greatly benefit from async query support, perhaps added as additional functionality to the connection interface.
For example, introducing a function that will kick off a query and return an object containing QueryExecutionId, then another function that may use that object to poll for query completion, and another for cancelling that query, etc.
If any thought has been given to this I'd love to know.
Thanks!
Currently missing values return an empty string or default zero value https://github.com/uber/athenadriver#missing-value-handling, or an error is raised.
Can an additional configuration option be set to allow the user to allow nil values? Something like Config.SetMissingAsNil(true)? And add IsMissingAsNil() here:
Lines 301 to 308 in 7f6d139
It can be checked here and return nil, nil if driverConfig.IsMissingAsNil() is true:
Line 275 in 7f6d139
With the current options we run into errors or undesirable behaviour. In the case where we set missing values to return a default value, it's not clear to the end user whether a value was missing or if it's actually the default value. This is undesirable because the end user can't differentiate between real data and missing data. In the other case where an empty string is returned, when we Scan the Rows returned we hit a conversion error https://cs.opensource.google/go/go/+/refs/tags/go1.20.4:src/database/sql/sql.go;l=3253 if the type of the column isn't a string.
Let me first thank you for this amazing work, seems like you're deeply invested on this but tbh, it was really hard to find 🤔
I wanted to use my existing bucket but in a different folder, so I added a folder in the bucket and changed the URL to:
conf, _ := drv.NewDefaultConfig("s3://athenaplayground/local/",
it works, but when I went to S3 to check my bucket I noticed this minor bug:
I also tried creating local folder in my bucket first, but the results were in //local
<3
Hi,
I am using this driver with squalus and I am having some issues with serialization.
Regarding reads I cannot read time.Time since I get an error like sql: Scan error on column index 8, name \\"created_at\\": unsupported Scan, storing driver.Value type string into type \*time.Time
Which is weird since the type is time.Time and not *time.Time.
The same seems to happen to int64.
The object being read is:
type myStruct struct {
ID int64 `db:"id"`
//.....
CreatedAt time.Time `db:"created_at"`
//.....
}
Also when creating the query and passing parameters if the type is time.Time it gets serialised as a string instead of a timestamp.
After reading the README I was expecting this driver to support this native type, did I read wrong? What am I doing incorrectly?
$athenareader -b s3://henrywutest/ -q '33fb2309-06e7-4630-b209-80d7a5267ad1'
writing to Athena database is disallowed in read-only mode
Something like cqlsh output
WARNING: DATA RACE
Read at 0x00c0008f0118 by goroutine 33:
github.com/uber/athenadriver/go.(*DriverTracer).Scope()
/home/user/go/pkg/mod/github.com/uber/[email protected]/go/trace.go:99 +0xfe4
github.com/uber/athenadriver/go.(*SQLConnector).Connect()
/home/user/go/pkg/mod/github.com/uber/[email protected]/go/connector.go:112 +0xfb3
database/sql.(*DB).conn()
/home/user/Applications/go/src/database/sql/sql.go:1395 +0xb0a
database/sql.(*DB).query()
/home/user/Applications/go/src/database/sql/sql.go:1732 +0x99
database/sql.(*DB).QueryContext()
/home/user/Applications/go/src/database/sql/sql.go:1710 +0xf9
In some environments it may be preferable for the AWS sdk to determine the credentials instead of having them passed in.
Proposal:
If the AWS_SDK_LOAD_CONFIG environment variable is unset and the access id is not passed, the session should be created with just the region. Credentials will be read by the AWS sdk.
Details:
I am willing to create a pull request for this if approved (and when my employer's legal dept approves).
The README specifies a get_cost command. However there doesn't appear to be any examples, tests, or documentation on how to use this pseudo command.
Hi 👋
I am using this driver but since we have our own implementation to handle AWS connections, I'm just using the Rows struct to parse responses from Athena. In particular, I am using the NewRows function. The problem is that the function requires a *DriverTracer instance but there is no way to create that outside this package.
The methods to create a *DriverTracer are there but they are unexported. Can I send a PR to expose them so I can use NewRows directly?
Thanks!
Hi,
I'm having troubles accessing Athena from a Lambda function. I want to use an AccessKeyID and SecretAccessKey from an IAM User I created, which has the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": [
"arn:aws:s3:::webbeds-input-zone/datalake-api/*",
"arn:aws:s3:::datalake-api/*"
]
},
{
"Sid": "a1",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::aws-athena-query-results-webbeds-datareader"
]
},
{
"Sid": "a2",
"Effect": "Allow",
"Action": [
"iam:CreateAccessKey",
"iam:GetUser",
"iam:ListAccessKeys"
],
"Resource": [
"arn:aws:iam::*:user/${aws:username}"
]
},
{
"Sid": "a3",
"Effect": "Allow",
"Action": [
"athena:StartQueryExecution",
"athena:StopQueryExecution",
"athena:GetQueryResultsStream",
"athena:GetQueryResults",
"athena:GetNamedQuery",
"athena:DeleteNamedQuery",
"athena:ListQueryExecutions",
"athena:ListNamedQueries",
"athena:GetWorkGroup",
"athena:CreateNamedQuery",
"athena:GetQueryExecution",
"athena:BatchGetNamedQuery",
"athena:BatchGetQueryExecution"
],
"Resource": [
"arn:aws:athena:eu-west-1:137693930675:workgroup/webbeds_datareader",
"arn:aws:athena:eu-west-1:137693930675:workgroup/primary"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:GetBucketPublicAccessBlock",
"s3:GetBucketPolicyStatus",
"s3:Get*",
"athena:GetNamespace",
"athena:ListWorkGroups",
"s3:List*",
"s3:GetBucketAcl",
"athena:GetCatalogs",
"athena:GetNamespaces",
"s3:GetAccountPublicAccessBlock",
"s3:ListAllMyBuckets",
"athena:GetExecutionEngine",
"athena:GetExecutionEngines",
"athena:GetTables",
"athena:GetTable",
"s3:GetBucketLocation",
"athena:ListWorkGroups",
"athena:ListDataCatalogs"
],
"Resource": "*"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*",
"s3:*"
],
"Resource": [
"arn:aws:s3:::aws-athena-query-results-webbeds-datareader/*",
"arn:aws:s3:::aws-athena-query-results-*"
]
},
{
"Sid": "VisualEditor3",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:CreateDatabase"
],
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:database/default"
]
},
{
"Sid": "VisualEditor4",
"Effect": "Allow",
"Action": "glue:GetTables",
"Resource": "arn:aws:glue:eu-west-1:137693930675:database/*"
},
{
"Sid": "VisualEditor5",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetPartition",
"glue:GetTables",
"glue:GetPartitions",
"glue:BatchGetPartition",
"glue:GetDatabases",
"glue:GetTable"
],
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:database/default",
"arn:aws:glue:eu-west-1:137693930675:database/webjet_data_lake",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_callcentre",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_callcentre/*",
"arn:aws:glue:eu-west-1:137693930675:table/webjet_data_lake/*",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_searches",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_searches/*",
"arn:aws:glue:eu-west-1:137693930675:table/default",
"arn:aws:glue:eu-west-1:137693930675:table/default/*",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_callcentre/*",
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:table/default/bookings_stream_last",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_masterdb",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_masterdb/*",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_datacc",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_datacc/*"
]
},
{
"Sid": "CreateEditDropViews",
"Effect": "Allow",
"Action": [
"glue:CreateTable",
"glue:UpdateTable"
],
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:database/webjet_data_lake",
"arn:aws:glue:eu-west-1:137693930675:table/webjet_data_lake/*"
]
},
{
"Sid": "VisualEditor9",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::webbeds-input-zone*",
"arn:aws:s3:::webbeds-reporting-zone*",
"arn:aws:s3:::webbeds-master-zone*",
"arn:aws:s3:::webbeds-*-datasource*",
"arn:aws:s3:::webbeds-jactravel-*",
"arn:aws:s3:::webbeds-sunhotels-*",
"arn:aws:s3:::webbeds-five9-logs"
]
},
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "glue:GetTable",
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_analysis/pricescrawler_*",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_analysis"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "glue:BatchCreatePartition",
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_analysis",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_analysis/pricescrawler_*"
]
},
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::webbeds-sapcheck-logs/uploads/reports/*"
}
]
}
As you can see, there's plenty of permissions to execute queries to athena.
Also, I'm unsetting the environment variable AWS_SDK_LOAD_CONFIG by using
os.Unsetenv("AWS_SDK_LOAD_CONFIG")
as the README says.
But, whenever I make a call to athena, the next error message pops up:
I don't know what the problem could be, and this is driving me nuts.
I talked to amazon and they told me that maybe this is something related to this driver.
Thanks a lot in advance.
My athena tables has uuid pattern in query filter, but it goes into QID branch and lead it always failed.
Is there a way to override the athena endpoint like s3 or dynamo can be overridden?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.