livebook-dev / req_athena Goto Github PK

In this case, we need to somehow force a mechanism to compute a new cache. The simplicity of this option will depend if Athena allows us to force a query. If not, we will have to come up with some scheme within Kino.

See https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html#API_GetQueryResults_RequestSyntax

We're doing Code.ensure_loaded? at runtime which can be very slow: https://github.com/livebook-dev/req_athena/blob/v0.1.4/lib/req_athena.ex#L392.

I believe an easy fix is in that spot to change:

- Code.ensure_loaded?(:aws_credentials)
+ unquote(Code.ensure_loaded?(:aws_credentials))

However there are some other things to consider:

• we should add it as an optional dep: https://github.com/livebook-dev/req_athena/blob/v0.1.4/mix.exs#L46. Since we worked on this it has been published to Hex: https://hex.pm/packages/aws_credentials
• I don't remember if we discussed this at the time but we probably shouldn't configure lib1 from lib2: https://github.com/livebook-dev/req_athena/blob/v0.1.4/lib/req_athena.ex#L407
• maybe we should only ever call this once? https://github.com/livebook-dev/req_athena/blob/v0.1.4/lib/req_athena.ex#L409. For example, we add a ReqAthena.Application and call it there.

Athena returns Parquet, which means we can perform the query and point Explorer.from_parquet to it (probably lazy by default).

Ref: https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html

credential = [...]

Req.new()
|> ReqAthena.attach(credential)
|> Req.post!(athena: {"SELECT ?", [1]})

throws error

** (CaseClauseError) no case clause matching: {nil, false}
    (req_athena 0.1.2) lib/req_athena.ex:188: ReqAthena.handle_athena_result/1
    (req 0.3.5) lib/req/request.ex:705: anonymous fn/2 in Req.Request.run_response/2
    (elixir 1.14.2) lib/enum.ex:4751: Enumerable.List.reduce/3
    (elixir 1.14.2) lib/enum.ex:2514: Enum.reduce_while/3
    (req 0.3.5) lib/req/request.ex:704: Req.Request.run_response/2
    (req 0.3.5) lib/req.ex:327: Req.post!/2
    (req_athena 0.1.2) lib/req_athena.ex:271: ReqAthena.execute_prepared_query/1
    #cell:5eknz4ezrwqs47sqb5n4owthwp2eqjr3:6: (file)

We need to prepare aws_credentials to it's first release (aws-beam/aws_credentials#30)

With this option, we can have 3 use cases:

1. Use :workgroup, removing :ResultConfiguration from request and adding :Workgroup, using the output location already configured inside AWS Athena Console
2. Use :output_location and :workgroup to override output location from AWS Athena Console
3. Use only :output_location

We should define a rule with :output_location and :workgroup to define if both are empty ("" or nil), we should raise. If one of them are valid, we should execute one of the use cases.

token is also know as AWS Session Token or AWS Security Token

See:

• https://docs.aws.amazon.com/athena/latest/ug/lf-governed-tables-supported-data-types.html
• https://docs.aws.amazon.com/athena/latest/ug/data-types.html