livebook-dev / req_athena Goto Github PK

We're doing Code.ensure_loaded? at runtime which can be very slow: https://github.com/livebook-dev/req_athena/blob/v0.1.4/lib/req_athena.ex#L392.

I believe an easy fix is in that spot to change:

- Code.ensure_loaded?(:aws_credentials)
+ unquote(Code.ensure_loaded?(:aws_credentials))

However there are some other things to consider:

• we should add it as an optional dep: https://github.com/livebook-dev/req_athena/blob/v0.1.4/mix.exs#L46. Since we worked on this it has been published to Hex: https://hex.pm/packages/aws_credentials
• I don't remember if we discussed this at the time but we probably shouldn't configure lib1 from lib2: https://github.com/livebook-dev/req_athena/blob/v0.1.4/lib/req_athena.ex#L407
• maybe we should only ever call this once? https://github.com/livebook-dev/req_athena/blob/v0.1.4/lib/req_athena.ex#L409. For example, we add a ReqAthena.Application and call it there.

We need to prepare aws_credentials to it's first release (aws-beam/aws_credentials#30)

credential = [...]

Req.new()
|> ReqAthena.attach(credential)
|> Req.post!(athena: {"SELECT ?", [1]})

throws error

** (CaseClauseError) no case clause matching: {nil, false}
    (req_athena 0.1.2) lib/req_athena.ex:188: ReqAthena.handle_athena_result/1
    (req 0.3.5) lib/req/request.ex:705: anonymous fn/2 in Req.Request.run_response/2
    (elixir 1.14.2) lib/enum.ex:4751: Enumerable.List.reduce/3
    (elixir 1.14.2) lib/enum.ex:2514: Enum.reduce_while/3
    (req 0.3.5) lib/req/request.ex:704: Req.Request.run_response/2
    (req 0.3.5) lib/req.ex:327: Req.post!/2
    (req_athena 0.1.2) lib/req_athena.ex:271: ReqAthena.execute_prepared_query/1
    #cell:5eknz4ezrwqs47sqb5n4owthwp2eqjr3:6: (file)

token is also know as AWS Session Token or AWS Security Token

See:

• https://docs.aws.amazon.com/athena/latest/ug/lf-governed-tables-supported-data-types.html
• https://docs.aws.amazon.com/athena/latest/ug/data-types.html

With this option, we can have 3 use cases:

1. Use :workgroup, removing :ResultConfiguration from request and adding :Workgroup, using the output location already configured inside AWS Athena Console
2. Use :output_location and :workgroup to override output location from AWS Athena Console
3. Use only :output_location

We should define a rule with :output_location and :workgroup to define if both are empty ("" or nil), we should raise. If one of them are valid, we should execute one of the use cases.

In this case, we need to somehow force a mechanism to compute a new cache. The simplicity of this option will depend if Athena allows us to force a query. If not, we will have to come up with some scheme within Kino.

Ref: https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html

See https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html#API_GetQueryResults_RequestSyntax