Error with http call about digdag HOT 5 CLOSED

Hi,

I noticed you configured method:POST in yaml file but HTTP error is happening with GET method.

Sending HTTP request: GET https://***

I'm not sure if the URL is same with what you configured in yaml file but I guess your access is redirected to another URL.
If your application needs complex access control, it might be better to use py> or rb> operators instead of http> operator.
https://docs.digdag.io/operators/scripting.html

This is an example coming from my workflow.

• yaml definition

+get_data_from_api:
  py>: test.send_http_request
  _env:
    PYTHONPATH: ./scripts
    api_token: ABCDEFG

• scripts/test.py

import requests
import backoff

MAX_RETRIES = 10
RETRY_INTERVAL_FACTOR = 15  # seconds
REQUEST_TIMEOUT = 30  # seconds

@backoff.on_exception(backoff.expo, Exception, max_tries=MAX_RETRIES, factor=RETRY_INTERVAL_FACTOR)
def send_http_request(api_token):
    url = f'https://example.com/path/to/somewhere'
    query = f'SELECT * FROM example_table'
    headers = {
        'Authorization': f'Bearer {api_token}'
    }
    try:
        r = requests.get(url, timeout=REQUEST_TIMEOUT, headers=headers, params={'query': query})
        r.raise_for_status()
        return r.json()
    except requests.exceptions.HTTPError as e:
        print('HTTP Error: ', e)
    except requests.exceptions.RequestException as e:
        print('HTTP connection failure: ', e)
    except Exception as e:
        print('Unknown error', e)

from digdag.

Hello, @makoslokos

Have you ever tried http tool like curl command?
You can debug -l debug option like digdag run -l debug config.yml.

It shows http request like

+disp_current_date:
  http>: http://localhost:8080/
  headers:
    - hoge: hogehoge

2023-04-16 00:16:05 +0900 [DEBUG] (HttpClient@548755293-19) org.eclipse.jetty.client.HttpSender: Request headers HttpRequest[GET / HTTP/1.1]@41ef043f
Accept-Encoding: gzip
User-Agent: Digdag/0.10.4 Jetty/9.3.z-SNAPSHOT
hoge: hogehoge
Host: localhost:8080

from digdag.

Yes, i have tried. In general the url is accessible, this is not the case. Is suspect that the problem is that if during WF execution appear some warning (e.g. too many subtasks) the http clause losts somehow credentials to call url and appears the error of 4xx.

from digdag.

If Digdag send the same request normal and error case, (you can check -l debug)
Is there any possibility server-side issue? (ie. Close/Expire session due to too many requests or retry.)

from digdag.

Ok, will try. Thanks

from digdag.