devoinc / python-sdk Goto Github PK

Add the ipAsString parameter in the Query API calls, following this: https://docs.devo.com/space/latest/95128301/Running+queries+with+the+Query+API

• Check current behavior
• Add the parameter in the module usage
• Add the parameter in the CLI
• Add tests
• Check if there are any other parameters worth adding

It seems that we are receiving a string containing some byte that falls in the surrogate range (\uD83D) and when it is passed to the Python SDK to send the string to Devo, the latter will encode it to UTF-8 and fail. The cause may be (guessing here) that the text is in UTF-16, which uses 2 ranges of surrogates:

• U+D800—U+DBFF (1,024 code points): high surrogates
• U+DC00—U+DFFF (1,024 code points): low surrogates

There's not much information about the logs causing this, we only have this example. It may be Chinese characters or even emojis. We'd like some advice or suggestions about how to proceed with this. Checking if every string is UTF-16 before sending seems overkill and will drop the performance.

Suggestion: we need to have the compatible solution implemented in the DevoSDK (Python) with the option replace or backslashreplace.

Hi,

We have the next issue with one lookup upload: We upload a lookup with one field with "int" type and it shows up as int8 (the domain doesn't matter, because we have tried to upload that same lookup to devo_101 and telefonicasistemas domains). The problem is that we have to upload that lookup with int4 type because that might interfere with mata/mafia parsers, but it keeps showing up as int8.

Greetings.

Some dependencies need to be updated:

• "pyopenssl~=24.0",
• "pytz~=2024.1",
• "certifi~=2024.2.2",
• "cryptography~=42.0.2",
• "pipdeptree~=2.13.2"

Remove urllib from dependencies as it is used by requests, not explicitly by the SDK.

Please comment the following line as "ParseError" is not being used anywhere and causing the error while using the API.

from parser import ParserError

Thanks
Raghu

I am experiencing this same error when using the sdk for a query call. I just started using the package today 3/22/2023 so I assume I have the newest version in which this issue was supposed to be fixed (#186) but just wanted to confirm and make aware that this issue is still appearing.

Check CodeQL results and how the code can be refactored.

The Snyk job seems to be running ok, but no file is generated with testing dependencies. It is for the static code test.

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Isolate the tests so unit tests are differentiated from integration or end-to-end tests.

Start using Pytest instead of the script run_tests.py.

• Reorganise tests and differentiate unit tests from integration tests.
• Consider including pytest to run the tests.

Hello,

We have a process using devo-sdk that sends events from on-demand-queries via e-mail. This process uses an very old version of devo-sdk. Now, we want to actually inject events on DEVO from that same process so we can configure alarms, but the way we use to use the devo.sender doesn't work with that old version (because we are always using the latest version of devo-sdk).

The question we have is: How to use the e-mail sending function with the latest version of devo-sdk, since it doesn't work with the way it's documented on github.

Thank you.

By checking the Lookup docs, I can see the following parameters recommendation for the Lookup.send_headers method:

If we check the Lookup.send_headers method:

We can see that the key parameter has the default "KEY" value. If we now check the method's first call, Lookup.list_to_headers:

We can see that the initial out will be always assigned since key is the first evaluated variable, leaving key_index unusable when the call comes from the initial method.

All this means that a snippet like this one:

headers = ["col1", "col2", "col3"]
fields = ["a", "b", "c"]
lookup.send_headers(headers=headers, key_index=0, event="START", action="FULL")
lookup.send_data_line(key_index=0, fields=fields)
lookup.send_headers(headers=headers, key_index=0, event="END", action="FULL")

Will create a lookup like this one:

Which is not the expected/desired behaviour and I believe should be fixed.

Here are the solutions I can propose:

1. Remove the "KEY" default argument → that might introduce some breaking changes; we do not know if someone is using this method expecting this behaviour.
2. Switch the Lookup.list_to_headers condition order → apparently this approach is less harmful and would not require changing any method's signature but I am unsure if this will work or not.

[EXTRA] Besides that, I also found that in Lookup.process_fields we are deleting the key's field:

fields = ["a", "b", "c"]
lookup.send_data_line(key_index=0, fields=fields)
logger.debug(f"sent {item}")

Would print ["b", "c"] only.

Review documentation and examples to change URL to ADDRESS. This was changed long ago but there are some discrepancies, like this in docs/api/api.md

  {
    "api": {
      "key": "MyAPIkeytoaccessdevo",
      "secret": "MyAPIsecrettoaccessdevo",
      "url": "https://apiv2-us.devo.com/search/query"
    }
  }

json example 2:

  {
    "api": {
      "auth": {
        "key": "MyAPIkeytoaccessdevo",
        "secret": "MyAPIsecrettoaccessdevo",
      },
      "address": "https://apiv2-us.devo.com/search/query"
    }
  }

HTTP API errors when performing queries on the V2 api return an empty string. It would be preferable if an exception was raised, or some actionable data was returned.

This has been verified with both HTTP 404 and 503 return codes. The problem can be reproduced by performing an API request with invalid credentials, or an invalid URI path.

Right now, the Lookup.clean_field method wraps the field's content in double quotes, unless it is a number:

However, it does not check the existence of double quotes present in the field's value itself. This leads to a problem in which a code like this:

lookup.send_headers(headers=['KEY', 'VALUE'], key='KEY', event='START', action="INC")
lookup.send_data_line(key="11", fields=["11", 'double quote: "'])
lookup.send_headers(headers=['KEY', 'VALUE'], key='KEY', event='END', action="INC")

Makes the update process fail silently, without any notice from the client's perspective.

I think we need to ensure that double quotes are escaped by replacing them like this:

field.replace('"', '""')

Or at least inform in the documentation that double quotes need to be escaped with two double quotes.

Add code of conduct

Dependendies to update:

• update urllib3 requirement from ~=1.26.5 to ~=2.0.6
• update responses requirement from ~=0.22.0 to ~=0.23.3
• update pyopenssl requirement from ~=23.0 to ~=23.2
• update pytz requirement from ~=2019.3 to ~=2023.3
• bump pyyaml from 6.0 to 6.0.1

devo-api query does not allow to use "DEVO_API_TOKEN" from env var; does not work, even "--env" argument was set for use env vars as configuration:

( export DEVO_API_TOKEN="d41d8cd98f00b204e9800998ecf8427e" ; devo-api query \
--address "https://apiv2-es.devo.com/search/query" \
--env \
--no-stream \
--from "yesterday()" \
--to "today()" \
--query 'from my.app.foo.bar select * limit 1' \
--response csv )

Traceback (most recent call last):
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/devo/api/client.py", line 352, in _make_request
    headers=self._get_headers(payload),
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/devo/api/client.py", line 450, in _get_headers
    raise DevoClientException(_format_error(ERROR_MSGS['no_auth']))
devo.api.client.DevoClientException: {"msg": "Error Launching Query", "status": 500, "object": "Client dont have key&secret or auth token/jwt"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/logtrust/python3/venv_tf_kpis/bin/devo-api", line 8, in <module>
    sys.exit(cli())
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/devo/api/scripts/client_cli.py", line 84, in query
    process_response(reponse, config)
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/devo/api/scripts/client_cli.py", line 105, in process_response
    for item in response:
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/devo/api/client.py", line 319, in _return_stream
    response = self._make_request(payload)
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/devo/api/client.py", line 369, in _make_request
    raise_exception(error.args[0])
  File "/opt/logtrust/python3/venv_tf_kpis/lib/python3.7/site-packages/devo/api/client.py", line 42, in raise_exception
    raise DevoClientException(proc_json()(error))
devo.api.client.DevoClientException: {'msg': 'Error Launching Query', 'status': 500, 'object': 'Client dont have key&secret or auth token/jwt'}

Pass this formatting chain and then check with pycodestyle:

# format_pep8 for bash, include in .bashrc
function format_pep8() {
    # Pass the source directory as the first argument
    local directory="$1"
    pushd "$directory" || exit 1
    yapf -i --style pep8 --recursive . &&
    black . &&
    isort --atomic --skip __init__.py --filter-files . &&
    flake8 --ignore=E501,W503,F401,F403 --max-line-length 99 . &&
    true
    popd || exit 1
}

Hi! We are using devo-sdk in Cisco Threat Response integration with Devo and encountered TypeError exception in the client code:

Here if first is equal to None and then it's passed to _is_correct_response it tries to be sliced and unhandled TypeError occurs

I think we might need to change this line to either requests==2.25.0 or requests>=2.25.0 and upload a new version:

We have detected one unexpected behavior when applying the method str() to one DevoSenderException object.
We are getting this error:

__str__ returned non-string (type DevoSenderException)

We have been analyzing the root cause of this issue and we have discovered that the following line is generating an exception with the type Socker error: [Errno 32] Broken pipe

It seems that there is a double nesting of an exception generated by the line:

The line that contains the called method self.socket.sendall(msg) is generating one exception which is encapsulated by another exception:

Update all the packages that need to be updated:

• to click 8.1.7
• to pyyaml 6.0.1
• to certifi 2023.7.22
• to cryptography 41.0.3

Publish a new version of the Python SDK.

socket.send() used instead of socket.sendall(). This causes to do not send all data in all circusntances.
To be fixed in v.3.6.2

In the latest version of https://github.com/DevoInc/python-sdk/blob/master/docs/api/api.md, the description about "Timestamp" is "Millis". In the examples, it is Second.

Date Formats
Fixed format: As described on Official Python Docs. Accepted formats are:

'%Y-%m-%d %H:%M:%S'
'%Y-%m-%d', the time will be truncated to 00:00:00
Timestamp: From epoch in millis

Problem

I've encountered an exception ReadTimeoutError. Here's the full trace:

Traceback (most recent call last):
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/Users/myuser/.pyenv/versions/3.8.12/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/Users/myuser/.pyenv/versions/3.8.12/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/Users/myuser/.pyenv/versions/3.8.12/lib/python3.8/http/client.py", line 277, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/Users/myuser/.pyenv/versions/3.8.12/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/Users/myuser/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/Users/myuser/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='apiv2-us.devo.com', port=443): Read timed out. (read timeout=300)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/devo/api/client.py", line 549, in _make_request
    response = self.__request(payload)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/devo/api/client.py", line 577, in __request
    return requests.post("{}://{}".format(
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='apiv2-us.devo.com', port=443): Read timed out. (read timeout=300)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/myuser/Projects/myproject/src/runners/download/__main__.py", line 81, in <module>
    main(config, logger)
  File "/Users/myuser/Projects/myproject/src/runners/download/main.py", line 60, in main
    data = data_downloader.download_data(from_date=from_date, to_date=to_date)
  File "/Users/myuser/Projects/myproject/src/data/devo.py", line 103, in download_data
    for row in data_generator:
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/devo/api/client.py", line 463, in _return_string_stream
    (_, iterStringResponse, _) = self._make_request(payload)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/devo/api/client.py", line 571, in _make_request
    return raise_exception(error)
  File "/Users/myuser/Projects/myproject/venv/lib/python3.8/site-packages/devo/api/client.py", line 95, in raise_exception
    raise DevoClientException(_format_error(error_data, status=None))\
devo.api.client.DevoClientException: HTTPSConnectionPool(host='apiv2-us.devo.com', port=443): Read timed out. (read timeout=300): HTTPSConnectionPool(host='apiv2-us.devo.com', port=443): Read timed out. (read timeout=300)

Code affected

After digging a bit into the code, I found out that it retries the request only after one type of exception:

except requests.exceptions.ConnectionError as error:
    tries += 1
    if tries > self.retries:
        return raise_exception(error)
    time.sleep(self.retry_delay * (2 ** (tries-1)))
except DevoClientException as error:
    if isinstance(error, DevoClientException):
        raise_exception(error.args[0])
    else:
        raise_exception(error)
except Exception as error:
    return raise_exception(error)

Possible solution

It would be better that other timeout errors are included in the retry. Here's the full list of exceptions: https://urllib3.readthedocs.io/en/stable/reference/urllib3.exceptions.html

Or just simply retry everything that it is not a Devo exception:

except DevoClientException as error:
    if isinstance(error, DevoClientException):
        raise_exception(error.args[0])
    else:
        raise_exception(error)
except Exception as error:
    tries += 1
    if tries > self.retries:
        return raise_exception(error)
    time.sleep(self.retry_delay * (2 ** (tries-1)))