Giter VIP home page Giter VIP logo

data-detective's Issues

[FEATURE] Support data-detective-airflow for python 3.10

🚀 Feature request

Some code must be improved for python3.10

Broken DAG: [/opt/airflow/dags/dag_generator.py] Traceback (most recent call last):
  File "/opt/airflow/.local/lib/python3.10/site-packages/data_detective_airflow/operators/sinks/pg_single_target_utils.py", line 11, in <module>
    from data_detective_airflow.utils.notnull import is_not_empty
  File "/opt/airflow/.local/lib/python3.10/site-packages/data_detective_airflow/utils/notnull.py", line 1, in <module>
    from collections import Iterable
ImportError: cannot import name 'Iterable' from 'collections' (/usr/local/lib/python3.10/collections/__init__.py)

[FEATURE] Поддержка фикстур для airflow 2.2.*

🚀 Feature request

Is your feature request related to a problem?

На версии 2.2.1 тесты падают с ошибкой вида.

ERROR tests/operators/extractors/test_request_dump.py::test_source[test_dag0] - airflow.exceptions.DagRunNotFound: DagRun for 'test_source' with date 2020-02-02T00:00:00+00:00 not found

comparison dataframes in assertions.py

DISCLAIMER
I came here without deep knowledge of data-detective. Following question is purely based on the contents of the assertions.py file
(https://github.com/Tinkoff/data-detective/blob/master/data-detective-airflow/data_detective_airflow/test_utilities/assertions.py)

Function assert_frame_equal compares two dataframes. How? By computing two scalar values with to_bytes and comparing them. This method has some limitations:

a = pd.DataFrame([1, 1, 2, 4])  
b = pd.DataFrame([3, 2, 4, 3])  
c = pd.DataFrame([2, 4])  
d = pd.DataFrame([1, [1, 2], 4])  
e = pd.DataFrame([{"a": 2}, {"a": 4}])  
     
assert to_bytes(a) == to_bytes(b) == to_bytes(c) == to_bytes(d) == to_bytes(e)  # True

Probably you compare dataframes of the same size or the same type. So there's no chance you'll have dataframes C, D, E in test simultaneously. But having dataframes A and B equal makes one wondering.
Have you considered such cases?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.