zypp-io / twinfield Goto Github PK

View Code? Open in Web Editor NEW

9.0 9.0 2.0 374 KB

Python client for the Twinfield API to get and post data

License: Apache License 2.0

Python 100.00%

api oauth2 python twinfield

twinfield's People

Stargazers

Watchers

Forkers

melvinfolkers jankrl

twinfield's Issues

BUG: Lees alleen 1 regel in

Om efficientie slag te maken, lees alleen 1 regel in:

twinfield/twinfield/messages.py

Lines 301 to 319 in ef380f5

 def read_and_delete_table(tablename) -> pd.DataFrame: 

 """ 

  Parameters 

  ---------- 

  tablename: tablename in DWH to read an delete 

  Returns: dataframe containing records of selected tablename. original table will be deleted. 

  ------- 

  """ 

 conn = auth_azure() 

 engn = create_engine(conn, pool_size=10, max_overflow=20) 

 data = pd.read_sql_table(table_name=tablename, con=engn) 

 data.head(n=0).to_sql(tablename, engn, schema="dbo", if_exists="replace") 

 logging.info(f"{len(data)} regels gelezen. tabel verwijderd") 

 return data

ENH: Deploy alleen op merge main

DOC: add installation in README

BUG: for dimension in data.findall("dimensions/dimension"): AttributeError: 'NoneType' object has no attribute 'findall

De releases 2.1.0 en 2.1.1 geven onderstaande fout bij de insert/update van dimensies. Dit komt vaak/altijd voor, zodat ik mijn versies heb vastgezet op 2.0.5.

Het lezen van dimensies geeft geen problemen.

2023-03-16 08:21:56.807 [INFO ] [root] - Er zijn 6 dimensies met verschillen
2023-03-16 08:21:56.922 [INFO ] [root] - E-mail verzenden: Verschillen dimensies
2023-03-16 08:21:57.352 [INFO ] [root] - Er zijn 6 regels met wijzigingen en 3 regels met nieuwe dimensies.
2023-03-16 08:21:57.433 [INFO ] [root] - 6 dimensies geselecteerd.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/41_selfbilling_zzp/src/verwerken.py", line 31, in <module>
    alleen_verwerken(test=test)
  File "/41_selfbilling_zzp/src/verwerken.py", line 22, in alleen_verwerken
    verwerken_gegevens(stam=stam, df=df, test=test)
  File "/41_selfbilling_zzp/src/main.py", line 29, in verwerken_gegevens
    DimensionSync(
  File "/usr/local/lib/python3.9/site-packages/dimensions/core.py", line 400, in sync
    df_dim_errors, df_dim_correct = self.send_request()
  File "/usr/local/lib/python3.9/site-packages/twinfieldtools/dimensions.py", line 295, in send_request
    df_errors, df_correct = self.parse_response_dimensions(response)
  File "/usr/local/lib/python3.9/site-packages/twinfieldtools/dimensions.py", line 212, in parse_response_dimensions
    for dimension in data.findall("dimensions/dimension"):
AttributeError: 'NoneType' object has no attribute 'findall'

De response bevat de volgende tekst. Kan het zijn dat er geen token wordt meegestuurd?

<soap:Fault><faultcode>soap:Client</faultcode><faultstring>Access denied. Token invalid.</faultstring><faultactor>http://api.accounting2.twinfield.com/webservices/processxml.asmx?wsdl</faultactor><detail><message>Access denied. Token invalid.</message><code>354</code><source>ProcessXmlDocument</source></detail></soap:Fault>

ENH: add actual sec_wait to wait between new requests

When a request fails, we retry. And the idea was to wait couple of seconds between the new requests. We have it in the logging, but we never actually implemented it. This happens here:

twinfield/twinfield/core.py

Lines 38 to 59 in a4b90f8

 success = False 

 max_retries = 5 

 retry = 1 

 sec_wait = 10 

 response = None 

 while not success: 

 if retry > max_retries: 

 logging.warning(f"Max retries ({max_retries}) exceeded, " f"stopping requests for this office.") 

 break 

 response = requests.post( 

 url=f"{self.cluster}/webservices/processxml.asmx?wsdl", 

 headers={"Content-Type": "text/xml", "Accept-Charset": "utf-8"}, 

 data=browse.body(), 

 ) 

 if not response: 

 logging.info(f"No response, retrying in {sec_wait} seconds. Retry number: {retry}") 

 self.access_token = self.refresh_access_token() 

 retry += 1 

 else: 

 success = True 

 return response

Proposal, actually sleep:

time.sleep(wait_sec)

ENH: add browsecode 040_1

Adding functionality for consolidated figures

Update package versions

ENH: Package twinfield

ENH: add timeout to requests

[ENH] change twinfield_sv routine

The Twinfield SV routine should aggregate on an additional field "administratienummer"

[ENH] improve pickle import

ENH: Voeg zelfde functies in één

In transforms.py hebben we drie functies die hetzelfde doen, deze kunnen naar één functie worden samengevoegd. Gaat om: format_100, format_200, format_164

ENH: add option to clean up pickle dir

ENH: The default value of regex will change from True to False in a future version

Bij het runnen van projecten krijg je sinds enige tijd de volgende melding:

C:\Users\BenGroenHappyNurse\Workspace\facturen_ons_hnt_31\venv\lib\site-packages\twinfield\functions.py:450: FutureWarning: The default value of regex will change from True to False in a future version.
df.columns = df.columns.str.replace("fin.trs.", "")
C:\Users\BenGroenHappyNurse\Workspace\facturen_ons_hnt_31\venv\lib\site-packages\twinfield\functions.py:450: FutureWarning: The default value of regex will change from True to False in a future version.
df.columns = df.columns.str.replace("fin.trs.", "")

Dat lijkt hier te gebeuren:

def rename_column_labels(df, module):
if module != "040_1":
df.columns = df.columns.str.replace("fin.trs.", "")
else:
login = twinfield_login()
fields = get_metadata(module, login)
df.rename(columns=fields["label"], inplace=True)

return df

Kan dit worden aangepast?

ENH: include login in retry with custom error

Right now we catch ConnectionError and SSLError in our retries when we do a request, but in login we raise a ValueError. I propose to create a custom error when login fails, so we can specifically catch that error.

ENH: make project flake8 compatible

Add user to login start import

ENH: add amount error messages in teams

reformat code with black library

ENH: replace .format with f strings

BUG: clean up / write docstrings

Not all functions have (a correct) docstrings. These should be added / cleaned up.

FEATURE REQUEST: Printing progress (disable)

Feature request
An option not to display tqdm loop messages. They pollute logs (example below).


[2023-10-10T12:24:23.920Z] importing module 030_1 - XXX - {'from': '2023/00', 'to': '2023/00'}...:   0%|          | 0/1 [00:00<?, ?it/s]

[2023-10-10T12:24:25.319Z] importing module 030_1 - XXX - {'from': '2023/01', 'to': '2023/01'}...:   0%|          | 0/1 [00:00<?, ?it/s]

[2023-10-10T12:24:26.177Z] importing module 030_1 - XXX - {'from': '2023/02', 'to': '2023/02'}...:   0%|          | 0/1 [00:02<?, ?it/s]

[2023-10-10T12:24:26.983Z] importing module 030_1 - XXX- {'from': '2023/03', 'to': '2023/03'}...:   0%|          | 0/1 [00:03<?, ?it/s]

[2023-10-10T12:24:28.157Z] importing module 030_1 - XXX- {'from': '2023/04', 'to': '2023/04'}...:   0%|          | 0/1 [00:03<?, ?it/s]

[2023-10-10T12:24:28.998Z] importing module 030_1 - XXX- {'from': '2023/05', 'to': '2023/05'}...:   0%|          | 0/1 [00:05<?, ?it/s]

[2023-10-10T12:24:29.700Z] importing module 030_1 - XXX- {'from': '2023/06', 'to': '2023/06'}...:   0%|          | 0/1 [00:05<?, ?it/s]

[2023-10-10T12:24:30.627Z] importing module 030_1 - XXX- {'from': '2023/07', 'to': '2023/07'}...:   0%|          | 0/1 [00:06<?, ?it/s]

[2023-10-10T12:24:31.423Z] importing module 030_1 - XXX- {'from': '2023/08', 'to': '2023/08'}...:   0%|          | 0/1 [00:07<?, ?it/s]

[2023-10-10T12:24:32.338Z] importing module 030_1 - XXX- {'from': '2023/09', 'to': '2023/09'}...:   0%|          | 0/1 [00:08<?, ?it/s]

[2023-10-10T12:24:32.923Z] importing module 030_1 - XXX- {'from': '2023/10', 'to': '2023/10'}...:   0%|          | 0/1 [00:09<?, ?it/s]

[2023-10-10T12:24:33.430Z] importing module 030_1 - XXX- {'from': '2023/11', 'to': '2023/11'}...:   0%|          | 0/1 [00:09<?, ?it/s]

[2023-10-10T12:24:34.139Z] importing module 030_1 - XXX- {'from': '2023/12', 'to': '2023/12'}...:   0%|          | 0/1 [00:10<?, ?it/s]

[2023-10-10T12:24:34.872Z] importing module 030_1 - XXX- {'from': '2023/13', 'to': '2023/13'}...:   0%|          | 0/1 [00:11<?, ?it/s]

[2023-10-10T12:24:35.554Z] importing module 030_1 - XXX- {'from': '2023/55', 'to': '2023/55'}...:   0%|          | 0/1 [00:11<?, ?it/s]

[2023-10-10T12:24:35.554Z] importing module 030_1 - XXX- {'from': '2023/55', 'to': '2023/55'}...: 100%|██████████| 1/1 [00:12<00:00, 12.47s/it]

[2023-10-10T12:24:35.554Z] importing module 030_1 - XXX- {'from': '2023/55', 'to': '2023/55'}...: 100%|██████████| 1/1 [00:12<00:00, 12.48s/it]

Potential solution
Use TQDM feature to detect TTY device and does not print messages if device is incompatible (tqdm(..., disable=None)).
Some devices might not be detected properly, in this case suggest to implement optional LOG_LEVEL flag (env variable? config file?) which prints progress bar when flag != 'DEBUG'

References
https://tqdm.github.io/releases/#v490-2016-10-31
tqdm/tqdm#281

Background
I am using the library in a bigger script. I log certain steps and progress bar pollutes log output.

Looking forward to hear your feedback about this idea.

ENH: save errors to errors.pkl

Het is handig als we de errors niet in dezelfde pickle files laten landen als de resultaten. Als we errors opslaan in een errors.pkl, dan is dat netter. ook kunnen we de volgende functionaliteiten inbouwen:

als errors.pkl bestaat, mag de upload niet doorgaan (kennelijk is er een administratie onjuist geimporteerd)
in functie "set_rerun" kunnen we de errors.pkl inladen, in plaats van een filter te zetten op de resultatentabel op het bestaan van kolom "faultcode".

BUG: login method deprecated

The login method using SessionId seems to be deprecated. We need to develop a new login function.

FIX: Functie transform_soap_message lijkt niet gebruikt

Functie lijkt niet te worden gebruikt in het project, is deze nog nodig?

cc @melvinfolkers

twinfield/twinfield/messages.py

Lines 148 to 175 in ef380f5

 def transform_soap_message(run_params) -> list: 

 """ 

  Parameters 

  ---------- 

  run_params: input parameters of script (set at start of script) 

  Returns: list of xml messages imported from the temp 'rawdir' folder. 

  ------- 

  """ 

 xml_messages = [] 

 for filename in os.listdir(run_params.rawdir): 

 if filename.endswith(".xml"): 

 f = open(os.path.join(run_params.rawdir, filename), "r", encoding="utf-8") 

 message = f.read() 

 return message 

 d = dict() 

 d["office_code"] = "ZZ_1060255" # let op deze moet variabel worden 

 d["xml_msg"] = message 

 xml_messages.append(d) 

 return xml_messages