Union.pm needs rework about finance-quote HOT 20 CLOSED

You are correct, a hard coded path is not viable for a F::Q module. I'm sorry to say that unless someone volunteers to rewrite Union.pm, the module will likely get removed from future releases.
Thank you.

from finance-quote.

To read the information from the site of the individual fund, would mean to extract the data from an HTML file, right? Is it really a good Idea to parse an GUI to read the Information? Already small changes in the GUI maybe broken the parsing implementation. In my opinion there must be an machine readable interface (like CSV or REST-API) to access the informations.

In the mailing list, I read that someone write to Union Investment and there answer was the new CSV interface is the solution for such cases. But the new CSV interface is made for using with a browser, which makes it difficult or nearly impossible to use automatically in finance-quote. Did I understand that right so far?

from finance-quote.

Yes, you have to select one or several (or all) files to get a CSV. Maybe the source code of that web page offers a possibility to mark all funds as selected to make a call without user interaction.
They don't offer a REST-API or JSON, so the GUI is the only other chance to get data from them. Other FQ modules like fondsweb do the same.

from finance-quote.

The GUI seems to make many of the functionality with javascript. I have not many javascript experience, so I didn't unterstand the code completely.
Maybe someone can reconstruct how the API works, which is access by this javascript code. I am not sure if this is possible.
Of course it would be easier with some informations from @Union-Investment

from finance-quote.

I've done a little exploring. When selecting funds from https://legacy-apps.union-investment.de/handle?action=initDownloadSearch&standalone=true#historischepreise, the URL that fetches the CSV file (using a POST command) is https://legacy-apps.union-investment.de/handle. The request is passed via data containing something like:

generate=true&action=doDownloadSearch&start_time=27.07.2022&end_time=01.08.2022&csvformat=de&choose_indi_fondsnames=DE0005326565&choose_indi_fondsnames=DE0005314231&choose_indi_fondsnames=LU1093788872&choose_indi_fondsnames=DE0005314249&choose_indi_fondsnames=DE0008491077&choose_indi_fondsnames=DE000A2AR3V2&choose_indi_fondsnames=DE0009766865...

I have not tested curl commands to directly fetch the CSV yet, the day job beckons. I think that cookies and other information passed in the header from the first URL may be required is what gets sent in the POST command.
Once successful it's relatively easy to parse the data from the returned CSV.

from finance-quote.

No special headers/cookies required. I believe the older CSV file included WKN numbers (the union.t test file used the 6-digit WKN to identify the record in the CSV), only ISIN in the new data.

curl --include --silent 'https://legacy-apps.union-investment.de/handle? 
generate=true&action=doDownloadSearch&start_time=27.07.2022&end_time=01.08.2022&csvformat=us&choose_indi_fondsnames=DE0005326565&choose_indi_fondsnames=DE0005314231'
HTTP/1.1 200 OK
Date: Mon, 01 Aug 2022 19:51:46 GMT
Content-disposition: attachment; filename=historische-preise.csv
Content-Type: text/csv
X-Hostname: uidvhuop03
Vary: User-Agent
Set-Cookie: JSESSIONID=997D3A30690724CB62C08F77032FB7A0; Path=/; HttpOnly;Secure;SameSite=none
Age: 0
X-HIT: 0
Accept-Ranges: bytes
Content-Length: 735
Connection: keep-alive
Set-Cookie: TS01119b35=01a6ce534adcabe46ca1a559c732ec8ed974894031a6ffee97d57c3d8c23382e1a3418e4cae32ae0a6bbecf6385441262c98d3a34b; Path=/; Domain=.legacy-apps.union-investment.de

Name,ISIN,Waehrung,Ausgabepreis,Ruecknahmepreis,Ausgabeaufschlag,Datum,Aktiengewinn EStG in %,Aktiengewinn KStG in %,Zwischengewinn (Ertragsausgleich wurde durchgef?hrt),Immobiliengewinn in %,TIS
BBBank Kontinuit?t Union,DE0005314231,EUR,77.95,76.42,0.02,27.07.2022,0.00,0.00,0.00,0.00,-
BBBank Kontinuit?t Union,DE0005314231,EUR,78.71,77.17,0.02,28.07.2022,0.00,0.00,0.00,0.00,-
BBBank Kontinuit?t Union,DE0005314231,EUR,79.17,77.62,0.02,29.07.2022,0.00,0.00,0.00,0.00,-
BBBank Dynamik Union,DE0005326565,EUR,79.65,77.71,0.03,27.07.2022,0.00,0.00,0.00,0.00,-
BBBank Dynamik Union,DE0005326565,EUR,80.66,78.69,0.03,28.07.2022,0.00,0.00,0.00,0.00,-
BBBank Dynamik Union,DE0005326565,EUR,81.56,79.57,0.03,29.07.2022,0.00,0.00,0.00,0.00,-

In English the column headings are:
Name, ISIN, currency, issue price, redemption price, issue surcharge, date, share profit EStG in %, share profit KStG in %, interim profit (income equalization was carried out), real estate profit in %, TIS

from finance-quote.

I managed to add your URL to my Union.pm:
https://github.com/Andre-J/finance-quote/blob/master/lib/Finance/Quote/Union.pm

and it seems to work under GnuCash, but again not under KMyMoney. I'm not sure what they expect different. Other modules like deka.pm work under both apps, while fondsweb.pm does not.

I took start_time one week before end_time, because prices are always at least one day behind, and only given for business days.

As you stated, the lookup symbol has to be changed from WKN to ISIN.

from finance-quote.

It would need some cleaning up for a pull request. I don't believe tempdate is used at all, and the comment about being on the last line is misleading. Is that a holdover from the older CSV file?
I'm not familiar with KMyMoney, so I'm not sure. Perhaps it's expecting a label not being presented?

from finance-quote.

Yes, the old preise.csv had the date only in the last line. Strangely, the old file worked with KMM, and I tried to refill every label that was assigned there. Deka.pm is actually a clone of Union.pm I did some years ago, and that still works both with GnuCash and KMM.

KMM has a calling script financequote.pl which works for me on the command line, but not inside the programme.
I will ask the KMM people to take the updated Union.pm from here and test it.

The updated module is not designed to ask several funds at the same time. The URL would have to be adapted for such a use case.

from finance-quote.

If I understand the logic, the module uses the data from last line returned for each fund (one per GET request). Eliminating the need for logic determining the last possible trading date.

from finance-quote.

Is there a reason to choose using the legacy-apps over the current API? I don't fully understand the ISIN and WKN. For the newer funds where you can't deduce the WKN from the ISIN, is it okay for the updated Union.pm module to require symbols of the form "ISIN:WKN", for example. So if a user presents DE0008491002, the module would extract 849100 as the WKN. But the user could also present "DE0008491002:849100" to handle the new cases.

Given a valid ISIN and WKN, the module could do the equivalent of

1. curl -i -H "user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36" 'https://internal.api.union-investment.de/beta/web/search/results.json?api-version=beta-1.0.3&searchType=pkfondssuche2&searchIndex=pk2&q=isin:DE0008491002&disableQueryEscaping=true'
2. extract fund-search-api and fund-search-api-key from the ui-fund-portrait tag
3. curl -i -H "user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36" 'https://internal.api.union-investment.de/beta/web/search/results.json?api-version=beta-1.0.3&searchType=pkfondssuche2&searchIndex=pk2&q=isin:DE0008491002&disableQueryEscaping=true'

The second GET returns a json object, where date of trade, last price, etc are easy to extract.

It might be that you can use any fund for step 1 to get the api key (or maybe the api is fixed?), and then Union.pm could just do step 3.

from finance-quote.

I only poked around enough to get the URL I noted above as I am happy to mark the module as not working and remove it. I didn't dig onto using tcpdump or Chrome's devel tools and digging into what is possibly java script. The main page being German didn't help either... ;) Even with translate on.
BTW - the URLs for step 1 and 3 in your post are exactly the same. I'm thinking a cut and paste faux pas?

from finance-quote.

Quick test - this worked for me without needed to set a user agent.

curl 'https://internal.api.union-investment.de/beta/web/search/results.json?api-version=beta-1.0.3&searchType=pkfondssuche2&searchIndex=pk2&q=isin:DE0008491002&disableQueryEscaping=true&api-key=6d5b7ad050e948ce99516c20fbe37425'
{
  "results" : [ {
    "q" : "isin:DE0008491002",
    "url" : "",
    "of" : 0,
    "si" : 10,
    "actualQ" : "isin:DE0008491002",
    "availableDocumentCount" : 1,
    "matchingDocumentCount" : 1,
    "omittedDuplicatesCount" : 0,
    "totalDocumentCount" : 236,
    "hits" : [ {
      "articleClass" : "result__list__element clearfix",
      "title" : "UniFonds",
      "titleMarked" : "UniFonds",
      "content" : "",
      "thumbnail" : "283e0e75-a9ba-0480-e563-5254b9b54c1a",
      "url" : "",
      "iconClass" : "result__headline ui-icon-website",
      "documentId" : "283e0e75-a9ba-0480-e563-5254b9b54c1a",
      "date" : "03.08.2022",
      "fondsname" : "UniFonds",
      "fundPortraitUrl" : "/fonds/unifonds-DE0008491002-fonds-849100/?portrait=1",
      "waiUrl" : "https://cdn.dam.union-investment.de/425357_UniFonds_DE0008491002-20220513-1-DE.pdf",
      "pifUrl" : "https://cdn.dam.union-investment.de/429737_UniFonds.pdf",
      "type" : "Aktienfonds",
      "isin" : "DE0008491002",
      "kvg" : "Union Investment Privatfonds GmbH",
      "initPrice" : "54,35 EUR",
      "callPrice" : "51,76 EUR",
      "datePrice" : "03.08.2022",
      "preisdifferenzVortagAbsolut" : "0,84 EUR",
      "diffPrior" : "1,65 %",
      "perf1Jahr" : "-24,11 %",
      "riskClass" : "3",
      "morningstarStand" : "30.06.2022",
      "morningstar" : "3",
      "scopeStand" : "30.06.2022",
      "scope" : "C",
      "haltedauerempfJahr" : 7.0,
      "performance" : [ {
        "key" : "perfMonat",
        "value" : "4,67 %",
        "lastUpdate" : "31.07.2022",
        "chart" : "https://internal.api.union-investment.de/beta/web/portrait/perfchart?segment=uip&period=1m&res=low&isin=DE0008491002&api-version=beta-2.0.1"
      }, {
        "key" : "perf1Jahr",
        "value" : "-24,11 %",
        "lastUpdate" : "31.07.2022",
        "chart" : "https://internal.api.union-investment.de/beta/web/portrait/perfchart?segment=uip&period=1y&res=low&isin=DE0008491002&api-version=beta-2.0.1"
      }, {
        "key" : "perf3Jahre",
        "value" : "1,58 %",
        "lastUpdate" : "31.07.2022",
        "chart" : "https://internal.api.union-investment.de/beta/web/portrait/perfchart?segment=uip&period=3y&res=low&isin=DE0008491002&api-version=beta-2.0.1"
      }, {
        "key" : "perf5Jahre",
        "value" : "0,21 %",
        "lastUpdate" : "31.07.2022",
        "chart" : "https://internal.api.union-investment.de/beta/web/portrait/perfchart?segment=uip&period=5y&res=low&isin=DE0008491002&api-version=beta-2.0.1"
      }, {
        "key" : "perf10Jahre",
        "value" : "67,02 %",
        "lastUpdate" : "31.07.2022",
        "chart" : "https://internal.api.union-investment.de/beta/web/portrait/perfchart?segment=uip&period=10y&res=low&isin=DE0008491002&api-version=beta-2.0.1"
      }, {
        "key" : "perfSeitAuflegung",
        "value" : "14841,08 %",
        "lastUpdate" : "31.07.2022",
        "chart" : "/Magnolia/Shared/Fondswertentwicklung/dab693884ba101333916158ac32ca3ea.0.0/de0008491002_auflegung_400x200.png"
      }, {
        "key" : "perfSeitJahresbeginn",
        "value" : "-23,11 %",
        "lastUpdate" : "31.07.2022",
        "chart" : "https://internal.api.union-investment.de/beta/web/portrait/perfchart?segment=uip&period=default&res=low&isin=DE0008491002&api-version=beta-2.0.1"
      } ],
      "istAnsparplanfaehig" : 1,
      "istNachhaltig" : 0,
      "istKeinDirekterwerb" : 0,
      "footer" : "03.08.2022 - Fonds"
    } ],
    "resultsCounter" : "\u003Cp\u003E\u003Cstrong\u003E1\u003C/strong\u003E Ergebnisse für „\u003Cstrong\u003Eisin:DE0008491002\u003C/strong\u003E“.\u003C/p\u003E"
  } ]
}

from finance-quote.

The first call should have been

curl -i -H "user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36" 'https://www.union-investment.de/fonds/unifonds-DE0008491002-fonds-849100/?portrait=1'

I think this first call needs the user-agent and inside the result you get the api-key. So if the api-key is fixed, then Union.pm could just use the second call, which only needs the ISIN.

from finance-quote.

Name,ISIN,Waehrung,Ausgabepreis,Ruecknahmepreis,Ausgabeaufschlag,Datum,Aktiengewinn EStG in %,Aktiengewinn KStG in %,Zwischengewinn (Ertragsausgleich wurde durchgef?hrt),Immobiliengewinn in %,TIS
BBBank Kontinuit?t Union,DE0005314231,EUR,77.95,76.42,0.02,27.07.2022,0.00,0.00,0.00,0.00,-

Minor issue:
This is using the wrong character encoding.
Returned:Intended
durchgef?hrt:durchgeführt
Kontinuit?t:Kontinuität

Perhaps ISO-8859-1-Text in UTF-8-environment?

from finance-quote.

Perhaps ISO-8859-1-Text in UTF-8-environment?

The CSV file is encoded ANSI, not UTF-8. But the values needed are always ASCII or numerical, so it should not matter.

from finance-quote.

So if the api-key is fixed, then Union.pm could just use the second call, which only needs the ISIN.

If the company protects the API with an api key, I assume they will not be happy if we bypass that. So I would rather stick to the CSV file which is promoted to be for customer use.

In the meantime, the KMyMoney issue has been fixed. It was a Windows-only issue while reading the sources list from quotes.pm returning CR LF terminated output and not LF only as Linux does. Nothing to be done on the F:Q side.

I have now cleaned up the code of Union.pm
https://github.com/Andre-J/finance-quote/blob/master/lib/Finance/Quote/Union.pm
and updated the test file
https://github.com/Andre-J/finance-quote/blob/master/t/union.t

Please review both.

from finance-quote.

I see both sides of the debate, use the internal api call to fetch and parse json, or fetch and parse the csv file. Since @Andre-J has already reworked the module for the latter, while he didn't create a pull request, he has done most of the heavy lifting. If I have time I'm going to work on creating a pull request.

from finance-quote.

If I have time I'm going to work on creating a pull request.

I have created a pull request, hopefully correctly.

from finance-quote.

@Andre-J,
Thank you, but I have not reviewed it yet, I was out of state for a family medical issue. I returned a week or so ago, but still have limited free time outside of my day job.
Hopefully I'll have more time to commit to F::Q in another week or so.

from finance-quote.