wilsonfreitas / brasa Goto Github PK

Create a table to manage all files: imported, downloaded, and produced.
Each generated file has its checksum and this is a key to this file and asserts its uniqueness.
This allows to track ETL file generation and ETL's dependencies.

Estava lendo o texto o artigo "QuantLib, dez anos depois …" (https://www.wilsonfreitas.net/posts/2023-07-19-quantlib/quantlib.html) que utiliza os dados deste projeto.
Agora que não existe mais o servidor de download o caminho para gerar o histórico é coletar diariamente os dados da página url: https://www2.bmf.com.br/pages/portal/bmfbovespa/lumis/lum-ajustes-do-pregao-ptBR.asp?

Obrigado

It is necessary to handle imported files with templates.
The source encompasses more than just downloaded files; imported files are also included and must be duly logged within the cache management system.

Download/import workflow:

• process:
  • request -> downloaded file[s] -> processed file[s]
  • import -> downloaded file[s] -> processed file[s]

Improve display of brasa.cli

• show which BRASA_DATA_PATH is being used
• setup should show folders created and database created

The template filename and id must be the same.
This must be checked during the template loading.

• check template filename and id

Use logging to register every important detail, for example, company-info, company-details, ...
It is important to register events like download fails, and its causes.

Format returned tables

• b3-bvbg086
• b3-bvbg028
• b3-lending-trades
• b3-otc-trade-information
• b3-trades-intraday

Parsers

• b3-companies-options
• b3-cash-dividends
• b3-company-details
• b3-company-info
• b3-bvbg027
• b3-indexes-composition
• b3-indexes-historical-prices
• b3-indexes-theoretical-portfolio-with-sectors
• b3-indexes-theoretical-portfolio

Utils

• reread and redownload arguments to get_marketdata - redownload implies reread

New files

• DeltaOpcoes: options on futures (currency, interest rate, and commodities)
  • http://www.bmf.com.br/Ftp/DeltaOpcoes/DO%y%m%d.ex_
• Equity options http://www.bmf.com.br/Ftp/IPN/GPS/OpcoesdeEquities/PE231212.ex_
• Volatility of equity options http://www.bmf.com.br/Ftp/IPN/GPS/VolatilidadedeEquities/VE231211.ex_
• Historical CDI https://www.b3.com.br/pt_br/market-data-e-indices/indices/indices-de-segmentos-e-setoriais/serie-historica-do-di.htm
• Derivatives prices and volume https://www2.bmf.com.br/pages/portal/bmfbovespa/lumis/lum-sistema-pregao-ptBR.asp?Data=01/02/2013&Mercadoria=IDI
• Companies classification https://sistemaswebb3-listados.b3.com.br/listedCompaniesPage/classification
• Funds (ETFs, FIIs, Fixed Income ETFs) https://sistemaswebb3-listados.b3.com.br/fundsProxy/fundsCall/GetListedFundsSIG/eyJ0eXBlRnVuZCI6MjAsInBhZ2VOdW1iZXIiOjEsInBhZ2VTaXplIjoyMH0=

The templates provided below include a 'refdate' parameter, which specifies the initial dates or periods to be configured when initiating the download of all files.

• b3-bvbg086 - start: 2018-01-01
• b3-bvbg028 - start: 2018-01-01
• b3-bvbg087 - start: 2016-07-01
• b3-futures-settlement-prices - start: 2010-01-01
• b3-otc-trade-information - start: 2019-10-18
• b3-cotahist-daily - start: current year
• b3-cotahist-yearly - start: 2000, end: previous year
• b3-lending-trades - last 20 days
• b3-trades-intraday - last 20 days
• b3-economic-indicators-price - last 20 days
• b3-equities-volatility-surface - start: 2016-11-04
• b3-equity-options - start: 2016-11-04

Build a command to execute the initial setup downloading all files related to these templates.

Build a graph to connect templates.
Connected templates improve the dependency system.
Luigi should be used as the dependency system.

Each asset (equity, company, curve, ...) has its symbol and structure (columns, for example).
A central table with all assets would facilitate access to all assets.
Queries could be made and return assets that match and one function like get_asset would be used to return a dataset corresponding to the requested asset.

get_asset("DI1F16") # returns dataset with historical data of DI1F16 future contract
get_asset("DI1") # returns dataset with historical data of all DI1 future contracts
get_asset("DI1T") # returns dataset with historical data of DI1 curve based on traded contracts
get_asset("DI1ST") # returns dataset with historical data of DI1 curve with standardized terms
get_asset("PETR4") # returns dataset with historical data of PETR4

Tesouro Direto files can be downloaded from the same URL but the arguments vary.
The easy solution is to create different template files, one for each varied argument.
The following questions must be answered:

• How can multiple files be downloaded from a single template?
• How will these files be stored?
• How will the meta be saved? One meta or multiple metas?

Proposed structure to downloaded files.

raw/<template>/<varied-argument>/checksum/file

another proposal

raw/<template>-<varied-argument>/checksum/file

Template example:

id: td-historical-prices
filename: ~
filetype: CUSTOM
description: Tesouro Direto preços e taxas históricas
downloader:
  verifyssl: false
  function: brasa.downloaders.multi_download
  url: https://cdn.tesouro.gov.br/sistemas-internos/apex/producao/sistemas/sistd/{year}/{contract}_{year}.xls
  format: xls
  use-filename: true
  args:
    year: ~
    contract: ~
  multi:
    contract:
      lft: LFT
      ltn: LTN
      ntnb: NTN-B
      ntnb_principal: NTN-B_Principal
      ntnf: NTN-F
      ntnc: NTN-C