Rewrite the networking client with base go in order to understand it better

If an account does not have paid within 24 hours delete all data.

Refractor city districts to do not use the database. Prepares for https://github.com/WoningFinder/woningfinder.nl/issues/22

. This will be used later on to make the weekly mail update

Funda Soep

Ongeveer de helft van de jurisprudentie over databankrecht is “Nederlandse Vereniging van Makelaars versus iemand die Funda wil scrapen”, en dat is niet zo gek, want de huizenmarkt is erg lucratief. De Telegraaf was de eerste met El Cheapo, dat de zoekresultaten van Funda schraapte en toonde alsof het haar eigen zoekmachine was. Dat mocht niet, oordeelde de Hoge Raad in 2002, want Funda was een beschermde databank. Funda had flink geïnvesteerd in die site. Onder andere had ze iedere NVM-makelaar van een terminal voor het uploaden van advertenties voorzien - zo’n 19.000 gulden per stuk.

Zoekallehuizen had het dan handiger bekeken: die namen Funda niet over, maar gingen gewoon naar de sites van de individuele makelaars, waar immers al die advertenties ook staan. Kost iets meer werk, maar daar heb je spiders voor.
Dit mocht: de makelaars waren immers niet bezig met een databank met huizenadvertenties, maar met het verkopen van huizen. De advertenties op hun site waren dus niet beschermd onder het databankenrecht. En Zoekallehuizen toonde niet de hele advertentie, maar alleen een kort stukje tekst, een verkleinde foto en een hyperlink naar de originele advertentie op de site van de makelaar. Dat valt onder het citaatrecht; Zoekallehuizen kondigt zo aan wat er te zien is achter de hyperlink.

http://sync.nl/andermans-site-scrapen-wanneer-mag-dat/3

WoningFinder is now using Nominatim for querying the city district of a house.
It seems like at a higher rate the API calls will be limited and WoningFinder does not comply with the acceptable usages policy.

• Find alternative (possibly Here Maps API)
• Add fallback to find city district from address if coordinates not available (or simply use address to find city district)

If a zoekopdracht includes a maximum price that falls in the free sector the zoekopdracht becomes paying. Otherwise it is 2 euro.

• Remove as well the "Sneller garantie"
• Remove housing allowance
• Do not let people in the basis plan select maximum price (based on their incomes)

The housing_preferences_housing_types and housing_preferences_cities are not deleted when deleting housing preferences.

Replace auto guess cron (and remove env) by adding checking time default to WoningFinder and to houses corporations. Checks only twice a day at those times. Allow as well to check only one housing corporation

• Improve SQL query
• Check performance

Right now it is hard to see which logs belongs to which connector for housing-finder. Improve it's logging.

At the same time verify the logging fo any other service.

Maybe add a sort that Pro plans gets first instead.

Check list on woningcorporaties.nl

When adding a housing corporation as well add in the list of supported housing corporations.

Overijssel

• https://dewoonplaats.nl
• https://www.roomspot.nl/ (sjht and de veste)
• https://www.domijn.nl
• https://www.onshuis.com
• https://www.woningnethengeloborne.nl
• https://www.mijande.nl

https://www.woontwente.nl/over-woon/corporaties-woon

End 2021 check integration with WoonTwente.

Revert 0788634 when DO supports go 1.16

Those errors are nothing we can do about (bad data), this should be logged as Info. The daily test would report those errors.

Follow up of #9.

• https://blog.questionable.services/article/testing-http-handlers-go/
• Handler tests

• Update preferences handler and test handler
• Create service for updating user preferences (basically delete + re-create?)

Right now a http client to connect to housing corporation is shared by all the user. This cause problem because we are logging for each user.

The testing must be complete and the code coverage must be high. It is extremely important to ensure everything is tested as the core of WoningFinder is based on using external APIs.

• read https://martinfowler.com/bliki/TestPyramid.html
• database mock and tests
• client tests
• external api tests
• basic unit tests

Note: Yet unit tests calls external party, they should use mock, the real test should be end-to-end tests

• add tests that ensure that all connector fills-up all the data of the Housing (end to end test)

• retry policy
• Avoid querying the database at each request, check if a cache (with redis or on memory) can be used
• rate limiter
• random user agent, change of it etc, etc..
• add timeout

Check more on go resiliency pattern

• When fetching houses, check if the city exists in the database, if not add it to the corporation (because the corporation now supports that city) -> in order to avoid doing a check at each request, store the list of city in an array and check it against the database.
• Add tools to update housing corporation based on their Info
• When reacting to an offer the users should be sorted by creation date (earlier first) and plan (pro first)

• De Woonplaats
• (in progress) OnsHuis
• Domijn
• SHJT (Roomspot)

• Make the secret key dependent of multiple factors to limit security issue in case of database leak

The ClientProvider interface permits to get a corporation.Client from a housing corporation entity.
Yet it is hardcoded and gotten by name which is not nice. Possibly find a solution to that (maybe using reflection?)

The emails does not look good on black screen. Please do not use that much purple but more white.

• Use SendCorporationCredentialsError

Unit testing of services is not yet supported because go-pg cannot (yet) be mocked. Add services unit testing go-pg/pg#959.

Logging is now not super well supported, same as error handling.
Investigate the possibility of using Sentry, New Relic, as well as Uber Zap library.

Backend

• Look at https://github.com/go-chi/chi and net/http
• Stripe webhook for payment to confirm a user payed and save it in the database (queue it in redis then validate it with worker). More information here

Use cases

• Payment should be done for validating completion of the registration. WoningFinder should not look for a house if the user has not paid.
• Possibly needs to add more field in the database about notions (containing the user plan, the payment date, and if he found a house using his housing preferences).

WoningNet often contains the 3 way of reactions (Random, First come first served and registration period).

Maybe it would be nice to split the multi fetch of first come first served with the rest

Right now if a user quit the payment process, the user can never pay again. Add an handler for that purpose.

Related: https://github.com/WoningFinder/woningfinder.nl/issues/8

Right now a user that didn't paid automatically get deleted after 48h. Add a reminder to the user to finish its order 8h after the user is created.

Do watch out that the email is sent only once.

Once someone has found a house the zoekopdracht is automatically deleted. The user will have to pay again if he wants to looks longer. This means maybe create as well a daily cron that checks users with zoekopdracht that should be deleted (and send notification about it #3).

Right now someone can take the basis plan and react to all houses. Add a maximum incomes for the basis plan (maybe update offering?).

This requires API and website change

Implemented in e7f7a51

Right now the JWT token is valid 6 hours. It is a bit long, maybe make it valid 2h instead.

Implement email notifications:

• Welcome email (explaining again weekly email)
• Notifies to which house WoningFinder has applied and where (with link)
• Notifies if a housing corporation credentials is not valid anymore - Postponed to #11
• Notify if zoekopdracht deleted because a house has been found - Postponed to #15

• Roomspot needs to enrich the houses details, this could be paralellized.
• Look for other place to run request concurrently.

Remove mention "goed nieuws" in weekly update email. Could cause false joy to customers.

Right now if a user looks for a house and an appartement and wants a balcony and a garden, both the house and the appartement should have a balcony and a garden to be selected.

We want smart preferences such as a house with a garden and an appartement with a balcony would be selected instead of nothing.