[IMPROVEMENT] replace http client (HttpFetcher) about skrape.it HOT 3 CLOSED

Can you clarify a bit, do you want to replace the Browser/HttpFetcher or add an additional one?

from skrape.it.

hey,
the httpFetcher is currently using jsoup to make http requests.
https://github.com/skrapeit/skrape.it/blob/master/src/main/kotlin/it/skrape/core/fetcher/HttpFetcher.kt

we want to use KoHttp instead because it ships with a bunch of nice features like async requests and use jsoup for the document parsing only.

one thing to notice:
currently it is possible to add validateTLSCertificates and maxBodySize parameters to the request. since I don't think this is possible with HoHttp this ones can be removed from Request if it is not possible to manage this behavior with KoHttp.

The other Request options should be working further on.

since KoHttp provides different functions to handle certain http-request verbs (e.g. httpGet(), httpPost()) and scrape{it}'s approach is to pass the http verb as an parameter (Enum) it looks like we will need a mapping from org.jsoup.Connection.Method (which is the Enum that is currently used for method field in Request.kt) or even better providing an Enum our own and get rid of the jsoup one.

from skrape.it.

I just opened #64, let me know if it's what you're looking for

from skrape.it.