Issue with decoding quotation mark about crawlee HOT 2 OPEN

So, a little investigation writeup:

To construct the context object for the request handler, HttpCrawler._runRequestHandler() calls this._parseResponse(), which, in turn, calls this._parseHTML().

_parseHTML() is overriden in CheerioCrawler, and returns

{ 
  get body() {
    return isXml ? "..." : $.html({decodeEntities: true})
  },
  // ...
}

Apparently, this was added there to "save memory for highly parallel runs" (source). However, currently we don't get this effect anyways, since the result of _parseHtml() is immediately destructured here, so when requestHandler, context.body is already a string, and not the getter. (You can trivially verify this by putting a breakpoint/debugger;/console.log() into the getter, and checking at what moment (with what call stack) it is called.

Also, I don't think there's a way to have $.html() return what we want. When a website responds with ``, content-type: text/html:

/* consider a website that responds with
content-type: text/html
{"foo": "<p>bar &lt; &quot; baz</p>"}
*/
const crawler = new CheerioCrawler({
    requestHandler({ $ }) {
        console.log($.html({ decodeEntities: false }));
    },
});
// --> {"foo": "<p>bar < " baz</p>"}

const crawler = new CheerioCrawler({
    requestHandler({ $ }) {
        console.log($.html({ decodeEntities: false }));
    },
});
// --> {&quot;foo&quot;: &quot;<p>bar &lt; &quot; baz</p>&quot;}

const crawler = new HttpCrawler({
    requestHandler({ body }) {
        console.log(body.toString("utf8"));
    },
});
// --> {"foo": "<p>bar &lt; &quot; baz</p>"}

First case is current behavior of ctx.body, and imo it's bad, because it breaks HTML. Also, it's really confusing that CheerioCrawlingContext.body and HttpCrawlingContext.body return different values (that's kind of what prompted the original report by @gullmar).

The second case is probably also unjustifiable, since it would change current behavior heavily (websites probably contain a lot of quotes, &s and idk what else cheerio decides to escape).

Therefore, I propose we remove body getter, and instead return the original body buffer .toString("utf8"), to have the same data like HttpCrawler, but also keep body as a string to avoid breaking Actors.

It is also ok with me to call this a won'tfix, since website returning JSON with content-type: text/html is just weird.

from crawlee.

Possibly (quite remotely) related to #2317

from crawlee.