pisa-engine / warcpp Goto Github PK

When we parse a WARC file, we split the content into two parts: header and content.
In my opinion, this should be (eventually) done by an HTTP/HTML parser.

As a consequence, warcpp should never fail if anything in the HTTP content is wrong, either the HTTP header or the HTTP content.

I propose to change a bit the design of this library in order to remove any reference to HTTP. Any library users, such as pisa, should have additional HTTP/HTML functions/classes in order to strip out or parse HTTP header (depending what they want to achieve or need) .

WARC/0.18
WARC-Type: response
WARC-Target-URI: http://fb.esnips.com/doc/cdf9a27f-a19c-497b-8aee-88d6e7341eb2/<9C><AE>-^Q<97><89><BA>
<84><C4>^O<8E>=<84>~<B1>__

WARC-Warcinfo-ID: cf5c4749-1434-4453-9233-1f670f3bf017
WARC-Date: 2009-03-65T10:36:03-0800
WARC-Record-ID: <urn:uuid:f25c86df-37cf-4948-b9eb-bb7fe0dd6491>
WARC-TREC-ID: clueweb09-en0001-41-14942
Content-Type: application/http;msgtype=response
WARC-Identified-Payload-Type: 
Content-Length: 29106

I'm 99% sure this is not allowed by the protocol, but let's be pragmatic here. I suggest the following:

1. At the beginning, when reading version, read until found a correct version line or EOF. Ignore anything invalid until then.
2. If, after reading fields, the record is not valid, then start over (read version again). Ignore that record.

In our case, it will stop reading fields after URI, then it will see there's no content length, so we skip to the next record.

The worst thing that can happen, there is something wrong after all mandatory fields are discovered but still header is not done (very improbable since usually content length is last). Then, our content might be read wrongly, but we'll still correct it from the next record.

There must be a variable that turns off compiling and running tests in CMakeLists.txt.