cervisiarius / wikimedia Goto Github PK

My comments will live here.

1. what are these for? documentation would be good.
2. makeKey: why not include accept_language? It adds some entropy (not much, but some). Additionally, you should be using IP or X_FORWARDED_FOR, not both; those machines going through SSL terminators may change terminator between requests, altering the IP and, as a result, the hash. In fact, I would consider it highly likely. Those without HTTPS will have an X_FORWARDED_FOR of -, meanwhile, adding almost no value to it. I can point you to some examples of this decision tree in a pending UDF, if that would help.
3. The pageview filters can be found (already in Java) here. It is probably the most convenient resource for this filtering.
4. Are you doing anything to limit the types of file you run over? i.e., does the codebase run over all webrequest files in a period, and then apply filters, or is it limited to just text/mobile, or text, say?