Tor2web request logging about tor2web HOT 8 CLOSED

A method to log-rotate the logs must be in place.

from tor2web.

For estimating users in a privacy preserving way: https://metrics.torproject.org/papers/countingusers-2010-11-30.pdf.

Before we can put this inside of the next to come milestone we need to figure out how to do this. By this I mean:

1. What questions we want answered from the data we collect
2. What data we want to collect.
3. How we are going to aggregate it and collect it.
4. If this can have a negative privacy effect on the users.

from tor2web.

mmmm i'm wondering if it would not be better to consider this ticket only a "debug log"and to move the "data collection privacy enforced" issue to #13 that represent the statistics collection (so also a little part of log collection).

That way we can implement the "logging" as a debugging features, to be activated only for debugging purposes, and when activated follow the normal apache logging format with privacy enhancement for Client IP like done in Tor2web 2.0:
https://github.com/globaleaks/tor2web-2.0/blob/master/apache/tor

That way the "debugging log" can be analyzed (while preserving only privacy of client ip address) with standard apache-format-compatible realtime and batch tool (in case of debugging session), while specific operation and maintenance statistics of tor2web would be provided by #13 that need to archive all the data during time with the privacy consideration you highlighted?

from tor2web.

With the last commit (429c76b) i've enabled request logging with apache legacy format.

from tor2web.

From talking about this with Karsten:

You could sanitize tor2web logs similar how we sanitize Apache logs of
the various .tpo web servers. See the tor-dev thread here:

https://lists.torproject.org/pipermail/tor-dev/2011-August/002892.html

The code that implements the sanitizing is here:

https://gitweb.torproject.org/webstats.git/blob/HEAD:/src/org/torproject/webstats/Main.java

But... That approach doesn't remove parts which are particularly
sensitive in tor2web's case: .onion addresses and paths of the requested
resources. I'd think both parts are too sensitive to be published in
logs even if you apply the webstats thing.

What's the purpose of publishing or using tor2web logs? If you want to
analyze how hidden services differ in popularity, you could cut off
.onion addresses after 8 chars and drop the path part. Or if you want
to avoid that people recognize a known .onion address, you could run
.onion addresses through SHA1 after concatenating a secret, monthly
changing string. That's similar to how we sanitize bridge IP addresses:

https://metrics.torproject.org/formats.html#bridgedesc

Hope that helps. Feel free to paste my reply in the linked GitHub issue.

from tor2web.

Imho we should consider carefully the suggested options, in the two different use-cases of access logs:

• Debugging
  Debug non-working condition / issues
  This require full-logging, also of referral, now missing, except the source IP address
• Statistics
  Those can be much less invasive, however we may not be able to use access-log for statistics purposes.
  For example the geo-ip referentiation of clients accessing tor2web cannot be found from access log, but must be recorded at runtime in an aggregated way as per #13 .

Said that, i have a proposal:

• Do we want to keep access log even more detailed (just removing source ip address) but for uses only for debugging? (Ie: to be disabled during normal operations) and use #13 for statistics just collecting the data we are interested in in the final aggregated statistics format we'll define?

from tor2web.

In my opinion it's better to keep access logs and statistics logs separated.
I agree to have a complete Apache standard format filtering out the
client ip only.

I'm going to implement a custom log function to override the Twisted
default one.

from tor2web.

Debug custom log format implemented actually filtering out only the client ip.
f15044d

from tor2web.