Allow options for reports about ripe-atlas-tools HOT 15 CLOSED

I would suggest using a local cache to store IP-address/ASN mappings, something similar to my ipdetailscache library (it gathers info from RIPEStat API), in order to speed up resolution/rendering of reports. Mine depends on Google Python IP address manipulation library (ipaddr) that would be another dependency to be added to the list. A new one built on IPy would be better IMHO.
@danielquinn: if I'm not mistaken you wrote a similar cache too, right?

from ripe-atlas-tools.

I don't have a cache like this, but I believe that one of us over here does. Your suggestion is a good one though. Currently, the reports aren't smart enough to do IP/ASN lookups, but when they do, a unified caching framework within the tool should be there.

from ripe-atlas-tools.

Yes, I did write a similar cache that does lookups in Ripestat and
Openipmap for both ASN and geolocation lookups:
https://github.com/emileaben/ixp-country-jedi/blob/master/lib/Atlas/IPInfoCache.py

prototype, but it works. could probably be improved by some code from Pier
Carlo Chiodi (private range IPs etc.), as well as using a Radix structure
to cache prefixes, instead of single IPs for IP2AS type lookups.

On Tue, Sep 22, 2015 at 6:25 PM, Daniel Quinn [email protected]
wrote:

I don't have a cache like this, but I believe that one of us over here
does. Your suggestion is a good one though. Currently, the reports aren't
smart enough to do IP/ASN lookups, but when they do, a unified caching
framework within the tool should be there.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

from ripe-atlas-tools.

I added a simple cache system today that to which we can apply your asn/geolocation/ipwhatever lookups. Basically the interface is like this:

from ripe.atlas.tools.cache import cache

my_cache_time = 3600
cache.set("my-cache-key", "my value", my_cache_time)
cache.get("my-cache-key")  # returns None if the value has expired

It's totally untested at this point, but I just wanted to keep you guys in the loop.

from ripe-atlas-tools.

In the meanwhile I pushed a testing branch of my ipdetailscache where I added support for IPy: pierky/ipdetailscache@5481d51

from ripe-atlas-tools.

Wouldn't it be better if we use a memoize decorator for that??https://wiki.python.org/moin/PythonDecoratorLibrary#Memoize
I am not really sure I need yet another file on my system. You would cache ofcourse per single process, but I am not convinced it would be much different in our purpose.
A decorator would also be cleaner and more pythonic way to do stuff :)

from ripe-atlas-tools.

I like this pattern, but I also think it should be combined with the caching engine I've already added. The key here is that the goal is to be able to cache output from these methods for future requests so that they don't need to be done again. Since IP:ASN mappings (for example) don't change very often, it makes sense to have a persistent cache file for this sort of thing.

But making use of it in a decorator-like fashion, that's much cleaner than the whole cache.set() thing. I shall make it go exactly like that.

from ripe-atlas-tools.

After a chat with Robert, it looks like we're going to introduce global aggregate options that may or may not be available to a given renderer. I'm still working out what this will look like code-wise, but in terms of user-interface:

# Group the output by country
ripe-atlas report <measurement-id> --aggregate=country

# Group the output by asn
ripe-atlas report <measurement-id> --aggregate=asn

This will require a lot more work on the side of the renderer, and since I don't want to have to repeat code anywhere, I may code this into the parent Renderer class so that subclasses can make use of it somehow. We'll see how that goes.

from ripe-atlas-tools.

I'd indeed love to see that kind of aggregation. It cold be applicable to new measurements as well as fetching results of older ones. So in addition to the above, this could work too:

ripe-atlas ping whatever.com --aggregate=country

It would give you the same thing, based on the just-collected results. If the msm spec happens to restrict probes to a single country, then it'll be a one-liner output...

from ripe-atlas-tools.

@danielquinn I think it would make sense to detach aggregator logic from renderer. The best route for me would be to create Aggregator class/function and apply it to results before you render them. thoughts?

from ripe-atlas-tools.

@astrikos this could work actually. We could create string-like objects to represent each row, and then catch the output, rendering it one way or another based on these flags.

Yeah that might do nicely.

from ripe-atlas-tools.

Will that work in cases where the aggregation is renderer-specific? ie. if
I write a renderer that I want to aggregate-by-country , but the aggregator
logic (Aggregator class) hides the information i need for my renderer, how
do i get to the raw information?
I'd all be for a Aggregator class that can be overridden by the renderer,
so it still can access the raw results if needed.
but maybe i'm misunderstanding.

On Thu, Sep 24, 2015 at 5:58 PM, Andreas Strikos [email protected]
wrote:

@danielquinn https://github.com/danielquinn I think it would make sense
to detach aggregator logic from renderer. The best route for me would be to
create Aggregator class/function and apply it to results before you render
them. thoughts?

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

from ripe-atlas-tools.

@emileaben I think I'd need to see a more detailed use case for something like this, but let's keep things simple to start?

from ripe-atlas-tools.

agree. simple imho would be not abstract away too much from the renderers
(as you may limit the ability of renderers)

On Fri, Sep 25, 2015 at 9:43 AM, Daniel Quinn [email protected]
wrote:

@emileaben https://github.com/emileaben I think I'd need to see a more
detailed use case for something like this, but let's keep things simple to
start?

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

from ripe-atlas-tools.

I think we can close this one, anyone wants to add anything more?

from ripe-atlas-tools.