I’m trying to change the value of the parameter “interval” in the AtlasCreateRequest function, but the web interface of Ripe is still displaying 900s interval time ( the default value) . I need to increase the interval value so it creates measurements every 12h . Thank you for your help.

The object returned by ProbeRequest does not have total_count set even when a non-zero number of probes have matched the filter. Here is the version information:

Name: ripe.atlas.cousteau
Version: 0.9.2
Summary: Python wrapper for RIPE Atlas API
Home-page: https://github.com/RIPE-NCC/ripe-atlas-cousteau
...
Location: /Library/Python/2.7/site-packages

When I have multiple AtlasCreateRequest(), the api only seems to POST the first three requests.

I was wondering if there was a way around that/what would be the best way to create ~100 measurements in one script.

I needed a list of probes in Canada, so I did this:

list(ProbeRequest(country="ca"))

Then I waited a Very Long Time until I just killed the process.

After some poking at the source code, I realised that country= was an invalid filter, so then I went looking for the right value in the documentation. The answer was not immediately obvious and Jasper pointed me to the reference manual which I didn't find while looking.

I finally realised that I needed country_code=, which is fine, but it'd be nicer if Cousteau was smart enough to either (a) tell me that country= was invalid (rather than just grabbing all of the probes) or (b) pointed out my mistake and did some fuzzy matching to guess what I wanted (or at least present all options).

Could you please support "resolve_on_probe"? I am testing some anycast domain name, but resolving on ripe atlas servers won't give me the real end user data.

I'm using fetching latest results described here: https://ripe-atlas-cousteau.readthedocs.io/en/latest/use.html#fetching-latest-results

% python3 latest_results.py
{'error': {'detail': 'Method "GET" not allowed.', 'status': 405, 'title': 'Method Not Allowed', 'code': 104}}

Method is not allowed. Is this code outdated?

I'm currently working on an academic project which uses the RIPE Atlas API heavily. Many lookups are performed, and the process performing these lookups lives for a long time (many hours).

Unfortunately, it appears that socket connections aren't getting closed by the socket.io library used by this module, which means that eventually my Python process runs out of open file descriptors. I can confirm this by running my tool and then examining the file list in htop - there are thousands of the form XXX@XXX -> stream.atlas.ripe.net:80:

8    IPv4                                      ***:44210->stream-tmp.ripe.net:80
    9    IPv4                                  ***:44212->stream-tmp.ripe.net:80
   10    IPv4                                  ***:44276->stream-tmp.ripe.net:80
   11    IPv4                                  ***:44278->stream-tmp.ripe.net:80
   12    IPv4                                  ***:44332->stream-tmp.ripe.net:80
   13    IPv4                                  ***:44400->stream-tmp.ripe.net:80
   14    IPv4                                  ***:44526->stream-tmp.ripe.net:80
   15    IPv4                                  ***:44446->stream-tmp.ripe.net:80
   16    IPv4                                  ***:44554->stream-tmp.ripe.net:80
   17    IPv4                                  ***:44402->stream-tmp.ripe.net:80
   18    IPv4                                  ***:44618->stream-tmp.ripe.net:80
   19    IPv4                                  ***:44650->stream-tmp.ripe.net:80
   20    IPv4                                  ***:44414->stream-tmp.ripe.net:80
   21    IPv4                                  ***:44448->stream-tmp.ripe.net:80
   22    IPv4                                  ***:44580->stream-tmp.ripe.net:80
   23    IPv4                                  ***:44652->stream-tmp.ripe.net:80
   24    IPv4                                  ***:44424->stream-tmp.ripe.net:80
   25    IPv4                                  ***:44426->stream-tmp.ripe.net:80
   26    IPv4                                  ***:44582->stream-tmp.ripe.net:80
   27    IPv4                                  ***:44608->stream-tmp.ripe.net:80
   28    IPv4                                  ***:44610->stream-tmp.ripe.net:80
   29    IPv4                                  ***:44620->stream-tmp.ripe.net:80
   31    IPv4                                  ***:44556->stream-tmp.ripe.net:80
   32    IPv4                                  ***:44416->stream-tmp.ripe.net:80
   33    IPv4                                  ***:44624->stream-tmp.ripe.net:80
   34    IPv4                                  ***:44626->stream-tmp.ripe.net:80
   35    IPv4                                  ***:44656->stream-tmp.ripe.net:80
   36    IPv4                                  ***:44658->stream-tmp.ripe.net:80
   37    IPv4                                  ***:44662->stream-tmp.ripe.net:80
   38    IPv4                                  ***:44664->stream-tmp.ripe.net:80
...

this goes on for many thousands of lines, until my Python process crashes due to running out of file descriptors.

The code that I'm using is along the lines of the following snippet, essentially the same as is described in the docs for this module:

results = []

def on_result_response(*args):
    # append result to results array here...

channel = "atlas_result"
atlas_stream = AtlasStream()
atlas_stream.connect()
atlas_stream.bind_channel(channel, on_result_response)

(is_success, response) = atlas_request.create()

stream_parameters = {"msm": response["measurements"][0]}
atlas_stream.start_stream(stream_type="result", **stream_parameters)

logger.debug("Waiting for results")

tries = 0
while len(results) < len(probes) and tries < 20:
    atlas_stream.timeout(5)
    tries += 1

logger.debug("Got result, exiting")

atlas_stream.disconnect()

return [result for result in results if result]

It seems you are using https://github.com/invisibleroads/socketIO-client/, a library which hasn't been updated for many years, and has a long list of issues - including invisibleroads/socketIO-client#147, which describes the same issue I'm having here. A more modern library exists at https://github.com/miguelgrinberg/python-socketio, it might be worth using this instead.

I note that there is also AtlasResultsRequest which can be used to fetch results, but (in my very anecdotal experience) this seems to get results slower - not sure whether that is expected behaviour or not.

In an Archlinux up to date, i get this error when I try to install ripe-atlas-tools.
To reproduce it I just try to install ripe-atlas-tools form AUR with yaourt.

Copying ripe.atlas.cousteau.egg-info to /var/tmp/yaourt-tmp-franc/aur-python-ripe-atlas-cousteau/pkg/python-ripe-atlas-cousteau/usr/lib/python3.5/site-packages/ripe.atlas.cousteau-1.0.6-py3.5.egg-info
Installing /var/tmp/yaourt-tmp-franc/aur-python-ripe-atlas-cousteau/pkg/python-ripe-atlas-cousteau/usr/lib/python3.5/site-packages/ripe.atlas.cousteau-1.0.6-py3.5-nspkg.pth
running install_scripts
rm: cannot remove 'ripe/pycache': No such file or directory
rm: cannot remove 'ripe/atlas/pycache': No such file or directory
==> ERROR: A failure occurred in package().
Aborting...
==> ERROR: Makepkg was unable to build .

There are two issues with proxies:

1. AtlasCreateRequest is using requests and based on current implementation of request.py there was no easy way of using proxy.
2. We wanted to fetch results using the stream api and based on current implementation of stream.py and the fact that it is using SocketIO we can't use proxy.

We managed to work around the first issue by setting an env variable for the proxy and that worked, so creating request works. However, fetching results with stream api and SocketIO proved to be problem with proxy.

It looks like you can use proxies with SocketIO but this would need adding: https://pypi.python.org/pypi/socketIO-client

Hi there, thank you for the project, I just noticed (during packaging this library for NixOS) that latest release is tagged 1.4.2 instead of v1.4.2, which is different from the usual naming convention from what I gather.

That would be awesome if it could be tagged v1.4.2 too!

We'd like to set the user-agent for Magellan-based requests to something like RIPE Atlas Magellan, so we need Cousteau to allow for this.

We need better and more verbose docs.

As of writing, ripe-atlas-cousteau depends on python-socketio[client]<5, while version 5 was released in December 2020 (see: https://pypi.org/project/python-socketio/). Do you see any chance for effectively supporting python-socketio[client]>=4,<6?

Hi there,

While fetching stream results with cousteau, I got the following error message in my terminal:
WARNING:root:atlas-stream.ripe.net:80//stream/socket.io [connection error] unexpected status code (400 {"code":1, "message":"Session ID unknown"})

And cousteau hangs forever (notice that I haven't set a timeout) so there is no way to recover from this error. I have googled the error and it seems that this could be solved on the server side (e.g. socketio/socket.io#1739 and socketio/socket.io-client-swift#237)

Hey guys,

first off: great library guys, really makes working with the atlas nodes easy :-)
Right now I am having some trouble seeting up measurments on nodes, as it keeps exploding in my face due to some problem with the time stamps.
Right now I just want to make sure I didn't make some stupid mistake....

So, I was following your basic examples in your documention and therefore stated the start time by using:

datetime.datetime.utcnow()

Which then gives my this as on output:

Current time UTC is: 2016-02-17 05:42:14.802293  ==  1455655334
Current time is: 2016-02-17 14:42:14.802372  ==  1455687734
Measurement starts at: 2016-02-17 05:44:14.802278  ==  1455655454
--->Shit went wrong
Atlas start: 1455655454
T_Time: 1455655454
{'status': 400, 'detail': 'start_time: Ensure this value is greater than or equal to 1455687970.', 'title': None, 'code': 104}

from a starting a simple DNS query following the example "Creating a measurment" in your docs.

I am currently working in Japan, therefore UTC and my timestamp differ.
So far, no harm done - I can just use my currenty time stamp - meaning

datetime.datetime.now()

I can submit those request, but then my measurments will not start right away but in 8 hours (=time difference between UTC and Tokyo time).
As far as I see, somewhere along the way, something gets mixed up with the time stamps!

No I am not sure if I missinterpreted the documention of your tools somehow and made a mistake along the way or if this is acutally an issue.

e.g. in ripe-atlas-cousteau/ripe/atlas/cousteau/api_listing.py, l.63ff., you reduce several objects to simpler objects, i.e.:

for k, v in self.api_filters.items():
            if isinstance(v, datetime):  # datetime > UNIX timestamp
                self.api_filters[k] = int(calendar.timegm(v.timetuple()))
            if isinstance(v, (tuple, list)):  # tuples & lists > x,y,z
                self.api_filters[k] = ",".join([str(_) for _ in v])

Yet, on my computer, I get this:

>>> import datetime
>>> import calendar
>>> datetime.datetime.utcnow()
datetime.datetime(2016, 2, 17, 5, 33, 34, 832888)
>>> datetime.datetime.utcnow().timetuple()
time.struct_time(tm_year=2016, tm_mon=2, tm_mday=17, tm_hour=5, tm_min=34, tm_sec=16, tm_wday=2, tm_yday=48, tm_isdst=-1)
>>> datetime.datetime.utcnow().strftime("%s")
'1455654879'
>>> calendar.timegm(datetime.datetime.utcnow().timetuple())
1455687298

This explains at least why I get an error when using the normal UTC time...

Anyways, sorry for the long text, hope it is all understandable.
I'd love your oppinion on this :)

Cheers,
Chris

Hello,

I used the code [1] provided here to add some probes to msm ID 3513990: I wanted the system-ipv4-works to be added to the source, but even if the change request was fine I can't find the tag in the web UI.

I did it for 3 different ASNs, I tried also a direct call to the v2 API [2] but I had no success.

As you can see from the msm only the first set of probes that I added via the web UI has the tag.

Could you point me in the right direction to have change requests considering tags?

Thanks

1. the code I used:

from ripe.atlas.cousteau import AtlasChangeSource, AtlasChangeRequest

ATLAS_MODIFY_API_KEY = "MY_CHANGE_KEY"

# Add probes
source = AtlasChangeSource(
        value="1234567890",
        requested=5,
        type="asn",
        tags={
                "include": ["system-ipv4-works"],
                "exclude": ["system-anchor"]
        },
        action="add"
)

atlas_request = AtlasChangeRequest(
        key=ATLAS_MODIFY_API_KEY,
        msm_id=3513990,
        sources=[source]
)

(is_success, response) = atlas_request.create()

print(is_success)
print(response)

2. API call:

curl --dump-header - -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{
 "action": "add",
 "requested": 5,
 "type": "asn",
 "value": "1234567890",
 "tags": {
   "include": [
     "system-ipv4-works"
   ],
   "exclude": []
 }
}' https://atlas.ripe.net/api/v2/measurements/3513990/participation-requests/?key=XXXXXXXX

Hi,

ripe-atlas-cousteau is a very useful tool, I use it to find some special data, however, a request return the results on a array, which is a structure different to this one of archived data by RIPE Atlas. I am working to apply automatic processing to :

1. data from API REST (of RIPE ATlas)
2. data archived before 15 mars 2018 ( many files as one for : builtin, udm, etc)
3. data archived after 15 mars 2018 ( one file by one hour)

But:

Archives in 2. and 3. have the same structure: one record by line. Ex: one traceroute by line as:
traceroute1
traceroute2
...
Archives in 1. has one array, the members are the records. Ex: [traceroute1, traceroute2, ...]

Can I customize the structure of the results? Is there some parameters that I miss using them?

Thanks in advance.

Hayat

Before submitting a PR, I wanted to ask if the maintainers or community are interested in async version of HTTP requests and WebSocket operations (for streams). I have a proof of concept under my fork.

Reading the setup.py, I see that there is a dependency on websocket-client<0.99, I have two questions regarding that:

(1) is websocket-client directly used as a dependency of this project? It appears to me this is the responsibility of socketio-client, am I right assuming that?
(2) moving forward with this assumption, it looks like there was a will to pin the transitive dependency to avoid a bug or something, is it right? Do you know if the desired behavior is now in the latest version of the library, can the pin be removed or documented instead?

(for context, this is still about packaging it in NixOS, I had to patch out the <0.99 constraint as socketio-client have a quite recent version of that!)
Many thanks!

Hi,

Can I have some details about the value that can take status in filters and what does mean?
For example :
filters = {"status": 1}
measurements = MeasurementRequest(**filters)
Thanks.

Hayat

Hello,

I have a small script to stop measurements, but it’s not working anymore :

$ ./stop_atlas_mesure.py -id 31605474
Error - Method "DELETE" not allowed.

I'm using «AtlasStopRequest» from https://ripe-atlas-cousteau.readthedocs.io/en/latest/use.html#stopping-measurement

Version :
ripe.atlas.cousteau (1.5.0)

$ cat stop_atlas_mesure.py
#!/usr/bin/env python3.6
# coding: utf-8

# Necessite la variables "atlas_api_key" dans le fichier credentials

import sys
import os
from optparse import OptionParser
from ripe.atlas.cousteau import AtlasStopRequest
sys.path.insert(0, '/etc/accounts/')
from credentials import *

# Definition des arguments
usage = "usage: %prog -id ID"
parser = OptionParser(usage)
parser.add_option("-i", "--id", dest="id", default="", help="ID Atlas de la mesure")
(options, args) = parser.parse_args()
id = options.id

if(id == ""):
        parser.error("Please provide an ID")

# stop la mesure
atlas_request = AtlasStopRequest(msm_id=id, key=atlas_api_key)
(is_success, reponse) = atlas_request.create()

if is_success:
    print("Ok - ID " + id + " stopped.")
else:
        print("Error - " + reponse["error"]["detail"])

Something has changed ?
Thanks

Tried:
(Python3.5, ripe.atlas.cousteau==1.2)
(Python3.5, ripe.atlas.cousteau==1.3)

I'm using the example code from the README.rst file:

from ripe.atlas.cousteau import AtlasStream


def on_result_response(*args):
    """
    Function that will be called every time we receive a new result.
    Args is a tuple, so you should use args[0] to access the real message.
    """
    print(args[0])

atlas_stream = AtlasStream()
atlas_stream.connect()
# Measurement results
channel = "result"
# Bind function we want to run with every result message received
atlas_stream.bind_channel(channel, on_result_response)
# Subscribe to new stream for 1001 measurement results
stream_parameters = {"msm": 4412911}
atlas_stream.start_stream(stream_type="result", **stream_parameters)

# Timeout all subscriptions after 5 secs. Leave seconds empty for no timeout.
# Make sure you have this line after you start *all* your streams
atlas_stream.timeout()
# Shut down everything
atlas_stream.disconnect()

If I'm using the Javascript Code however, this works:

<script src="https://atlas-stream.ripe.net/socket.io.js"></script>
<script>

    // Create a connection (it can be also http on port 80)
    var socket = io("https://atlas-stream.ripe.net:443", { path : "/stream/socket.io" });

    // Subscribe to results coming from all the probes involved in the measurement 4412911
    socket.emit("atlas_subscribe", { stream_type: "result", msm: 4412911 });

    // Declare a callback to be executed when a measurement result is received
    socket.on("atlas_result", function(result){
        console.log("I received ", result);
    });

</script>

In case of the python script, the function on_result_response is not called at all. The console.log() in the javascript code however works fine.

I already started digging a little bit: The port / protocol (http/https) seems not to be the issue. Any ideas?

Hi!

I would like to use Cousteau for launching periodic measurements for one week (every 30 minutes).
However, I can't find a possibility to set the time interval between my measurements. When launching my measurements, they are indeed launched for one week, but with the default time interval between them (so 240 seconds / 900 seconds).

My code looks as follows

for IP in SERVER_IPS:
    pings.append(Ping(af=4, target=IP, description=MEASUREMENT_DESCRIPTION))
    traceroutes.append(Traceroute(af=4, target=IP, protocol='ICMP', description=MEASUREMENT_DESCRIPTION))
atlas_request = AtlasCreateRequest(start_time=datetime.strptime('24 Feb 2017 17:00:00', '%d %b %Y %H:%M:%S'),
                                   stop_time=datetime.strptime('03 March 2017 17:00:00', '%d %B %Y %H:%M:%S'),
                                   key=API_KEY, measurements=pings + traceroutes,sources=[PROBES])
(is_success, response) = atlas_request.create()

I unfortunately didn't find any argument for AtlasCreateRequest() that allows setting the interval to a custom value. Maybe I have missed something in the documentation?

If this functionality is not yet implemented, it would be great to be able to call

atlas_request = AtlasCreateRequest(start_time=datetime.strptime('24 Feb 2017 17:00:00', '%d %b %Y %H:%M:%S'),
                                   stop_time=datetime.strptime('03 March 2017 17:00:00', '%d %B %Y %H:%M:%S'),
                                   interval=1800,
                                   key=API_KEY, measurements=pings + traceroutes,sources=[PROBES])
(is_success, response) = atlas_request.create()

to launch measurements every 30 minutes for the week from 24.02 until 3.3.

After new refactoring for using requests lib 4xx responses don't seem to decode json and return only text.
Old behaviour used to read json for 4xx responses and return the decoded structure.

Hey!

Under normal operation, cousteau works fine. After some hours of running however, it does not receive any data any more. It only receives what looks similar to heartbeats or the like:

Faulty Operation

Process 122648 attached
restart_syscall(<... resuming interrupted call ...>) = 1
recvfrom(8, "\201\1", 2, 0, NULL, NULL) = 2
poll([{fd=8, events=POLLIN}], 1, 60000) = 1 ([{fd=8, revents=POLLIN}])
recvfrom(8, "3", 1, 0, NULL, NULL)      = 1
poll([{fd=8, events=POLLIN}], 1, 60000) = 1 ([{fd=8, revents=POLLIN}])
recvfrom(8, "\201\1", 2, 0, NULL, NULL) = 2
poll([{fd=8, events=POLLIN}], 1, 60000) = 1 ([{fd=8, revents=POLLIN}])
recvfrom(8, "3", 1, 0, NULL, NULL)      = 1
poll([{fd=8, events=POLLIN}], 1, 60000      

I get similar system calls when its under normal operation but with data inbetween:

Normal Operation

Process 85326 attached
restart_syscall(<... resuming interrupted call ...>) = 0
poll([{fd=9, events=POLLIN}], 1, 1000)  = 1 ([{fd=9, revents=POLLIN}])
recvfrom(9, "\201~", 2, 0, NULL, NULL)  = 2
poll([{fd=9, events=POLLIN}], 1, 1000)  = 1 ([{fd=9, revents=POLLIN}])
recvfrom(9, "\1\273", 2, 0, NULL, NULL) = 2
poll([{fd=9, events=POLLIN}], 1, 1000)  = 1 ([{fd=9, revents=POLLIN}])
recvfrom(9, "42[\"atlas_result\",[{\"af\":4,\"prb_"..., 443, 0, NULL, NULL) = 443

Restarting the script resolves the issue immediately.

The code (nothing fancy) goes something like this:

measurement = config['measurement']

atlas_stream = cousteau.AtlasStream()
atlas_stream.connect()

write_probe_measurement = functools.partial(write_measurement_to_kafka, kafka_producer)
atlas_stream.bind_channel("atlas_result", write_probe_measurement)
atlas_stream.start_stream(
    stream_type="result",
    msm=measurement,
)

try:
    atlas_stream.timeout()
except KeyboardInterrupt:
    atlas_stream.disconnect()

It clearly is not the Atlas' problem, since restarting fixes the issue. If the Atlas were broken, restarting would not help.

websocket-client document says:
it needs http_proxy_auth attribute when proxy requiring authentication as described below.
https://websocket-client.readthedocs.io/en/latest/examples.html#connecting-through-a-proxy

but that attribute is not involved in below code:

  File "/usr/local/lib/python3.11/site-packages/ripe/atlas/cousteau/stream.py", line 135, in connect
    self.ws = websocket.create_connection(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/websocket/_core.py", line 601, in create_connection
    websock.connect(url, **options)
  File "/usr/local/lib/python3.11/site-packages/websocket/_core.py", line 244, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/websocket/_http.py", line 132, in connect
    sock = _tunnel(sock, hostname, port, auth)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/websocket/_http.py", line 299, in _tunnel
    raise WebSocketProxyException(
websocket._exceptions.WebSocketProxyException: failed CONNECT via proxy status: 407

A call to AtlasCreateRequest with an interval argument for recurring measurements does not set the measurement interval. The created measurement ends up having the default measurement interval of 240s.

There is a small error in the documentation. In the "fetch results" section the coma are missing in the kwargs dictionary (and the datetime package is not imported). I guess it should be:

from ripe.atlas.cousteau import AtlasResultsRequest
from datetime import datetime

kwargs = {
"msm_id": 2016892,
"start": datetime(2015, 05, 19),
"stop": datetime(2015, 05, 20),
"probe_ids": [1,2,3,4]
}

is_success, results = AtlasResultsRequest(**kwargs).create()

if is_success:
print(results)

Hello, I'm using the API's meta data from Measurement to identify the number of iteration (using the timestamp of a result, the start time and the interval).

When I was testing this, the differences didn't make sense, since I'm using UTC values and I assumed msm.start_time would return a timestamp in UTC (as seen on, for example, https://atlas.ripe.net/api/v2/measurements/4496949/), but it returns a naive datetime object in localtime. (My local time is GMT+2:00 DST). Also, I'm using Python 3.2 so I don't have datetime.timestamp().

I couldn't found specification of this in the docs, so I took a look at the code. I found in api_meta_data.py that the method populate_times() uses datetime.fromtimestamp() that returns the local date and time, instead of datetime.utcfromtimestamp().
I think maybe using UTC values would be better since the timestamps in Ripe are in UTC.

Thank you.

compared to the API reference at https://atlas.ripe.net/docs/apis/rest-api-reference/ with
current master of ripe-atlas-cousteau, cousteau's Probe does not support options_field.

more over I found that measurement API on server side (atlas.ripe.net/api/v2/measurement/{pk}) supporting optional_field while testes with official WebUI at https://atlas.ripe.net/measurements/{pk}

actually, we have workaround which speficifying fields=[all fields for default and optional] at creation of Probe (and Measurement) but it is not smart to list up all default fields for requesting optional fields.