Add more history stats about starlink-grpc-tools HOT 16 CLOSED

Is it possible to get to a single aggregate dataset (and thus collection script) across history and status do you think?

from starlink-grpc-tools.

It's certainly possible, but there is a tradeoff between making the scripts more complicated and do more things at once vs simpler and do just one thing. I don't mind complexity myself, but the whole point of publishing this stuff is to make it useful for other people, so I'm open to input of which is preferred.

That being said, I'm already heading in the more-complex direction, so I may just take a stab at unifying the scripts and see what it looks like. That would cut down on some of the duplication, at least.

from starlink-grpc-tools.

Took a look and generally the History does not look very friendly in its native form for a time series database (mainly because it has its own timestamps). It would be a boon if we could get the data to that point though. I think this would need to be tackled before merging the statistics anyhow. Thoughts?

Hmm maybe I am misunderstanding the data here ... Forgive me if I am.

from starlink-grpc-tools.

You mean the full second-by-second data? That could, in theory, be unwound from the ring buffers and the timestamps back-computed, but it would be quite a large amount of data. I was assuming that would be too much to store in a database on an ongoing basis, so have been focused on crunching it down to summary stats per time period (such as every hour, or whatever) and storing those. But I don't have much experience with time series databases, so maybe that concern is overblown.

I've had the CVS history script running every hour and dumping the ping drop stats to a file for the past few weeks and am getting meaningful data out of it. That's not a real database, though, so it takes manual effort to make use of it.

from starlink-grpc-tools.

Thinking about this a little more, I've decided I will run an experiment to see how well my local InfluxDB instance holds up on just dumping all the raw data in. InfluxData claims it can handle "ingesting millions of data points per second", so I guess it should be OK.

Actually polling the ring buffer data and pulling out the exact right samples to avoid missing or duplicate data points, as well as avoiding time drift, will be a little tricky, but I think doable. It will require keeping state across subsequent fetches of the history data, so either run in a loop from a single script invocation or saving off state to a file or something (or both). Anyway, I can run the experiment without that level of fidelity.

from starlink-grpc-tools.

After running a hacked-up version of the history InfluxDB script that writes a minute's worth of per-second data every minute for about a day, I've concluded that my concerns about it being too much data were entirely unfounded. I guess I was conflating how much data the entire 12 hour history buffer would be with how often you'd typically want to poll it, but of course if you're polling more often than every 12 hours, you would only need to write the data that is new since last write. Anyway, this works out to about 4MB of data per day.

I still want to have an option for output of summary stats instead of the full data set, since I think the full data set is really only appropriate for sending to InfluxDB, or similar if added in future, but I'm going to focus on getting full data working first. Since it's just InfluxDB and just history data, it'll just be the one script affected, so I'll worry about how to unify the history and status scripts at a later point.

from starlink-grpc-tools.

This is awesome news and exactly what I was hoping to pull out of this project, super excited, I will take a look at the scripts tonight.

from starlink-grpc-tools.

I just checked in a rough cut of this into my working branch. It's not done yet, and I'm not planning to pull it into main until it is, but it should be sufficient to start testing against. To switch to bulk mode, pass the new '-b' option to dishHistoryInflux.py. I will likely change how to invoke this mode in the future, probably if/when I unify the scripts.

The primary thing that is missing right now is logic to track exactly which samples have been processed already, so there is potential for a second or 2 of overlap, or a second 2 of gap, between loop iterations. Also, I haven't decided whether or not to do anything special about the latency value in samples with 100% ping drop. The raw history data just repeats the prior value, and dishHistoryInflux.py is just passing that along right now. That strikes me as a bit wrong, but I'm not sure there's a better solution.

from starlink-grpc-tools.

Does Dishy (gRPC data) not provide its own timestamps? I believe Influx will overwrite existing fields which have the same exact timestamp and append any new fields which would address this.

from starlink-grpc-tools.

Does Dishy (gRPC data) not provide its own timestamps?

Not exactly. It has a sample counter, which can be used for that purpose by converting counter values to some time base, I'm in the process of implementing that now.

I believe Influx will overwrite existing fields which have the same exact timestamp and append any new fields which would address this.

The problem is that the timestamps won't be exact right now, they will have drifted by some number of microseconds, since the script is pulling current system time every loop iteration. This is addressable, too, I just haven't gotten to it, yet.

from starlink-grpc-tools.

Bummer, and nothing, anywhere in the data, even Status or elsewhere, which gives you what Dishy thinks the time is? I see a status - device - timestamp in the app debug that could be used as a reference? Can History and Status be grabbed in the same call?

from starlink-grpc-tools.

Nope. The "device" section of the app debug is about the device the app is running on (ie: the phone, tablet, or whatever), not the dish or router, so that timestamp is presumably the equivalent of the scripts pulling current system time (which is exactly what they are doing).

That's OK, it just makes things a little complicated. It wouldn't be as much fun if it were simple....

from starlink-grpc-tools.

Ahh, I see that now, well that actually surprises me (no time reference) yes, fun! Anyhow would to just trim to the current second and poll every minute for a minute+buffer's data & flush? re latency w/100% ping drop - I agree that seems wrong to me as well ( by definition if you have no reply you would be infinitely latent ).

from starlink-grpc-tools.

Anyhow would to just trim to the current second and poll every minute for a minute+buffer's data & flush?

That's what the script I checked into 'working' is doing (if I understand you correctly). But if the timing loop gets off by more than a second because the system is busy, or whatever, it will miss some samples, and then repeat some on the next iteration (because of how I coded the interval timer). This won't really break InfluxDB, I've accidentally dumped the same 12 hours worth of data into it several time now and it seems fine with it, but it's not ideal.

Anyway, the "current" counter in the history data can be used determine exactly which samples are new. I have that running now and will probably get that checked in tomorrow. There's still a potential for it to get out of sync with real time if the dish's clock is running slightly fast or slow, but I haven't seen that happen, yet.

from starlink-grpc-tools.

... so after all that buildup, the actual change to implement counter tracking was not that complicated. And it actually made the sample range computation logic a little easier to understand vs how I had it before.

There's one more bit of functionality I want to implement before merging this into main branch, which is to track counter state across script invocations. It occurred to me that I can just keep that in the database along with the last written data point, so no need to keep it in a local file. I'm running into some annoying issues with the influxdb-python client lib wrt queries, but I think I can hack around them reasonably enough.

from starlink-grpc-tools.

OK, bulk mode is now in the 'main' branch. I'm not going to close this issue out until summary stats are implemented for the other scripts, or I decide that's not worth pursuing. Before doing that, though, I'm going to take a stab at unifying the status and history scripts, since that will probably involve reworking the argument parsing logic to support selection of different stats/modes.

from starlink-grpc-tools.