API returns unreliable number of records when using unknown query parameters about goatcounter HOT 7 CLOSED

Do you have an example script? I ran the following a few times over the span of a few days:

#!/bin/zsh

repeat 30; do
	curl -sq 'https://goatcounter.goatcounter.com/api/v0/paths?limit=100' \
		-H 'Content-Type: application/json' \
		-H 'Authorization: Bearer [..]' | grep -c '"path"'
	sleep 1
done

Which is a bit crude, but should be good, and that always seems to return 100 as expected.

Are you using goatcounter.com or self-hosted?

from goatcounter.

Hey Martin, appreciate you looking at this. Yes, self-hosted.

Ran a bash script like yours just now, as well as a ruby script (below), both look good at the moment. I'll run it again a couple times over the weekend.

If it all looks good, I'll have to assume it was an issue with my consuming code that was outputting the logging in my original issue and I'll slink away with my tail between my legs ;).

require "http"

idx = 0
more = true
while more
  response = HTTP
    .auth("Bearer ...")
    .get("https://stats.domain.com/api/v0/paths?limit=200&after=#{200 * idx}")
  puts response.parse["paths"].count
  more = response.parse["more"]
  idx += 1
end

from goatcounter.

Are you using SQLite or PostgreSQL? It's possible it only shows with SQLite?

I ran it against my "self-hosted" SQLite instance, and that seems to work as well – that does use the latest master branch though.

Slightly improved script which also does pagination:

#!/bin/zsh

integer sz=100
repeat 30; do
	integer after=0
	repeat 20; do
		url="https://stats.arp242.net/api/v0/paths?limit=$sz&after=$after"
		print -nf '%-60s' $url
		data=$(curl -sq $url \
			-H 'Content-Type: application/json' \
			-H 'Authorization: Bearer [..]')
		jq -M '.paths | length' <<<$data

		[[ $(jq '.more' <<<$data) != 'true' ]] && break
		after+=$sz
	done

	sleep 1
done

from goatcounter.

On SQLite.

I figured it out, it's my bad, but my code was inserting start and end parameters, which are not supported for this endpoint, along with the limit and after.

You can see the inconsistent size of records returned with this:

#!/bin/zsh

repeat 5; do
	curl -sq 'https://goat.domain.com/api/v0/paths?start=2023-01-01&end=2023-12-01&limit=200' \
		-H 'Content-Type: application/json' \
		-H 'Authorization: Bearer ...' | grep -c '"path"'
	sleep 1
done

returns

20
200
20
20
200

Strange that the record size is inconsistent, but not a bug as I'm not sending a valid API request.

Thanks for the help Martin.

from goatcounter.

Right – it stops parsing the URL once it encounters an unknown field. Because this is a map the order is undefined, so it depends on what it parsed first.

This should be fixed in the library GoatCounter uses: https://github.com/monoculum/formam

from goatcounter.

Thanks Martin, will close issue as it's not a bug.

from goatcounter.

will close issue as it's not a bug

I think it is :-) A simple mistake or typo really shouldn't introduce this kind of weird behaviour, it should either outright ignore the unknown parameters or return "400: unknown parameter: start". I can see myself making this exact same mistake and having the exact same confusion in the future.

from goatcounter.