Notice on ship drop data storage and dumping about poi-server HOT 24 OPEN

Ping @Gizeta @Acrone, the server url is updated to http://poidb.0u0.moe/. The old http://citrus.dazzyd.org may be unusable someday.

from poi-server.

pinging @Acrone because you've posted a question before

pinging @Gizeta

from poi-server.

Got it.

from poi-server.

Exciting stuff. Thanks for the ping!

from poi-server.

@yukixz There are few issues with the site / dumps that just pop up recently.

1. SSL certificate is expired on poi.io and poi.0u0.moe

$ curl -vIk https://poi.io 2>&1 | grep 'expire date'
*  expire date: Sep 15 02:53:34 2019 GMT

$ curl -vIk https://poi.0u0.moe 2>&1 | grep 'expire date'
*  expire date: Sep 15 02:53:34 2019 GMT

# just in case, not expired yet
$ curl -vIk https://poidb.0u0.moe 2>&1 | grep 'expire date'
*  expire date: Oct  4 02:06:07 2019 GMT

2. https://poidb.0u0.moe/dump/ has no dropshiprecords dump for August.

3. Current https://poi.0u0.moe/dump/dropshiprecords.gz is just records from Aug 31 ~ Sep 03, missing the rest of the month's records.

from poi-server.

@gakada so terrible...

1. fixed.
2. seem data of august is lost for unknown reason.
3. ->2

from poi-server.

Seems like there is a significant decline in data due to decline in reports from poi.

from poi-server.

@KagamiChan can it be due to poooi/poi@dd8ab77? Right now plugin-report is sending data via http://api.poi.moe -> https://api.poi.moe redirect, while previously it was (and still can be) http://poi.0u0.moe, maybe that is affecting some users in a weird way.

from poi-server.

Thanks for catching this issue

It turns out that request won't follow the redirection for non GET requests, and won't throw either, thus we were not aware of this.

I'll update the reporter plugin to use https instead

from poi-server.

@gakada The new plugin is live, could you please help monitor 1 or 2 days later?

from poi-server.

Yeah, I will check if % goes back up in next few dumps.

from poi-server.

@gakada Thanks, I've confirmed that we have data from newer version of poi

from poi-server.

It is back to normal mostly, 2-3x more reports from poi and 2x for the percentage.

from poi-server.

@yukixz having problems with the current dump for the last few days:

tmp $ curl -sI https://poi.moe/dump/dropshiprecords.gz | grep content-length
content-length: 128815104
tmp $ wc -c dropshiprecords.gz
128815104 dropshiprecords.gz
tmp $ sha1sum dropshiprecords.gz
a6c6b569f9e287340e2293a2db3868653c7f6765  dropshiprecords.gz
tmp $ gunzip dropshiprecords.gz

gzip: dropshiprecords.gz: unexpected end of file

from poi-server.

@gakada Dumping again now.

from poi-server.

Thanks, now it is fine.

from poi-server.

@yukixz It looks like current dumps are just giving home page HTML.

For example, https://poi.moe/dump/passeventrecords.gz just shows the poi home page. Happens both in browser and with curl.

from poi-server.

Hi @Tibowl

please try https://api.poi.moe/dump/passeventrecords.gz

we've made a separation for website and api server, I thought the change has been announced but seems not yet. sorry for that

Btw, it is a known issue that due to network incompatibility, we have less reports since last December

from poi-server.

I added some question marks in https://github.com/poooi/poi-server/wiki

as those seem to be enabled, but I don't know what are the links to their current dumps (two of them have monthly archived dumps).

from poi-server.

Thanks @gakada

We disabled file listing for security reasons, but I can still log in to the machine and check the files. I'll make it up later and next time you may just let me update that.

from poi-server.

As a follow-up of data loss issue in Dec 2019, it turns out that a POST request could become a GET request with 301 redirection, this may prevent some clients which are still using http scheme

Returning 307 code could solve the issue.
https://rtfm.co.ua/en/http-redirects-post-and-get-requests-and-lost-data/

Also an update for data loss starting from Dec 2020, we've added event and tracing reports in both client side and server side, but for now we don't have any new findings yet

from poi-server.

@KagamiChan it seems quests are not being reported? https://api.poi.moe/dump/quests.gz only has January quests and https://api.poi.moe/api/report/v3/known_quests is empty.

Also reminding that https://github.com/poooi/poi-server/wiki is missing quest reward dumps, etc.

from poi-server.

@gakada ack, will take a look later

from poi-server.

@gakada There were multiple issues, I've fixed them and restored back some data sets

quests should be ready with next daily dump, and I'll add rewards later

from poi-server.