Comments (17)
Medwynd
commented on June 1, 2024
1
Oh come on. A little thanks for me fixing your problem for you?
Much appreciated for the assist! Sorry not used to leaving anything but the facts in bug reports, force of habit.
from mybgg.
EmilStenstrom
commented on June 1, 2024
Hi! You should be able to install the requirements with pip install -r scripts/requirements.txt -- does that work?
from mybgg.
EmilStenstrom
commented on June 1, 2024
Also, if you update to the latest version, there is an option to automatically sync your games every hour using GitHub Actions.
from mybgg.
Medwynd
commented on June 1, 2024
Hi! You should be able to install the requirements with
pip install -r scripts/requirements.txt-- does that work?
Sorry for the lack of clarity, that is what I meant by "Installing the requirements says decl.xml is up to date".
Using the command you listed says it is installed properly and up to date.
from mybgg.
EmilStenstrom
commented on June 1, 2024
Thanks. That should mean that the command you run does not find the packages you installed with pip. What command are you running the returns the above error?
from mybgg.
Medwynd
commented on June 1, 2024
Ok I seem to have gotten the command to work, I think I was leaving the "python" off the front of it and just using
"scripts/download_and_index.py --apikey XXXXXX". I think I just copy and pasted the line and missed the "python" on the first line. Can you add a line break there in the documentation so the command is all on one line?
Now I get this error:
BGGException: BGG returned status code 429
Traceback (most recent call last):
File "scripts/download_and_index.py", line 77, in <module>
main(args)
File "scripts/download_and_index.py", line 17, in main
extra_params=SETTINGS["boardgamegeek"]["extra_params"],
File "E:\Users\User\Documents\mybgg-git\mybgg\scripts\mybgg\downloader.py", line 34, in collection
**extra_params,
File "E:\Users\User\Documents\mybgg-git\mybgg\scripts\mybgg\bgg_client.py", line 26, in collection
data = self._make_request("/collection?version=1", params)
File "E:\Users\User\Documents\mybgg-git\mybgg\scripts\mybgg\bgg_client.py", line 85, in _make_request
return self._make_request(url, params=params, tries=tries + 1)
File "E:\Users\User\Documents\mybgg-git\mybgg\scripts\mybgg\bgg_client.py", line 85, in _make_request
return self._make_request(url, params=params, tries=tries + 1)
File "E:\Users\User\Documents\mybgg-git\mybgg\scripts\mybgg\bgg_client.py", line 85, in _make_request
return self._make_request(url, params=params, tries=tries + 1)
[Previous line repeated 2 more times]
File "E:\Users\User\Documents\mybgg-git\mybgg\scripts\mybgg\bgg_client.py", line 94, in _make_request
f"BGG returned status code {response.status_code} when "
mybgg.bgg_client.BGGException: BGG returned status code 429
from mybgg.
Medwynd
commented on June 1, 2024
I ran it again and the second time seemed to get further but now I get:
Traceback (most recent call last):
File "scripts/download_and_index.py", line 77, in <module>
main(args)
File "scripts/download_and_index.py", line 34, in main
indexer.add_objects(collection)
File "E:\Users\User\Documents\mybgg-git\mybgg\scripts\mybgg\indexer.py", line 184, in add_objects
self.index.save_objects(games)
File "E:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\site-packages\algoliasearch\search_index.py", line 72, in save_objects
response = self._chunk("updateObject", objects, request_options)
File "E:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\site-packages\algoliasearch\search_index.py", line 527, in _chunk
raw_responses.append(self._raw_batch(requests, request_options))
File "E:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\site-packages\algoliasearch\search_index.py", line 538, in _raw_batch
request_options,
File "E:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\site-packages\algoliasearch\http\transporter.py", line 35, in write
return self.request(verb, hosts, path, data, request_options, timeout)
File "E:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\site-packages\algoliasearch\http\transporter.py", line 72, in request
return self.retry(hosts, request, relative_url)
File "E:\Users\User\AppData\Local\Programs\Python\Python36-32\lib\site-packages\algoliasearch\http\transporter.py", line 92, in retry
raise RequestException(content, response.status_code)
algoliasearch.exceptions.RequestException: Record at the position 354 objectID=bgg146791 is too big size=11931 bytes. Contact us if you need an extended quota
from mybgg.
Medwynd
commented on June 1, 2024
https://boardgamegeek.com/boardgame/146791/shadows-brimstone-city-ancients
146791 is Shadows of Brimstone, perhaps it cant handle that many expansions?
from mybgg.
EmilStenstrom
commented on June 1, 2024
You are getting the strangest error messages :)
The 429 is just because BGG is very flaky. Seems 50% of requests ends up working with them.
The "Record too big" is new for me. Makes sense that too many extensions makes the record too big for algolia. Might be something that can be done to make the records smaller... hmm...
from mybgg.
Medwynd
commented on June 1, 2024
The "Record too big" is new for me. Makes sense that too many extensions makes the record too big for algolia. Might be something that can be done to make the records smaller... hmm...
Can it possibly just truncate the record for now until a proper solution is come up with? I think this blocks any updates of the collection.
from mybgg.
EmilStenstrom
commented on June 1, 2024
Yes, you could try that. Or removing it from your collection temporarily.
from mybgg.
Medwynd
commented on June 1, 2024
Since these all share expansions I had to remove both of these to get the upload to work
https://boardgamegeek.com/boardgame/146791/shadows-brimstone-city-ancients
https://boardgamegeek.com/boardgame/150997/shadows-brimstone-swamps-death
Also it is interesting that the command says
Indexed 456 games and 1273 expansions in algolia, and removed everything else.
But when I look in Algolia under indices it says
No. records 443
Is some discrepancy there expected?
from mybgg.
EmilStenstrom
commented on June 1, 2024
Ok, so now I've took some of my free time to look at this.
- I've added a linebreaks in the docs as you suggested. Hoping this might save someone in the future.
- I'm now capturing the 429 error, which means we are making requests to BGG too fast. When this happens, the script will pause 30 seconds and then continue. Adding --debug to the commands prints an info message when this happens.
- To make sure all Shadows of Brimstone expansions can be added, I've removed the game name from the beginning of the expansion name. The only place it shows up is in the expansion list, and it's fairly redudant there, so I think this is a nice tradeoff. I've tried adding all Brimstone expansions, and things seem to work well.
from mybgg.
EmilStenstrom
commented on June 1, 2024
ping @Medwynd
from mybgg.
Medwynd
commented on June 1, 2024
Was able to successfully pass and upload.
When this happens, the script will pause 30 seconds and then continue.
This did not happen for me, the script just ended.
Traceback (most recent call last):
File "scripts/download_and_index.py", line 77, in <module>
main(args)
File "scripts/download_and_index.py", line 17, in main
extra_params=SETTINGS["boardgamegeek"]["extra_params"],
File "D:\mybggnew\mybgg\scripts\mybgg\downloader.py", line 34, in collection
**extra_params,
File "D:\mybggnew\mybgg\scripts\mybgg\bgg_client.py", line 26, in collection
data = self._make_request("/collection?version=1", params)
File "D:\mybggnew\mybgg\scripts\mybgg\bgg_client.py", line 85, in _make_request
return self._make_request(url, params=params, tries=tries + 1)
File "D:\mybggnew\mybgg\scripts\mybgg\bgg_client.py", line 85, in _make_request
return self._make_request(url, params=params, tries=tries + 1)
File "D:\mybggnew\mybgg\scripts\mybgg\bgg_client.py", line 85, in _make_request
return self._make_request(url, params=params, tries=tries + 1)
[Previous line repeated 2 more times]
File "D:\mybggnew\mybgg\scripts\mybgg\bgg_client.py", line 101, in _make_request
f"BGG returned status code {response.status_code} when requesting {response.url}"
mybgg.bgg_client.BGGException: BGG returned status code 429 when requesting
from mybgg.
EmilStenstrom
commented on June 1, 2024
Oh come on. A little thanks for me fixing your problem for you?
from mybgg.
EmilStenstrom
commented on June 1, 2024
@Medwynd As a open source maintainer, thanks is the only thing you get for toiling away over the years. Thanks!
from mybgg.
Related Issues (20)
- Game descriptions getting truncated by a colon? HOT 4
- code organization suggestion HOT 2
- download_and_index.py errors HOT 1
- Error when running scripts/download_and_index.py HOT 2
- Sorting HOT 7
- Feature request - Use version art of games owned HOT 4
- Allow users to set the sort order in the UI HOT 2
- Is there a way to change back to the old board game backgrounds? HOT 4
- Moving filters back to the right side HOT 1
- Issue pulling down games from bgg HOT 4
- Feature Request: Add accessories to detail view of an item HOT 1
- UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images HOT 2
- How did you make one request instead of one per game? HOT 2
- Feature Request: Add a filter-by-minimum-recommended-age (listed, community) HOT 3
- Replace Algolia with lunr.js
- Table tooltips do not respect screen width
- Incorrect Player Count HOT 10
- BGG throttling the public API? HOT 2
- Issue with adding game to collection, objectID=bgg150997 is too big size=10763/10000 bytes HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mybgg.