sandik's Introduction

sandik

oy ve ötesi sitesi üzerinden sandık verilerini toplamak için script

Install

git clone https://github.com/kiliczsh/sandik.git
cd sandik
python -m venv .venv
pip install - requirements.txt

Run

# To get city, district, neighborhood and school list

python main.py
Enter city plate: 69
# 69 is the plate of Bayburt
# Check sample/BAYBURT.json for sample output

# To get results of a school

python tutanak.py
python tutanak.py
Enter school id: 184742
# 184742 is the id of the school
# Check sample/school_184742.json for sample output

# To get bulk results

python bulk_tutanak.py
Enter city json path: ANTALYA.json
# In this case, results will be written to data/7/school_{30535, 30529, 30540, ...}.json

Notes

SLEEP_TIME = 1 in main.py and tutanak.py is the sleep time between requests. You can change it if you want.

Contribution

Feel free to contribute ☘️

sandik's People

Contributors

Stargazers

Watchers

sandik's Issues

Fetching images and uploading to an image bucket

@BYK @egeakman after @shazar has shared full database we can try to obtain images from urls, what do you think?

aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected

cc: @BYK

INFO:root:Sleeping before retry for https://api-sonuc.oyveotesi.org/api/v1/cities/65/districts/819/neighborhoods/42111/schools (try: 6)...
INFO:root:Sleeping before retry for https://api-sonuc.oyveotesi.org/api/v1/cities/65/districts/819/neighborhoods/42069/schools (try: 6)...
INFO:root:Sleeping before retry for https://api-sonuc.oyveotesi.org/api/v1/cities/65/districts/818/neighborhoods/42045/schools (try: 6)...
Traceback (most recent call last):
  File "/Users/xxx/Development/sandik/main.py", line 272, in <module>
    asyncio.run(main())
  File "/Users/xxx/.pyenv/versions/3.11.1/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/xxx/.pyenv/versions/3.11.1/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/.pyenv/versions/3.11.1/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/xxx/Development/sandik/main.py", line 258, in main
    city = await City(
           ^^^^^^^^^^^
  File "/Users/xxx/Development/sandik/main.py", line 186, in download
    self.districts = await asyncio.gather(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/Development/sandik/main.py", line 139, in download
    self.neighborhoods = await asyncio.gather(
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/Development/sandik/main.py", line 103, in download
    for school in await self.fetch(self.schools_url)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/Development/sandik/main.py", line 26, in fetch
    async with self.session.get(url, raise_for_status=True) as response:
  File "/Users/xxx/Development/sandik/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/Users/xxx/Development/sandik/.venv/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/Users/xxx/Development/sandik/.venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 899, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxx/Development/sandik/.venv/lib/python3.11/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected