Consider thread safety for sqlite about rio-mbtiles HOT 5 CLOSED

@dazza-codes sqlite is only used in the main process in a single thread. The child processes only do processing of imagery.

Can you be more specific about the deadlock? I've never encountered one. Is it the one described here https://bugs.python.org/issue35267 or a different one?

from rio-mbtiles.

Dunno for sure. While processing 1000s of GeoTIFFs on AWS-batch, using RGBA inputs that are fairly big (esp. uncompressed) and zoom levels of about 4..14, the conversions get hung up sometimes. So far, I've forked and sacrificed multiprocessing speed for reliability. While I have a bit of time, revisiting this speed tradeoff to learn more about the project and mess it up 😁 Maybe my mental model of the process is not entirely clear; I probably need to learn to generate a sequence diagram for this or something. I don't know whether the process pool/worker is still active when sqlite is accessed to write out the result of each call to process_tile. There is no explicit termination of the process pool and no context manager at work. I considered just setting the worker-tasks to None so that all the workers live as long as the pool because maybe something is wacky in sharing the global src when the process cleans up after 100 tasks, I dunno. I started to wonder what happens with a map_async callback pattern that might try to save results as they return, but that would mess with sqlite thread safety if that's in the callback handler. I'm also considering whether each process_tile can just throw out a tmp-file (give reproject a tmp-file instead of a MemoryFile) and always return None, then gather up all the tmp-file tile outputs into sqlite after they're all done. Also curious about using dask.delayed in case it has better memory management for large inputs and 1000s of tiles. I see some new work with futures, will take a look at that 👓

from rio-mbtiles.

Explored various options and trade-offs in

• https://github.com/dazza-codes/rio-mbtiles/branches
• need to test several options with larger inputs and compare against current master

from rio-mbtiles.

There is an interesting comment in https://docs.python.org/3/library/concurrent.futures.html

Changed in version 3.3: When one of the worker processes terminates abruptly,
a BrokenProcessPool error is now raised. Previously, behaviour was undefined
but operations on the executor or its futures would often freeze or deadlock.

This behavior might also apply to Pool when the worker init fails (don't know for sure).

from rio-mbtiles.

I'm closing for now. The sqlite3 module is only used from a single thread. Python's multiprocessing is more likely to be the cause of deadlocks.

from rio-mbtiles.