ExecuteSQL issues about bindings-gdal HOT 10 CLOSED

One thing I'm wondering tho, is that for now it seems I have to call OGR.openReadOnly on each request. I wonder if there is an overhead for this? I tried to open it in the main thread only once, but because of the phantom types you mentioned I seem to be unable to get it out of runGDAL.
I'm not sure if this is a performance issue, since I don't know how Gdal works internally.

Yes, there's probably a noticeable overhead for opening the datasource on every request. They could be cached in a resource pool which ensures exclusive access to them since they're not safe to share among concurrent threads. However, as you have seen, there's no easy way to store these pointers for subsequent requests since they can't escape the runGDAL region (by design, since out of this region they're already closed and deallocated).

Perhaps some unsafe* functions could be exported to obtain a raw pointer to datasets and un-register them from the cleanup machinery for this use-case.

from bindings-gdal.

Hi.

Strange, this compiles fine on my system with 7.10.3:

{-# LANGUAGE ScopedTypeVariables, OverloadedStrings #-}
import GDAL
import OGR
import qualified Data.Conduit.List as CL

main :: IO ()
main = do
    withGDAL $ runGDAL_ $ do
      ds <- OGR.openReadOnly "ne_10m_admin_0_countries.shp"
      let src = sourceLayer_ $ executeSQL DefaultDialect "SELECT * FROM fondo" Nothing ds
      (fs :: [Feature]) <- runOGR (src $$ CL.consume)
      liftIO (print fs)

What version of GHC are you using?

Also, try inlining src inside runOGR, I suspect type inference is getting confused since there's an l phantom type variable to prevent layer pointers from exiting the runOGR region.

from bindings-gdal.

Wow that was actually the fault of {-# LANGUAGE GADTs #-}.
Removing it made it works.

from bindings-gdal.

Thank you @albertov.
Do you plan on releasing this lib on Hackage?

from bindings-gdal.

Eventually yes but i'd like to write some haddocks first and do some cleanup (like extracting the conduit functionality into another package, i've just created #4 for it)

from bindings-gdal.

Alright, for now I'll be using it for my project as a Git repo with Stack.
Thanks for you lib btw :)

from bindings-gdal.

@albertov I think there is a memory issue somewhere. Calling the above code twice in a row makes the second fail with a segmentation fault. Or should I close something myself?

Edit: seems like I can only call withGDAL once. Maybe gdal is not thread safe?

from bindings-gdal.

seems like I can only call withGDAL once. Maybe gdal is not thread safe?

withGDAL should only be called once from the main thread since it it registers and deregisters the driver manager which is a process-wide singleton. It is not thread-safe, calling it from several threads will surely cause a segfault.

runGDAL is however thread-safe and can run GDALactions concurrently in different threads (I use this pattern extensively). However, there are some caveats since GDAL itself is mostly thread-safe but not completely. The bindings try make it hard to express unsafe behaviour using the public API (eg: using an existential phantom type to prevent sharing a dataset among threads/GDAL-actions) but it is not bullet-proof. For example, it wont prevent you from writing to geotiff dataset A from one thread and read from geotiff dataset B from another thread, which is not safe. (I work around this using the API proxy

Thanks for you lib btw :)

You're welcome :) Thanks for the feedback!

from bindings-gdal.

@albertov This is what I ended up doing. Works pretty well so far.
I'm actually doing a web service (just an API) that queries a shapefile.

One thing I'm wondering tho, is that for now it seems I have to call OGR.openReadOnly on each request. I wonder if there is an overhead for this? I tried to open it in the main thread only once, but because of the phantom types you mentioned I seem to be unable to get it out of runGDAL.
I'm not sure if this is a performance issue, since I don't know how Gdal works internally.

from bindings-gdal.

I will for now close this issue. I will reopen one if I think that not being able to store the OGR.openReadOnly pointer is a performance issue in my case.

from bindings-gdal.