consider using a dictionary instead of a tuple for passing SQL query insertion arguments.
See here: https://docs.python.org/2/library/sqlite3.html#sqlite3.Cursor.execute

At parallel access a 'pysqlite2.dbapi2.OperationalError' is thrown because the database is locked. How can parallel access be achieved or the error best be circumvented?

While for Sentinel-1 there is only one source of files, for other sensors different datasets are available. E.g. DELFT and PRARE for ERS. This needs further investigation. Which dataset is best used in which situation? Where to best download them? Within SNAP this is implemented, how to apply this to the gamma module? I.e. own download implementation independent of snap so it can be used to both modules.

This thing is too long. Let's restructure it into some more markdonw files for better readability. E.g. move dependency installation to docs.

All this functionality has grown substantially and might also be used outside of the pyroSAR environment. Let's make this easier by providing it in a separate package without all the pyroSAR overkill.

The following steps are yet to be done:

• settle on the final package name
• decide on location of cross-dependency ancillary.py
• set up coveralls
• set up readthedocs
• host package on PyPI
• remove spatial functionality from pyroSAR and import new package instead
• move tests of spatial functionality to new package
• clean pyroSAR git history
• integrate all open issues for the spatial tools in pyroSAR into new package

This issue deals with introducing an option to write GeoTiff files optimized for cloud handling as described e.g. here and here.
modules affected:

• snap.auxil.gpt
• snap.util.geocode
• gamma.util.geocode
• spatial.raster: Raster.write. reproject, stack, rasterize
• spatial.auxil: gdalwarp, gdal_translate, gdal_rasterize

Whilst at it, it would be worthwhile putting some thought into how all these above can be centralized or at least simplified. It would be great to have a central file writing module.

Is there a reason why the SNAP workflow file (e.g., S1B__IW___A_20170106T161500_bnr_orb_cal_tf_tc_dB_proc.xml) is not removed after the geocode function has finished processing?

In my processing for each scene I have the following files:
S1B__IW___D_20170129T042958_bnr_orb_cal_tf_tc_dB_proc.xml
S1B__IW___D_20170129T042958_VH_bnr_orb_cal_tf_tc_dB
S1B__IW___D_20170129T042958_VV_bnr_orb_cal_tf_tc_dB

If it is important for you to keep this file, maybe an optional parameter could be introduced to remove clean up and remove this file. Other processing files are removed automatically.

I would recommend to print some logging messages when doing some processing (e.g., executing the geocode function). Maybe there are some outputs in the stdout from the SNAP gpt command line tool that could be redirected to the Python output console?

Currently within function gamma.auxil.process only stdout is written to log.

The ssl module does not support the current implementation of creating an unverified ssl context in Python version < 2.7.9. Since Travis currently uses version 2.7.6, the download test is currently omitted. How to download osv files with this version? See method S1.OSV.init and function test_scene of test_drivers.py.

I wrote some unit tests to test the classes ExamineExe and ExamineSnap. After that I would like to change the .travis.yml file and download snap from "http://step.esa.int/downloads/6.0/installers/esa-snap_sentinel_unix_6_0.sh" to install it within travis. Thus, I add the following code block to the before_install section:

  - sudo apt-get install wget
  - wget -O esa-snap_sentinel_unix_6_0.sh http://step.esa.int/downloads/6.0/installers/esa-snap_sentinel_unix_6_0.sh
  - sudo chmod +x esa-snap_sentinel_unix_6_0.sh
  - sudo ./esa-snap_sentinel_unix_6_0.sh

The problem is now travis uses the config file from the original repository and not from the forked one. Do you guys have any suggestions how we could solve this?

So far this is done in function drivers.parse_date which gathers known date string formats and parses them. It would be nice to have a general parsing approach instead of trying to parse all known formats until one works. Is there something available in the python modules time or datetime?

It would be convenient to have the cluster nodes report their status to the master process during processing to better monitor progress. How can this be done?

Currently the method Raster.matrix only returns single bands, by default the first one. It would be more intuitive if the default would be all bands as one array.
This is directly related to the internal band management in Raster.__data, which also needs to be adjusted in this context.

Hi,

I just tried to test it on Windows 10 (load the scene into pyroSAR for analysis of the metadata)

from pyroSAR import identify
name = 'S1A_IW_GRDH_1SDV_20150222T170750_20150222T170815_004739_005DD8_3768.zip'
scene = identify(name)
scene.summary()

The problem occured on the third line. Please see the error as following:

RuntimeError Traceback (most recent call last)
in ()
1 name = 'S1B_IW_GRDH_1SDV_20170119T041316_20170119T041341_003912_006BD4_26C0.zip'
----> 2 scene = identify(name)

c:\users\mustuner\anaconda3\lib\site-packages\pyrosar-0.3-py3.6.egg\pyroSAR\drivers.py in identify(scene)
75 except (IOError, KeyError):
76 pass
---> 77 raise RuntimeError('data format not supported')
78
79

RuntimeError: data format not supported

Thanks

I found a way to improve our Storage class. With this we could allow only particular keys. this could help not to mess around with this class (look at __setattr__)

class Storage(dict):
     def __getattr__(self, name):
        try:
            return self[name]
        except KeyError:
            raise AttributeError(name)

    __delattr__ = dict.__delitem__

    KEYLIST = ['KEY1', 'KEY2', 'KEY3']

    def __repr__(self):
        if self.keys():
            m = max(map(len, list(self.keys()))) + 1
            return '\n'.join([k.rjust(m) + ': ' + repr(v)
                              for k, v in sorted(self.items())])
        else:
            return self.__class__.__name__ + "()"

    def __dir__(self):
        return list(self.keys())

    def __setattr__(self, key, value):
        if key not in Storage.KEYLIST:
            raise KeyError("{} is not a legal key of this Storage".format(repr(key)))
        dict.__setitem__(self, key, value)

What do you think?

archivist is an own attempt to centralize all archiving functionality and contains several top-level functions for easier access to zip/tar archives. A lot of which is done in ID._unpack is better put there.

For a further processing of resulting files after conducting the geocode function, it would be useful to return a list of files that have been created during the geocode (or respective gpt) function.

Such a return could be integrated at the end of the gpt function in pyroSAR/snap/auxil.py and further passed as a return of the geocode function.

Problem description

Dear All,

so I was trying to use the stack function on the pre-processed Sentinel-1 data (circa 20 images) to mosaic, stack and crop them. Unfortunately, that resulted in an error. It is worth mentioning that a temporal folder with all the .vrt files of the scenes was created during the call. The Gdal version I am using is 2.2.3.
See below the code I was using and the error message.

Thank you very much in advance for your help.

Code

import os

from pyroSAR.ancillary import finder
from pyroSAR.ancillary import groupbyTime
from pyroSAR.ancillary import seconds
from pyroSAR.spatial import stack

dir_in = '/media/DATA4/balli001/Data/01_Sumatra/01_Sentinel_1/02_pre_processed/01_test_small/'

dstfile = '/media/DATA4/balli001/Data/01_Sumatra/01_Sentinel_1/03_stack/01_stack_vv/test_vv_stack'

shp = '/media/DATA4/balli001/Data/01_Sumatra/05_additional_data/01_small_test_site/Small_test_site.shp'

sep = False

srcfiles = finder(dir_in, ['S1*_VV_*_dB.tif'])
    
# check whether dstfile is already a file
if os.path.isfile(dstfile):
	raise IOError('dstfile already exists')

groups = groupbyTime(srcfiles, seconds, 30)


# final function call
stack(srcfiles=groups, dstfile=dstfile, resampling='bilinear', targetres=[20, 20], srcnodata=0, dstnodata=-99, shapefile=shp, sortfun=seconds, separate=sep, overwrite=True)

Error message

Traceback (most recent call last):
  File "02_stacking.py", line 36, in <module>
    stack(srcfiles=groups, dstfile=dstfile, resampling='bilinear', targetres=[20, 20], srcnodata=0, dstnodata=-99, shapefile=shp, sortfun=seconds, separate=sep, overwrite=True, cores=4)
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/spatial/raster.py", line 739, in stack
    gdalwarp(vrt, dstfile, options_warp)
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/spatial/auxil.py", line 126, in gdalwarp
    out = gdal.Warp(dst, src, options=gdal.WarpOptions(**options))
  File "/usr/lib/python2.7/dist-packages/osgeo/gdal.py", line 580, in Warp
    return wrapper_GDALWarpDestName(destNameOrDestDS, srcDSTab, opts, callback, callback_data)
  File "/usr/lib/python2.7/dist-packages/osgeo/gdal.py", line 3112, in wrapper_GDALWarpDestName
    return _gdal.wrapper_GDALWarpDestName(*args)
RuntimeError: Attempt to create 0x0 dataset is illegal,sizes must be larger than zero.

Greetings,

Johannes

This class converts between latlon to UTM coordinates, which is something very general which should go into the spatial module and not the gamma module.

The current Travis OS is Ubuntu Trusty, which comes with Python 3.4. Testing with Python 3.5 or 3.6 is not possible because system site packages need to be enabled and this seems to only work with the OS-installed Python version. Is there a way to still test the other versions?

Up to now pyroSAR raised an ValueError if snap was not installed. I changed this characteristic to an UserWarning. In my opinion it is more advantageous because if we check the snap installation in snap\__init__.py and snap is not installed we have not a chance to test the code due to the ValueError that is thrown at the import procedure. Now, if snap is not installed the following warning will appear:

warnings.warn("One of the executables {0} should be installed. You can download it from http://step.esa.int/main/toolboxes/snap/ or you can specify a path with snap_config.set_path(path_to_snap)".format(name), UserWarning)

Now I am able to test the thrown UserWarning and we increase our coverage up to 27%. Moreover, I could not test the code with TRAVIS CI because the command find_executable does not work with TRAVIS.

What do you think guys?

Dear reader,

I installed PyroSAR, the needed extensions (SQLite + SpatiaLite) and SNAP as described here on Github on a Ubuntu server. Afterwards I was following the instruction of the example and everything works fine. BUT when I try to pre-process one Sentinel-1 image via pyroSAR, the geocode function results in an error.

The script I am using follows the instructions also given by the Readme on the front-page:

Script

from pyroSAR import identify
from pyroSAR.snap import geocode

name = 'path/S1A_IW_SLC__1SDV_20180605T111503_20180605T111531_022220_026773_82A0.zip'
scene = identify(name)
resolution = 20
outdir = 'path'

geocode(infile=scene, outdir=outdir, tr=resolution, scaling='db')

Result

In the outdir a .xml-file was created, but nothing else.

Error

Traceback (most recent call last):
  File "pre_processing_test.py", line 17, in <module>
    geocode(infile=scene, outdir=outdir, tr=resolution, scaling='db')
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/util.py", line 266, in geocode
    gpt(outname + '_proc.xml')
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/auxil.py", line 214, in gpt
    proc = sp.Popen(cmd, stdout=sp.PIPE, stderr=sp.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Question

The Question I have is, whether I am missing something I should have done during the installation or what the origin of that error is in general. Thank you very much in advance and if you need any additional information, please let me know.

Greetings,
Johannes

The problem is we must deal with paths, like the path of snap, the path of etc directory or the paths of different DEMs and orbit files. The main idea was to focus all these information in one configuration file. But I think we have to discuss the best and suitable way to do it. The questions are:

• How should the file look like?
• Which header information we should use?
• Which file extension is the best (.txt, .CSV etc.)?
• Should we code a own class or a module?

At the moment the coordinates are those of the outer ground control points. Thus, this is likely incorrect by at least half a pixel (if the GCP is in the center of the outer pixel). This need to be checked.

With the decision to outsource the spatial package, let's restructure both into an organization.
Name suggestions welcome..

This seems to be good unit tests to have on Travis, because we don't need large datasets to test this.

Dear Reader,

when I want to pre-process several Sentinel-1 scenes I run into a problem executing the geocode function for single polarized Sentinel-1 Images. See below the Summary of one of the Images (Just for getting an idea of what kind of image I am talking about):

Summary of a mentioned Sentinel-1 image (example)

S1A_IW_GRDH_1SSV_20150107T230831_20150107T230856_004071_004EB4_B3EC.zip
acquisition_mode: IW
lines: 16882
orbit: A
polarizations: ['VV']
product: GRD
projection: GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.01745329251994328,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]]
samples: 26604
sensor: S1A
spacing: (10.0, 9.9913)
start: 20150107T230831
stop: 20150107T230856
['VV']

Problem description

However, If i want to use the geocode function to pre-process them, I run into following error message.
I have that for several scenes (~70 in total) and all of them are single polarized.

S1A_IW_GRDH_1SSV_20150107T230831_20150107T230856_004071_004EB4_B3EC.zip
Executing processing graph
 done.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder '.'.

Error: [NodeId: Calibration] Operator 'CalibrationOp': Value for 'Source Band' is invalid: 'Intensity'

Apparently there is something wrong with a non existing source band "Intensity", so I am not to sure whether the problem arises from the script or function itself.

Script

See below the straight forward script I am using. Where selection_proc is the selected Archive of Sentinel-1 scenes to be pre-processed.

for scene in selection_proc:
	id=identify(scene)
	if id.meta['IPF_version'] < 2.9:
		temp_bn=True
	else:
		temp_bn=False
    	geocode(infile=scene, outdir=outdir, tr=resolution, scaling='db',removeS1BoderNoise=temp_bn)

Thank you very much in advance for your help. If you need any additional information do not hesitate to contact me.

Best regards,
Johannes

Currently it is shapefile. Let's change to a more general open standard.

add new standardized attributes to LOCAL and integrate them in ID methods init and summary to ensure each subclass returns it.

• bounding box/corner coordinates
• number of looks
• ...

currently the product naming is taken from the respective satellites, which in the case of e.g. CEOS_PSR is not very meaningful (1.0, 1.1, 1.5). How to generalize these categories across sensors?

From pygdal package description:

Virtualenv and setuptools friendly version of standard GDAL python bindings

Switching from package GDAL to pygdal might make installing the dependencies easier, with less manual intervention.

It would be good to have the possibility to get the installed version of pyroSAR with:
pyroSAR.version
This should be defined at one place in the source code and used by setup.py.
Relevant links I found for this are:

https://pypi.python.org/pypi/setuptools_scm
https://packaging.python.org/guides/single-sourcing-package-version/

The functionality is available in function process and just needs to be activated and passed as argument.

I use the geocode function from pyroSAR.snap. When processing the data the resulting files are in GeoTIFF format, but unfortunately they have no file extension:
S1A__IW___D_20170130T042224_VH_bnr_orb_cal_tf_tc_dB S1B__IW___D_20170129T042958_VH_bnr_orb_cal_tf_tc_dB
S1A__IW___D_20170130T042224_VV_bnr_orb_cal_tf_tc_dB S1B__IW___D_20170129T042958_VV_bnr_orb_cal_tf_tc_dB

gdalinfo S1B__IW___D_20171130T061758_VH_bnr_orb_cal_tf_tc_dB
Driver: GTiff/GeoTIFF

I would propose to automatically add the file ending for GeoTIFF files, so it is easier for users to open such outupt files in GIS Desktop software. Otherwise users need to rename the files on their own.

I received the following error message while geocoding:

geocoding and normalization..
Traceback (most recent call last):
File "example_scoop_gamma.py", line 139, in
results = list(futures.map(worker, sites))
File "example_scoop_gamma.py", line 119, in worker
func_geoback=2, func_interp=2, sarSimCC=False, osvdir=osvdir, cleanup=False, allow_RES_OSV=False)
File "/home/c3urma/repositories/pyroSAR/pyroSAR/gamma/util.py", line 761, in geocode
shellscript=shellscript)
TypeError: geocode_back() got an unexpected keyword argument 'lookup_table'

At the moment, we are only testing on Linux and MacOS via Travis.
We should setup appveyor to also automatically test on Windows.

The geocode function from pyroSAR.snap provides the "scaling" attribute, which can be either "dB" or "linear". Personally I did not recognize the capital B in "dB", so I tried always using scaling="db". Maybe you could introduce an error message when trying to provide an input that is not "linear" or "dB". Maybe you could also support the "db" input with non-capital letters.

We should make a documentation guide for functions and classes. Because in many cases I do not have a clue which input type is required (str, tuple, list, class instance etc.). I tried to update the geocode function in snap but I am not sure if the required input types are correct:

def geocode(infile, outdir, t_srs=None, tr=20, polarizations='all', shapefile=None, scaling='dB',
            geocoding_type='Range-Doppler', removeS1BoderNoise=True, offset=None, externalDEMFile=None, externalDEMNoDataValue=None, externalDEMApplyEGM=True, test=False):
    """
    wrapper function for geocoding SAR images using ESA SNAP

    Parameters
    ----------
    infile: str or pyroSAR.ID class instance
        A pyroSAR.ID object or a file/folder name of a SAR scene
    outdir: str
        the directory to write the final files to
    t_srs: str or None, optional
        a target geographic reference system (a string in WKT or PROJ4 format or a EPSG identifier number)
    tr: int or float
        the target resolution in meters
    polarizations: list or {'VV', 'HH', 'VH', 'HV', 'all'}, optional
        The polarizations to be processed; can be a string for a single polarization e.g. 'VV' or a list of several polarizations e.g. ['VV', 'VH']
    shapefile: ???
        A vector file for subsetting the SAR scene to a test site
    scaling: {'dB', 'db', 'linear'}, optional
        should the output be in linear of decibel scaling? Input can be single strings e.g. 'dB' or a list of both: ['linear', 'dB']
    geocoding_type: str, optional
        The type of geocoding applied; can be either 'Range-Doppler' or 'SAR simulation cross correlation'
    removeS1BoderNoise: bool
        Enables removal of S1 GRD border noise
    offset: tuple
    A tuple defining offsets for left, right, top and bottom in pixels, e.g. (100, 100, 0, 0); this variable is overridden if a shapefile is defined
    externalDEMFile: str or None, optional
        The absolute path to an external DEM file
    externalDEMNoDataValue: int or float
        The no data value of the external DEM. If not specified the function will try to read it from the specified external DEM.
    externalDEMApplyEGM: bool, optional
        Apply Earth Gravitational Model to external DEM?
    test: bool, optional
        If set to True the workflow xml file is only written and not executed

    Note
    ----
    If only one polarization is selected the results are directly written to GeoTiff.
    Otherwise the results are first written to a folder containing ENVI files and then transformed to GeoTiff files (one for each polarization)
    If GeoTiff would directly be selected as output format for multiple polarizations then a multilayer GeoTiff
    is written by SNAP which is considered an unfavorable format

    See Also
    --------
    pyroSAR.ID
    """

GAMMA has it's own way of defining coordinate reference systems, which is extremely inconvenient to use. How to automatically convert between these and international standards like EPSG, WKT and PROJ4? A start was done in gamma.srtm.dempar, but is limited to EQA and UTM.

The last commit has introduced some errors (Linux environment) when calling pyroSAR.snap.geocode:

Traceback (most recent call last):
  File "/usr/local/bin/geocode.py", line 13, in <module>
    from pyroSAR.snap import geocode
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/__init__.py", line 4, in <module>
    snap_config = ExamineSnap()
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/auxil.py", line 250, in __init__
    self.__get_etc()
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/auxil.py", line 262, in __get_etc
    except WindowsError:
NameError: global name 'WindowsError' is not defined

If I adjust this manually to OSError than the following error occurs:

Traceback (most recent call last):
  File "/usr/local/bin/geocode.py", line 13, in <module>
    from pyroSAR.snap import geocode
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/__init__.py", line 4, in <module>
    snap_config = ExamineSnap()
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/auxil.py", line 250, in __init__
    self.__get_etc()
  File "/usr/local/lib/python2.7/dist-packages/pyroSAR/snap/auxil.py", line 260, in __get_etc
    self.config_path = os.path.join(self.etc, [s for s in self.auxdata if "snap.auxdata.properties" in s][0])
IndexError: list index out of range

I also do not understand why the "etc" directory is related to the path of the executable (self.path), see line 258 of pyroSAR/snap/auxil.py:
self.etc = os.path.join(os.path.dirname(os.path.dirname(self.path)), 'etc')

On my system I have an "etc" directory within HOME/.snap with configuration files.

There are a lot of docstrings hidden in the code.
To make them easy accessible it would be good to have them displayed on
readthedocs.io toghether with short introductions in the functionality of pyroSAR.

Therefore, we have to setup sphinx to automatically get the docstrings in a nice format.

Coveralls badge is stuck at 25%, despite the build being at 30%.

If I try to import pyroSAR.snap it fails with the following Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/qe89hep/Programmieren/pyroSAR/pyroSAR/snap/__init__.py", line 4, in <module>
    snap_config = ExamineSnap()
  File "/home/qe89hep/Programmieren/pyroSAR/pyroSAR/snap/auxil.py", line 253, in __init__
    self.__get_etc()
  File "/home/qe89hep/Programmieren/pyroSAR/pyroSAR/snap/auxil.py", line 263, in __get_etc
    self.config_path = os.path.join(self.etc, [s for s in self.auxdata if "snap.auxdata.properties" in s][0])
IndexError: list index out of range

My Snap etc folder is at
/home/username/snap/etc
this is the location that is used automatically by the SNAP installer.

We should test the metadata retrieval of all subclasses of ID which are defined in drivers.py
Herefore, we would need to:

• Find freely available testdata for every supported satellite
• Get the metadata of these testdata sets
• Add an entry to the test_drivers.py test cases list.

Today I tried to figure out how rasterio works. The first point is you should install rasterio in a virtual environment. After I follow these several steps gdal and in general Anaconda won’t work properly, e.g.:

>>> import gdal
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\gdal.py", line 2, in <module>
    from osgeo.gdal import deprecation_warn
  File "C:\Python27\lib\site-packages\osgeo\__init__.py", line 21, in <mo
    _gdal = swig_import_helper()
  File "C:\Python27\lib\site-packages\osgeo\__init__.py", line 17, in swit_helper
  _mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: DLL load failed: The specified procedure could not be found.

So, I must reinstall my python distribution. I don´t know if it is a specific problem with Windows. Even if I install it with conda-forge it did not work due to problems with dependencies. Can I try the install procedure on our Linux server?

This file gives a quick access to relevant metadata so that the mixed binary-ascii image file does not need to be read. If it does not exist it could be written to the scene archive so that in the future access is faster.

the URL for downloading 1 arcsec SRTM tiles un function srtm.hgt_collect now requires a login and thus nedds to be replaced.

Currently scoop is called via an extra bash script which calls a scoop-ready script via
python -m scoop --hostfile scheduler --prolog prolog script.py.

scheduler contains a lookup of nodes and the number of processes to start on each, e.g.
node1 1
node2 1

prolog contains additional OS variables which need to be available to the processes.

This is all quite inconvenient. It would be nice to have a general function which automatically writes these needed files to e.g. $HOME/.pyroSAR/scoop and calls the command via subprocess. A user would thus only need to define a list of nodes and the number of processes to run on each. Furthermore the process distribution is quite different between the snap and gamma modules, which is hard for a user to know and thus should be determined in the background.