actinia-org / actinia-core Goto Github PK

I'm following the install instructions here : https://actinia.mundialis.de/tutorial/installation.html

My OS is Linux Mint 20.1, v4.8.6 . The computer has not been used for much before so it is empty of most additional libraries not explicitly mentioned in the installation instructions.

Step 6 says:

Make sure that a redis service is running

I don't know what that is so I'll look it up. I would recommend turning this into a link to instructions for starting a redis service. I am now following these instructions: https://redis.io/topics/quickstart . Fairly easy to follow so that worked, and I turned it on.
From here, the instructions state:

cd /tmp
source actinia_venv/bin/activate

# Create the superuser
actinia-user create -u superadmin -w abcdefgh -r superadmin -g admin -c 100000000000 -n 1000 -t 6000

# Start the actinia service with
actinia_server --host 0.0.0.0 --port 5000

When I try to run actinia-user create, I get an error

Traceback (most recent call last):
  File "/home/halota/actinia/workspace/tmp/actinia_venv/bin/actinia-user", line 4, in <module>
    __import__('pkg_resources').run_script('actinia-core==0.99.26.post0.dev1+ge1510fc', 'actinia-user')
  File "/home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1462, in run_script
    exec(code, namespace, namespace)
  File "/home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/actinia_core-0.99.26.post0.dev1+ge1510fc-py3.8.egg/EGG-INFO/scripts/actinia-user", line 30, in <module>
    from actinia_core.resources.common.user import ActiniaUser
  File "/home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/actinia_core-0.99.26.post0.dev1+ge1510fc-py3.8.egg/actinia_core/resources/common/user.py", line 35, in <module>
    from actinia_core.resources.common.redis_user import redis_user_interface
  File "/home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/actinia_core-0.99.26.post0.dev1+ge1510fc-py3.8.egg/actinia_core/resources/common/redis_user.py", line 28, in <module>
    from .redis_base import RedisBaseInterface
  File "/home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/actinia_core-0.99.26.post0.dev1+ge1510fc-py3.8.egg/actinia_core/resources/common/redis_base.py", line 29, in <module>
    from .logging_interface import log
  File "/home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/actinia_core-0.99.26.post0.dev1+ge1510fc-py3.8.egg/actinia_core/resources/common/logging_interface.py", line 40, in <module>
    global_config.read()
  File "/home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/actinia_core-0.99.26.post0.dev1+ge1510fc-py3.8.egg/actinia_core/resources/common/config.py", line 334, in read
    with open(path, 'r') as configfile:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/default/actinia'

Not sure if this is related to some install location being different than expected, or if I missed some step. I do not have any actinia folder in /etc/default.

Like this one :) https://grasswiki.osgeo.org/wiki/GRASS_development_visualization

The barrier of entry installing and running actinia_core is currently very cumbersome because of the complex development environment. I want to reduce the barrier of entry installing the project and its many dependencies by creating Dockerfiles and shell scripts or makefile to build and run actinia_core. Two Dockerfiles could be used to separate the core API and the GRASS sessions GRASS Dockerfile, and any other containers could be easily added for storage or caching.

Ideally, a new developer should be able to download the project and start it with one command if they have Docker installed.

For example:

make server

or

./run.sh

I'm willing to work on implementing the new feature, so let me know if first, you all want this and second if you have any other ideas or suggestions of how to best accomplish this task.

Thanks for all of the hard work!

There is an empty /grassdb/ in the deployed docker container while data really reside in /actinia_core/grassdb/.

Perhaps a wrong path here?

https://github.com/mundialis/actinia_core/blob/fdc0d91d83a528a0584cf206f701a057ba01bb6d/docker/actinia-core-alpine/Dockerfile#L142

Maybe the purpose is something else - then just disregard this issue.

Check which documentation is available, update outdated information and document problems or incomprehensive parts where found.

Review needed of
https://github.com/mundialis/actinia_core/blob/master/CONTRIBUTING.md

I have drafted an initial version which needs to be improved.

Following instructions from here: https://actinia.mundialis.de/tutorial/installation.html
So far I believe I have gotten through steps 1-6. I have redis server running, and I started actinia-server using this command:

(actinia_venv) halota@basement-machine:~/actinia/workspace/tmp/actinia_core$ actinia-server --host 0.0.0.0 --port 5000
 * Serving Flask app "actinia_core.resources.common.app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off

Now I am following step 7 instructions, but it's not finding it.

halota@basement-machine:~$ export PORT=5000
halota@basement-machine:~$ export IP=http://localhost
halota@basement-machine:~$ export AUTH='-u superadmin:abcdefgh'
halota@basement-machine:~$ curl -u superadmin:abcdefgh -X GET http://localhost:5000/locations
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>

Follow the installation steps from https://actinia.mundialis.de/tutorial/installation.html and document deprecated information in this ticket or directly in the documentation.
Attention: the documentation changed from rst files to md: https://github.com/mundialis/actinia_core/blob/main/docs/docs/installation.md

When installed in a software stack, single sign-on would be an advantage instead of a separate authentication for actinia. Therefore it would be cool to enable authentication via anoter component, e.g. keycloak.

In order to get actinia_core under the umbrella of OSGeo as an OSGeo community project, we need to replace the script headers. The license of actinia (GPL 3) must be mentioned. I suggest s.th. like the GRASS team does:

/*!
\ SOME COMMENT
(C) 2018 by the actinia Development Team

This program is free software under the GNU General Public License
(>=v3). Read the file COPYING that comes with actinia for details.

\ MAYBE AUTHOR
*/

See https://github.com/github/renaming

Probably an update is also needed in this line?

https://github.com/mundialis/actinia_core/blob/2151de5e1b41a12897f37465678612a4c595f71e/src/actinia_core/testsuite.py#L179

Originally posted by @neteler in #37 (comment)

Via actinia API call, it can be checked if a mapset is locked or not by calling /locations/{location_name}/mapsets/{mapset_name}/lock endpoint. As the mapset might not be persisted when calculation is in progress, it will not be listed in all mapsets which makes it hard to find out all currently locked mapsets.
In redis db, all currently locked mapsets can be requested with KEYS RESOURCE-LOCK* .

It would be a nice enhancement to let the admin/superadmin view all currently locked mapsets via api call.

Placeholder issue to check regularly the plugin issues:

Issues
PRs (excluding renovate)
Discussions - Check if one should be transformed to an issue

Overview:

Plugins:

• https://github.com/mundialis/actinia-module-plugin
• https://github.com/mundialis/actinia-metadata-plugin
• https://github.com/mundialis/actinia_satellite_plugin
• https://github.com/mundialis/actinia_statistic_plugin
• https://github.com/mundialis/actinia-stac-plugin
• https://github.com/mundialis/actinia-tiling-plugin
• https://github.com/mundialis/actinia-parallel-plugin
• https://github.com/mundialis/actinia-example-plugin

Libs:

• https://github.com/mundialis/actinia-api

Also check for other related repositories:

• https://github.com/mundialis/actinia-python-client
• https://github.com/mundialis/actinia-jupyter
• https://github.com/mundialis/ace
• https://github.com/mundialis/exporter
• https://github.com/mundialis/importer
• https://github.com/mundialis/actinia-docker
• https://github.com/mundialis/helm-charts/tree/main/charts/actinia
• https://github.com/actinia-org/actinia-introduction
• https://github.com/actinia-org/slides
• https://github.com/actinia-org/actinia-org.github.io
• https://github.com/actinia-org/processing-examples

No check needed:

• https://github.com/actinia-org/actinia-core (auto added to project)
• https://github.com/actinia-org/actinia-gdi (archived)

There is a typo:

https://github.com/mundialis/actinia_core/blob/23ba5635b8d3962b3962c75f1e8b4393d20cc65a/docker/actinia-core-prod/actinia.cfg#L18

--> process_time_limt should be process_time_limit (may require more changes in the code)

Leftover openjdk8-...apk files in docker image
In this Dockerfile section openjdk8 packages are fetched:

https://github.com/mundialis/actinia_core/blob/fdc0d91d83a528a0584cf206f701a057ba01bb6d/docker/actinia-core-alpine/Dockerfile_runtime_pkgs#L99

but later not removed so that they remain in the docker image:

actinia:~$ docker exec -it 0cef  sh # dev
/src/actinia_core # cd /
/ # ls -la
total 62056
drwxr-xr-x   1 root root     4096 Jan  5 07:48 .
drwxr-xr-x   1 root root     4096 Jan  5 07:48 ..
-rwxr-xr-x   1 root root        0 Jan  5 07:48 .dockerenv
drwxr-xr-x   6 root root     4096 Dec 22 15:48 actinia_core
drwxr-xr-x   1 root root     4096 Oct 19 12:15 bin
drwxr-xr-x   1 root root     4096 Dec 22 15:46 build
drwxr-xr-x   5 root root      340 Jan  5 07:49 dev
drwxr-xr-x   1 root root     4096 Jan  5 07:49 etc
drwxr-xr-x   2 root root     4096 Dec 22 16:32 grassdb
drwxr-xr-x   2 root root     4096 May 29  2020 home
drwxr-xr-x   1 root root     4096 Oct 19 12:36 lib
drwxr-xr-x   5 root root     4096 May 29  2020 media
drwxr-xr-x   1 root root     4096 Jan  5 07:49 mnt
-rw-r--r--   1 root root 15755870 Oct 19 12:15 openjdk8-8.232.09-r0.apk
-rw-r--r--   1 root root   358826 Oct 19 12:15 openjdk8-jre-8.232.09-r0.apk
-rw-r--r--   1 root root  4591079 Oct 19 12:16 openjdk8-jre-base-8.232.09-r0.apk
-rw-r--r--   1 root root 42744528 Oct 19 12:16 openjdk8-jre-lib-8.232.09-r0.apk

Low prio but a bit of space could be saved removing them in the right moment (maybe already in line 103?) :-)

At time only http requests are supported due to a limitation by design in "python-requests".
It would be important to be able to import geodata files from local storage in addition to http requests.

Problem description

Example: Sentinel-1 data are moved fast to a long term archive, leading to this message:

i.sentinel.download ....
Product c0a3c23c-c68f-47d3-9812-8e7b98e19d08 is not online. Triggering retrieval from long term archive.

Such a case cannot easily be dealt with in a process chain, hence the separate S-1 scene retrieval and local storage is the only viable solution.

Needed code changes:

The code here needs to be changed:

https://github.com/mundialis/actinia_core/blob/59cdc55ffab5dea3f8f46f233c3569ffbddb129c/src/actinia_core/resources/common/geodata_download_importer.py#L114

Potential code solutions

Here some pointers/code snippets:

• https://stackoverflow.com/a/22989322/452464
• https://github.com/dashea/requests-file

I am trying to set up a local actinia grass server on Linux Mint. It's a new computer with nothing except the basics installed so far.
Specs:

• Operating System: Linux Mint 20.1 Cinnamon v.4.8.6
• Linux Kernel : 5.4.0-66-generic
• Processor: Intel Core i5-2500K CPU @ 3.30GHz x 4
• Memory: 15.6 GiB
• Hard Drives: 1282.4 GB
• Graphics Card: AMD/ATI Tahiti PRO [Radeon HD 7950/8950 OEM/R9 280] (prog-if 00 [VGA controller])

I am following these instructions: https://actinia.mundialis.de/tutorial/installation.html

Step one went mostly well, and step 2 to install grass as well.

Some minor issues:

• cd /tmp
  On my machine, /tmp automatically clears when the computer restarts, so it was not an ideal location. I would not recommend placing the downloads into that folder.

make -j16
sudo make install

• I had to install a few extra dev libraries for this step to work. The names were in general fairly obvious, always some sort of libdev version of whatever XXX.h file was said to be missing. Here is a list of the ones that I had to install:
  • zlib1g-dev
  • libghc-bzlib-dev
  • libstd-dev
  • libgdal-dev
  • libcairo2-dev

In order to use the grass GUI, located in grass_master/bin.x86_64-pc-linux-gnu/grass78, I also had to install:

• python3-wxgtk4.0

I am now stuck on trying to make t.rast.sample

cd temporal
git clone https://bitbucket.org/huhabla/temporal-raster-sampling.git t.rast.sample
cd t.rast.sample
make
sudo make install

It tells me I am missing the library openeo_udf. My best guess for where to find and install this library is here: https://github.com/Open-EO/openeo-udf . Is this correct one? This has a lot of dependencies as well, so I'm running into issues with their makefile that I will post there as well.

For deployment with kubernetes it would be helpful to be able to configure certain values via environment variable instead of one config file containing all values. This is especially usefull for the redis passwort as it can be set as secret from redis chart.

As a visitor of an actinia instance, it would be helpful to see all connected data, without a filter if I am allowed to use it or not. The current endpoints can list all locations which I am allowed to use or all mapsets of a certain location which I am allowed to use. It would be helpfull to see at one glance which data is available in the whole system.

Edit: it would additionally be helpful to see if the data is in the global grassdb or in the userdb.

Exporting a text file from actinia generally works but the packaging is not as advertised.

Export SLD file:

{
  "module": "r.colors.out_sld",
  "id": "r.colors.out_sld_1804289383",
  "inputs":[
     {"param": "map", "value": "peaks_CVA_map"},
     {"param": "style_name", "value": "GRASS color table"}
   ],
  "outputs":[
     {"export": {"format":"TXT", "type":"file"},
      "param": "output", "value": "$file::/tmp/peaks_CVA_map.sld"}
   ]
}

generates:

{'resources': ['https://actinia.mundialis.de/api/v1/resources/demouser/resource_id-deecf214-b355-4755-89bd-0f0b43eba5e3/peaks_CVA_map.sld.txt.zip',
...

which is fine but it contains

unzip peaks_CVA_map.sld.txt.zip
Archive:  peaks_CVA_map.sld.txt.zip
  inflating: actinia_core/workspace/temp_db/gisdbase_3367cd366c17486bb8b3af8673ae5bd4/.tmp/temp_file_1

instead of the expected peaks_CVA_map.sld.txt.

The reason might be:

https://github.com/mundialis/actinia_core/blob/29f3549a6ecfcc5cff13c5eee58f26aefb220f0f/src/actinia_core/resources/ephemeral_processing_with_export.py#L427

where temp_file_path might be undefined.

Any idea?

It would be more feasable to use markdown instead of rst. Therefore it would be good to transform the existing documentation and integrate it into CI, possibly use https://www.mkdocs.org, https://github.com/mkdocs/mkdocs/wiki/MkDocs-Themes.
While at it it would be also great to restructure existing documentation (API docs, user docs, docstring, tutorial, ...) to avoid rendundancy.

At time the directory tgis is not yet copied:

https://github.com/mundialis/actinia_core/blob/e1510fce08f8d40d5fc048dc7fd81e920ed5ffc3/src/actinia_core/resources/persistent_processing.py#L477

In addition, the support needs to be added also in some other places:

• update of mapset name in tgis sqlite.db in persistent part (see t.upgrade and t.downgrade)
• update mapset name in tgis sqlite.db when creating temporary DB
• add to exporter (separate from raster export)
  • e.g. netCDF
  • in case, augment also https://grass.osgeo.org/grass-devel/manuals/t.rast.export.html and https://grass.osgeo.org/grass-devel/manuals/t.vect.export.html

For inspiration, see
https://redocly.github.io/redoc/?url=https://actinia.mundialis.de/latest/swagger.json#tag/STRDS-Management

... to prepare for additional job manager later.

Running docker-compose build on a new root server (Fedora 29) led to this error:

...
Installed /usr/local/lib/python3.6/dist-packages/actinia_statistic_plugin-0.0.post0.dev9+g121813d-py3.6.egg
Processing dependencies for actinia-statistic-plugin==0.0.post0.dev9+g121813d
Finished processing dependencies for actinia-statistic-plugin==0.0.post0.dev9+g121813d
 ---> 2e4c507fbdda
Removing intermediate container 66e3a857c718
Step 21/51 : WORKDIR /src
 ---> 84ef6ff7b79b
Removing intermediate container faa8663af9ed
Step 22/51 : RUN git clone https://[email protected]/huhabla/d_rast_multi.git d.rast.multi
 ---> Running in c00ec409aa9d
Cloning into 'd.rast.multi'...
 ---> a4c805d2bace
Removing intermediate container c00ec409aa9d
Step 23/51 : RUN grass --tmp-location EPSG:25832 --exec g.extension -s extension=d.rast.multi url=/src/d.rast.multi
 ---> Running in a11278bcab78
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
ERROR: FileNotFoundError(2, 'No such file or directory')
Exiting...
ERROR: Service 'actinia-core' failed to build: The command '/bin/bash -c grass --tmp-location EPSG:25832 --exec g.extension -s extension=d.rast.multi url=/src/d.rast.multi' returned a non-zero code: 1
[20:12:19][STAGING][mundialis@calc2:~/repos/actinia_core/docker]$

This did not fail 3 hs ago on a different system, not sure what changed in actinia_core?

I'm testing out a new install of actinia.

I had issues with step 7 of the instructions for accessing via curl:
curl -u superadmin:abcdefgh -X GET http://localhost:5000/locations

That returned 404 Not found, as did trying it with http://0.0.0.0:5000/locations. Nothing was shown on the terminal that is running the actinia server.

I tried both of these again, this time as https. In this case I did see something actually show up on the server, but not correct. At least I see it is getting the message

halota@basement-machine:~$ curl -u superadmin:abcdefgh -X GET https://localhost:5000/locations
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
halota@basement-machine:~$ curl -u superadmin:abcdefgh -X GET https://0.0.0.0:5000/locations
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

The server shows after these requests (once each, only one shown here):

[2021-04-13 11:37:27,847] ERROR     : werkzeug._internal  -127.0.0.1 - - [13/Apr/2021 11:37:27]
 code 400, message Bad request version
 ("Ðßs3\x82W.\x9e\x00>\x13\x02\x13\x03\x13\x01À,À0\x00\x9f̨̩̪À+À/\x00\x9eÀ$À(\x00kÀ#À'\x00gÀ")
 [in /home/halota/actinia/workspace/tmp/actinia_venv/lib/python3.8/site-packages/werkzeug/_internal.py:113]

When using the & operator in r.mapcalc, the process chain is not valid.
Example:

{
  "version": "1",
  "list": [
    {
      "id": "r.mapcalc",
      "module": "r.mapcalc",
      "comment": "test the && operator",
      "inputs": [
        {
          "param": "expression",
          "value": "test_raster = if(elev_srtm_30m>10.0 && slope>1.0)"
        }
      ]
    }
  ]
}

Both process_chain_validation_sync as well as processing_async_export result in the error:

 "traceback": [
      "  File \"/usr/local/lib/python3.7/dist-packages/actinia_core-0.99.2.post0.dev33+g0914d88-py3.7.egg/actinia_core/resources/ephemeral_processing.py\", line 1350, in run\n    self._execute()\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/actinia_core-0.99.2.post0.dev33+g0914d88-py3.7.egg/actinia_core/resources/ephemeral_processing_with_export.py\", line 500, in _execute\n    EphemeralProcessing._execute(self)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/actinia_core-0.99.2.post0.dev33+g0914d88-py3.7.egg/actinia_core/resources/ephemeral_processing.py\", line 1221, in _execute\n    skip_permission_check=skip_permission_check)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/actinia_core-0.99.2.post0.dev33+g0914d88-py3.7.egg/actinia_core/resources/ephemeral_processing.py\", line 1254, in _create_temporary_grass_environment_and_process_list\n    skip_permission_check=skip_permission_check)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/actinia_core-0.99.2.post0.dev33+g0914d88-py3.7.egg/actinia_core/resources/ephemeral_processing.py\", line 495, in _validate_process_chain\n    process_list = self.proc_chain_converter.process_chain_to_process_list(self.request_data)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/actinia_core-0.99.2.post0.dev33+g0914d88-py3.7.egg/actinia_core/resources/common/process_chain.py\", line 569, in process_chain_to_process_list\n    return self._process_chain_to_process_list(process_chain)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/actinia_core-0.99.2.post0.dev33+g0914d88-py3.7.egg/actinia_core/resources/common/process_chain.py\", line 629, in _process_chain_to_process_list\n    module = self._create_module_process(process_descr)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/actinia_core-0.99.2.post0.dev33+g0914d88-py3.7.egg/actinia_core/resources/common/process_chain.py\", line 1001, in _create_module_process\n    \"description for %s\" % module_name)\n"
    ],
    "type": "<class 'actinia_core.resources.common.exceptions.AsyncProcessError'>"
  },
  "http_code": 400,
  "message": "AsyncProcessError:  Character '&' not supported in process description for r.mapcalc",
  "process_chain_list": [],
  "process_log": [],
  "progress": {
    "num_of_steps": 0,
    "step": 0

Following these instructions: https://actinia.mundialis.de/tutorial/installation.html

Step 4:

    mkdir -p $HOME/actinia/workspace/tmp

    cd /tmp

    git clone https://github.com/mundialis/actinia_core.git actinia_core
    virtualenv -p python3.5 actinia_venv
    source actinia_venv/bin/activate
    cd actinia_core
    pip install -r requirements.txt
    python setup.py install
    deactivate

If I see right I think it's accidentally directing me to root's /tmp instead of the newly created tmp directory? Instead of
cd /tmp I will follow directions using cd $HOME/actinia/workspace/tmp

This line:
virtualenv -p python3.5 actinia_venv Did not work for me, so far I only have python3.8 installed on my computer. I'm guessing it's related to that. I decided to let it decide which version to use and it defaulted to 3.8 and I will attempt to continue the install like that:virtualenv -p python3 actinia_venv

I needed to also install this library (in order to get uwsgi to install)

• python3-dev

I think we should move ace to a dedicated repository, as it is a IMO utility that helps working with actinia instances.

Or is it a central part of actinia, without which actinia isn't fulfilling it's central goal?

What do you think?

Good afternoon,

I'm having touble while building the docker image. I'm running the following:

git clone https://github.com/mundialis/actinia_core.git
cd actinia_core/docker/
docker-compose build
sudo docker-compose build

and I'm getting the following error:

Step 21/43 : COPY docker/actinia-core/snap /src/snap
ERROR: Service 'actinia-core' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder473168139/docker/actinia-core/snap: no such file or directory

I'm running Ubuntu 18.04.3 LTS with Docker version 18.09.7, build 2d0083d

Kind regards,

In the raster exporter currently there are hardcoded GeoTIFF settings:

https://github.com/mundialis/actinia_core/blob/8353b231c1a475955f68e4484e329c7d74b292e0/src/actinia_core/resources/ephemeral_processing_with_export.py#L251

Since GDAL 3.1+ there is a new COG driver which works nicely with r.out.gdal.

Wish: add COG driver support and use it if present. This would require a test:

r.out.gdal -l
  COG (rw): Cloud optimized GeoTIFF generator

• if COG driver present, use it for r.out.gdal export:
  • r.out.gdal -m input=mymap output=mymap.tif format=COG
• if COG driver not present, use the current GeoTIFF based r.out.gdal export.

I'm following the instructions here: https://actinia.mundialis.de/tutorial/installation.html

    cd /tmp

    virtualenv -p python2.7 grass_venv
    source grass_venv/bin/activate
    cd actinia_core
    pip install -r requirements-grass-gis.txt

As with step 4, I assume that we are meant to cd $HOME/actinia/workspace/tmp and not just /tmp.
The instructions say we need to have a python3 env for grass ,but the code sample shows 2.7. That should be 3?

There is no file "requirements-grass-gis.txt" in the actinia_core folder. I have one called "requirements.txt" and requirements_ubuntu19.txt".

Both seem to have notes about different versions, possible changes, etc. I will continue with "requirements_ubuntu19.txt" (I'm running Mint 20) and see how it goes.

Went ok except for this error:
ERROR: sphinx 3.5.4 has requirement docutils<0.17,>=0.12, but you'll have docutils 0.17 which is incompatible.
Since that's just for documentation, I'll ignore it for now.

With Flask > 1.0.0, the response of actinia API calls (via CURL) are not nicely formatted anymore. This is not a huge problem, as readability is given with tools like json or jq but it would be great to solve this without external dependencies.

Can you point out what the different considerations are as far as using actinia's API directly, versus going through the OpenEO API?

From looking at the projects it seems like learning to work with it through the OpenEO API will mean the methods I learn are more easily transferred to other datasets and data providers later on.

Is there any functionality from actinia that is lost or not implemented so far using OpenEO API? My biggest concern right now is that I want to be able to work with point data as well, which doesn't look to be directly supported with OpenEO, but is available with actinia (I'm assuming since it's using GRASS).

• https://github.com/mundialis/actinia_core/blob/master/src/actinia_core/__init__.py
• https://github.com/mundialis/actinia_core/blob/master/src/actinia_core/endpoints.py
  ...

The new SNAP docker image is ready at:

https://hub.docker.com/r/mundialis/esa-snap

The Dockerfile(s) may be updated accordingly.

and rename classes, methods + modules if more meaningfull name is possible.

With the current actinia logging implementation, log messages are passed to redis_fluentd_logger_base where they are initially logged. This way, it always logs as source e.g.
/usr/lib/python3.8/site-packages/actinia_core-0.99.5.post0.dev18+g4d82691.dirty-py3.8.egg/actinia_core/resources/common/redis_fluentd_logger_base.py:104
instead of where the message was generated. To change this, refactoring would be necessary.

When actinia is installed e.g. in Kubernetes and a Pod is restarted while a job is running, the job is displayed as "running" and the resource (mapset) is locked but the job was killed.
When trying to terminate the job via API call (to fix status + resource lock), actinia returns an error.

Wanted behaviour is that the job status will be set to error and the resource is freed. Another possibility would be to fix the termination endpoint when connection to job was lost and to allow an admin user to reset the resource lock.

At time the test test_landsat_dowunload_commads() fails (besides the fact, that it should be renamed to test_landsat_download_commands(), leading to blocked docker image creation:

======================================================================
ERROR: test_landsat_download_commands (__main__.GeoDataDownloadImportSupportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_geodata_import.py", line 199, in test_landsat_download_commands
    pl, import_file_info = lp.get_download_process_list()
  File "/usr/lib/python3.8/site-packages/actinia_core/resources/common/geodata_download_importer.py", line 149, in get_download_process_list
    self._setup()
  File "/usr/lib/python3.8/site-packages/actinia_core/resources/common/landsat_processing_library.py", line 178, in _setup
    for suffix in SCENE_SUFFIXES[self.landsat_sensor_id]:
KeyError: 'LC00'

Reason:
https://github.com/mundialis/actinia_core/blob/e66b6b5d2627e44624711b0a4c4a8e8f8881ce47/tests/test_geodata_import.py#L190

The (old) test dataset scene_id="LT41970251990147XXX03" no longer exists (https://storage.googleapis.com/gcp-public-data-landsat/LT04/PRE/006/016/LT40060161989006XXX02/LT40060161989006XXX02_MTL.txt).

An existing scene is for example scene_id="LC08_L1GT_001004_20130910_20170502_01_T2" (
https://storage.googleapis.com/gcp-public-data-landsat/LC08/01/001/004/LC08_L1GT_001004_20130910_20170502_01_T2/LC08_L1GT_001004_20130910_20170502_01_T2_B1.TIF).

The Landsat tests requires an update of the extract_sensor_id_from_scene_id() function to properly recognize the new scene directory structure. It is located here:
https://github.com/mundialis/actinia_core/blob/0d132114fe6a6007b25b06d8f485f784598e5470/src/actinia_core/resources/common/landsat_processing_library.py#L68

This is then input to scene_id_to_google_url() which constructs the download URL:
https://github.com/mundialis/actinia_core/blob/0d132114fe6a6007b25b06d8f485f784598e5470/src/actinia_core/resources/common/landsat_processing_library.py#L82

At time actinia docker image creation is failing due to this issue.

Either integrate in this plugin or create another one.

edit: issue was first created in mundialis/actinia-metadata-plugin

Currently the builds are configured in dockerhub. To accelerate the build, it should be build with github actions and then pushed to dockerhub. Then we can remove the grass79 branch as it is only needed to build different actinia docker images based on different GRASS GIS versions which can be managed by actions as well.

To be decided wether the tests are kept included or moved to separate action. (Separate would lead to faster builds but it must be ensured that every build code state is tested).

Hi all,
when I try to build and deploy actinia_core in a docker container I get the following error message:

Starting GRASS GIS...
ERROR: Unable to create new location because the location already exists.
Exiting...
Starting GRASS GIS...
ERROR: Unable to create new location because the location <latlong_wgs84> already exists.
Exiting...
Starting GRASS GIS...
ERROR: Unable to create new location because the location <nc_spm_08> already exists.
Exiting...
/src/start.sh: line 29: actinia-user: not found
/src/start.sh: line 30: actinia-user: not found
Failed to start actinia-user: 127
Starting GRASS GIS...
ERROR: Unable to create new location because the location already exists.
Exiting...
Starting GRASS GIS...
ERROR: Unable to create new location because the location <latlong_wgs84> already exists.
Exiting...
Starting GRASS GIS...
ERROR: Unable to create new location because the location <nc_spm_08> already exists.
Exiting...
/src/start.sh: line 29: actinia-user: not found
/src/start.sh: line 30: actinia-user: not found
Failed to start actinia-user: 127
Starting GRASS GIS...
ERROR: Unable to create new location because the location already exists.
Exiting...
Starting GRASS GIS...
ERROR: Unable to create new location because the location <latlong_wgs84> already exists.
Exiting...
Starting GRASS GIS...
ERROR: Unable to create new location because the location <nc_spm_08> already exists.
Exiting...
/src/start.sh: line 29: actinia-user: not found
/src/start.sh: line 30: actinia-user: not found
Failed to start actinia-user: 127
Starting GRASS GIS...
ERROR: Unable to create new location because the location already exists.
Exiting...
Starting GRASS GIS...
ERROR: Unable to create new location because the location <latlong_wgs84> already exists.
Exiting...
Starting GRASS GIS...
ERROR: Unable to create new location because the location <nc_spm_08> already exists.
Exiting...
/src/start.sh: line 29: actinia-user: not found
/src/start.sh: line 30: actinia-user: not found
Failed to start actinia-user: 127

Someone can give me some hint to try to solve it?
Thanks in advance for all the work done.
Ema.

and rename classes, methods + modules if more meaningfull name is possible.

Currently not all actinia-user commands are working. Example:

usage: actinia-user [-h] [-s SERVER] [-p PORT] [-a REDIS_PASSWORD] [-u USER_ID] [-g USER_GROUP] [-w PASSWORD] [-r {superadmin,admin,user,guest}] [-c CELL_LIMIT] [-n PROCESS_NUM_LIMIT]
                    [-t PROCESS_TIME_LIMIT] [-d DATASETS] [-m MODULES]
                    {create,delete,show,list,update,update_add,update_rm,update_rm_location,pwcheck}

/src/actinia_core # actinia-user list
...
['demouser',
 'openeo',
 ...]

/src/actinia_core # actinia-user -u demouser show
User <demouser> does not exist

While create, update and list work, show and delete return described error.

At time the importer runs r.import always with "nearest" (neighbor) resampling which is a problem when importing surfaces (temp, elev, ...) in case that internally a reprojection takes place.

Proposed solution:

• add two parameters to the actinia "importer" interface to support
  • method: nearest (default), bilinear_f (so, maybe not all offered by r.import)
  • resolution: numeric value

which internally constructs this code:

• resample=bilinear_f resolution=value resolution_value=XX

to be added here:

https://github.com/mundialis/actinia_core/blob/eb8bca301010a2ca71a40def4c0a962e26eb7fd6/src/actinia_core/resources/common/geodata_download_importer.py#L236

Note: Two parameters are sufficient since:

• left empy: "nearest" is used as before
• only resample is set: propagate "resample" to r.import
• only value is set: use "nearest
• both resample and value are set: propagate "resample" and "resolution_value" to r.import

add the actinia logo to starting page and include it in the source code.
Thanks!

With growing content, the Wiki page https://github.com/mundialis/actinia_core/wiki needs a TOC for easier navigation...

At time GPKG support is missing:

"message": "AsyncProcessError:  Mimetype <application/x-sqlite3> of 
 url <https://data.neteler.org/tmp/ne_10m_admin_1_states_provinces_canada.gpkg> is 
 not supported. Supported mimetypes are: application/zip,image/tiff,application/gml,text/xml,application/xml,text/plain,text/x-python", 

Is it correct to add "application/x-sqlite3" here?

https://github.com/mundialis/actinia_core/blob/63813698fa9cd00396dd78e6affab272a4827cc0/src/actinia_core/resources/common/geodata_download_importer.py#L43

And, what about export?

It would be nice to have a possibility to specify a project version which can be seen in the version endpoint.
Maybe the version can been set via an environment variable and if it is set, it will be displayed by requesting e.g. https://actinia.mundialis.de/api/v1/version.
It cood look like this:

{
      "grass_version": {...},
      "plugin_versions": {...},
      "plugins": "actinia_statistic_plugin,actinia_satellite_plugin,actinia_gdi",
      "project_version": "1.0.0",
      "python_version": "3.8.5 (default, Jul 20 2020, 23:11:29) - [GCC 9.3.0]",
      "version": "0.99.19"
}