A github release has been created. This should be zipped and released to QGIS repo

Prior to release user testing will need to occur.

• I will initially test with a windows machine (as this has had little attention)
• Send out to users for testing
• Document tests for users to conduct and confirm they had no issues

Store users favourites

This could be either via users selecting, marking and storing their favorites

or

Via storing import an talley and showing a top x layers by count of previous imorts

or

This could be a combination of both of the above

The QGIS Project has undergone significant redevelopment. This has resulted in the release of a new API (QGIS3) that breaks compatibility with QGIS2. For the plugin to remain operational this will need to be modified to match the new QGIS API.

I think relabelling the “import dataset” to “Add layer” is a better option as the dataset is not really being imported only referenced by remote webservices (well not entirely true for WFS which does a full dataset cache for the session, but that's a small detail!)

Currently the plug deletes all cached data and then replaces it in a thread.

However, if QGIS is closed while the thread update is in process this means some of the delete files will not have been replaces.

Therefore, The plugin should download the new files and then replace the old only once all files have been downloaded.

Currently filtering is regex this means that users must 1) know this 2) understand regex NZ

Currently users must input the below to return only NZ Coastlines topo 1:50k.
NZ Coastlines \(Topo, 1:50k\)
The needs to be improved to handle
NZ Coastlines (Topo, 1:50k)

Currently WGS 84 Web Mercator (EPSG:3857) is used by default as all LDS WMTS services are served in this projection and it is not always a good idea to reproject WMTS.

To improve on this I suggest:

• WMS should be added using the QGIS project projection if supported. If the project projections is not supported then default it to WGS84. Providing a combo dropdown box option on the row for user selection is also another good idea
• For WMTS provide all tilematrix grid options in the UI using a combo dropdown box option on the row
• WFS I think you are limited to the datasource CRS only.

Currently the service_data object has many return methods. These should merely modify the objects data

Also separate out locale store data and methods in to a localstore class. This class should be inherited by the service_data object and also have the ability to be initiated and manage local store data independent of a service_data object

When the user selects the item row from the dataset grid it would be nice to display the LDS thumbnail next to the metadata to give the user a nice preview of the data which can be added.

The thumbnail service URL is https://data.linz.govt.nz/services/tiles/v4/thumbnail/layer=LAYERID,style=auto/WIDTHxHEIGHT.png

Where:

• LAYERID = The LDS layer ID
• WIDTH = The width of thumbnail in pixels. Max 1200
• HEIGHT = The height of thumbnail in pixels. Max 1200

e.g https://data.linz.govt.nz/services/tiles/v4/thumbnail/layer=50772,style=auto/250x250.png

If I want to load e.g. the Auckland Lidar DEM WMS (not WMTS), there is apparently a vast number of CRSs available (perhaps this is an error though, because they seem too many).
They are all in a list with just the EPSG numbers (not ideal, but people can probably cope with it), but the list isn't sorted, so it is impossible to find the one you want - see screenshot.

I have heard the LINZ addressing team have has issues with the LINZ-Data-Importer with their work flow.

I have no other information so at this stage I will just visit them to understand what the issue is.

In the menu. the tool is call linz-data-importer. Looks like convention in QGIS is to have Title Case menu items, i.e. LINZ Data Importer

WMS is an old standard and is due to be turned off in the LDS in Aug/Sep 2020.

1. when you successfully save an API key the save button is still there next time you go to settings. Should it disappear or be replaced by something else like 're-save'?
2. when an API key is incorrect, the save/delete buttons are still there and the field still shows you the API key. Should it do something different, like show there was an error or something?

Plugin is no longer scoped to be LINZ specific but is still NZ specific

LDS needs to be removed from the icon.
@pkinglinz do you have an pixels laying about such as the below but without "LDS" on it?

See #7 at least within the LINZ network GetCapabilities requests are very slow.
This is also related to the speed of LDS server responses.

I will test the plugins speeds at home to bench mark the performance.

Options:

Threading
the different WxS services could be thread to run concurrently

Caching
Store data on the users machine and update in the back ground once a QGIS session has been started

as per QGIS docs

they plugins should make use of QgsNetworkAccessManager instead of home brew urllib2/requests/... solutions, which most of the times do not make use of proxy settings

As per below new QGIS releases released newer python owslib versions. But with 3.4 a much older version of the python owslib was packaged.

QGis.QGIS_VERSION
'2.18.16'
owslib.__version__
'0.14.0'

Qgis.QGIS_VERSION
'3.2.3-Bonn'
owslib.__version__
'0.16.0'

Qgis.QGIS_VERSION
'3.4.0-Madeira'
owslib.__version__
'0.10.3'

in-light of #44 not being resolved the QGIS2 version of the plugin, the repo could be updated with metadata to exclude qgis users using versions of qgis>2.18.20

This is because if the user is using qgis versions > 2.18.20 wfs data will not import.

Consideration: Do you let users with qgis > 2.18.20 have the plugin where wfs silently fails to import or do you stop these users having the plugin at all?

Have the abilities to add others services via the settings tables

This will need to handle the unique URL formatting for each service.
Currently the domain is substituted out for koordinates based portals

Will also need tot handle (start by storing in the config) the different service versions (wfs3.0 vs wfs2...)

CI needs to be scheduled.

For example, if I know I'm looking for a layer in https://data.linz.govt.nz/data/category/topographic/ but keep getting a bunch of Hydrographic charts in my search results.

Just a hypothetical use case. 😉

e.g:

• https://data.mfe.govt.nz/
• https://lris.scinfo.org.nz/
• https://datafinder.stats.govt.nz/

When adding plugin to LINZ plugin server, each time "No metadata.txt file" is return

• All methods doc stringed
• Update read me
• Include developer notes - Particularly concerning test methodology
• Update about tab (Text and remove LINZ icon)

This really only relates to services with WMS (currently only LDS).

When the domain details are saved for the first time and the plugin awaits get capability doc responses the plugin appears to freeze (as wms takes a long time to respond)

The currently solution has a label warning this may take some time. This has low visibility. When hitting save a pop up dialog should be shown and warn the user.

Users could style layers and push the qml to a cloud service.

On next import they could go and select that style for their layer.

These could also be publicly published and other user could user others layer style sheets. The plugin could show a list of other users styles associated with the current layer

Background

The package name (linz-data-importer) followed the poor practise of using hyphens. This became an issue when promoting a custom widget via QT Designer to solve #47. The problem was when the python UI code was dynamical generated by PytQ5.uic.loadUi, the statement import x from linz-data-importer.gui.x was output. The hyphens makes this an invalid python import.

To solve this, the package name was renamed to "linz_data_importer" (I note pep8 allows underscores).

Problem
The problem now however is that the plugin can not be loaded to the QGIS plugin server as part of the current repo as it will not allow versions with different names to be loaded.

Solution
It appears the only option available is to rename the plugin and start a 'new' plugin (as far as the QGIS plugin server is concerned).

The plugin will therefore need a new name.

• LINZ Data Importer v2?

Problems with the solution
I would prefer not to undertake the confusing step of renaming the plugin but am unaware of another solution.

Renaming the plugin will have the below negative outcomes:

• Old versions and download stats from the previously named plugin will not carry across to the renamed one.
• Users of the current plugin will not be notified of the existence of a new plugin.
• The plugin name will no longer directly relate to the source code repository name.
• having to plugins with the same name will be confusing to users.

Further consideration
Should the code repository be forked and renamed?

On stand-alone LINZ machines that IT have installed a LINZ corp windows build, WMTS layers appear to import correctly (layer is added to "layers Panel") but are not visible.

• This seems to be consistent for users of such environments
• I can not repeated this in other os environments that I have access to.

I have tried to replicate this problem with the below os environments but can not repeat it

• My ubuntu machine
• LINZ DaaS
• Topo's standard windows 10 machine
  • Windows 10 Enterprise (v1803)

I will need to get access to a LINZ corp built machine to debug further

Travis is failing to parse encrypted travis env var when in json format

The json env var must be escaped for bash parsing see note-on-escaping-certain-symbols.

envi var is in the format of

travis encrypt 'LDI_API_KEYS={\"data.mfe.govt.nz\":\"<KEY1>\",\"data.linz.govt.nz\":\"<KEY2>\",\"geodata.nzdf.mil.nz\":\"<KE31>\"}' --add

as well as the above I have escaped chars as per https://unix.stackexchange.com/questions/270977/what-characters-are-required-to-be-escaped-in-command-line-arguments

And others

This has seen the trail of combinations of the escaping of {, , }, .` and '.'

This has resulted in either an env var fail to be created on the travis vm or `LDI_API_KEYS={'

• All ogc koordinate platforms to be supported thus name is no longer representative
• "Plugin" should not be in the name

Ideals (Brain dump - Please add more):

• NZ Open Data
• NZ Data Importer
• NZ Govt Viewer - maybe broader than govt
• NZ Map
• Data NZ
• Open Data NZ
• NZ2Qgis
• NZ Data Browser

The layer column, which contains the name of the layer from the chosen data service, is currently the last column in the table and on the initial install of the plugin, it is outside the visible extent of the table. The user needs to either change the column widths or use a scroll bar to see this column.

I think that the table would be more user friendly if these changes were made:

• the "layer" column should probably be second (after the "domain") and renamed to "layer name"
• the "service" column could be removed, as its duplicating information - you're already on the WMTS tab, you don't need to know that all of the layers you're seeing are WMTS
• is the "type" column useful, in what context does it not say "layer"?
• the "id" column could then be third

There is a need to store a universal plugin config

For example a list of default web portal urls that can be updated and displayed in the plugins settings. By having this in S3 plugin admins can update all users defaults (for example as more koordinates syndicated users come on line).

This should be in json and read access to all.

Docker container running QGIS in CI is failing to connect to the internet. This must be resolved prior to release

When installing the plugin from the official QGIS repository the description text is:

Import LINZ (and others) OGC Datasets into QGIS

I think we could risk a few more words here to inform the casual plugin browser and sell the value of it. Would suggest getting someone with comms/marketing skills to help.

For example AusMap has this as their description.

AusMap gives quick access to a variety of feature sets and basemap layers provided by Geoscience Australia and other government agencies. There are also some additional layers provided such as Open Street Map. From the AusMap menu, simply select a desired layer and it will be added to the project with styling and labelling applied. The layers are provided via WMS, WFS, WCS and Tile Server which all require an active internet connection.

Epic managing tasks that must be complete prior to release

At the moment the QGIS UI locked up for over 2 minutes when the API key is saved or the plugin UI is started for the first time in a new QGIS session. This is because loading the LDS WxS capability documents is slow

It would be good to add some type of non-blocking progress bar when loading the WxS capability documents.

Trying to find 'NZ Aerial Imagery' layer returns no results

Story

As a new plugin user, I want it to be easier to get and add my API keys to the plugin, so that I can add my API keys quickly and get started using the plugin.

Acceptance Criteria

• Given I am a new user who has never used the plugin before, when I open the plugin for the first time, then I am presented information/links that helps me create and add and API key to the plugin
• Given I have created an API key, then I am presented with a list of data that I can start to use
• Given I am a new user who has never used a Koordinates service before, then I am presented information on how to create a Koordinates account
• Given I am a new user who has never used the plugin before, when I have obtained my API key, it is simple for me to add it to the plugin settings

Tasks

• Move help information from the 'about' page to a 'help' page
• Restructure and rewrite the 'help' page to make it more user friendly and get to the important info earlier
• Add direct links to API key pages, e.g. https://data.linz.govt.nz/my/api/ and https://basemaps.linz.govt.nz
• Add info on how to create a Kx account (is this needed?)
• Remove external links where possible (e.g. list of services)
• Default to the settings page if plugin is opened with no API keys saved
• If no API keys have been added then add a user friendly message that explains what the user needs to do to add an API key
• Remove the 'blank' option for the dropdown for API keys
• Reorder the dropdown of domains to: LDS, Basemaps and then others in alphabetical order
• Remove the UI freeze message from settings screen (no longer relevant)
• Add a message to the settings screen linking to the help screen to find out how to get an API key

Release addressing

• #36 - compatibility with 2.8 broken
• #28 - Make cache updating more robust
• and possibly #38 - WMTS Issues with plugin in linz corp windows build envi

The ability to filter layers via changeset

When handling the different capability documents the crs_options list is standardised.

for example "urn:ogc:def:crs:EPSG::" is striped from wms and wfs has 'EPSG:' stripped

Since adding this the ui is slightly slower.

This is implemented via list comprehension. Options for optimisation to be investigated

currently "settings" only handles the entry of up to 10 domains. Error handling must be enacted after 10

I have not been able to get the boundless docker image to execute QGIS3 unittests. However elpaso's image looks much more promising

Discovered resolving current mac issue #48

On the mac I am testing with urllib.urlopen is throwing an SSL cert. error

Running a simple test in the QGIS python console using urllib returns the bellow

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "<string>", line 10, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>

Below error reported by user

In trying to load the plug-in into QGIS (v3.4.4) running on my Mac (v10.13.6) the error shown below is produced. It is a showstopper.

Couldn't load plugin 'linz-data-importer' due to an error when calling its initGui() method

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 607: ordinal not in range(128)

Traceback (most recent call last):
  File "/Applications/QGIS3.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 344, in startPlugin
    plugins[packageName].initGui()
  File "/Users/petersharris/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/linz-data-importer/linz_data_importer.py", line 285, in initGui
    about_html=file.read()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 607: ordinal not in range(128)
Python version: 3.6.6 (v3.6.6:4cf1f54eb7, Jun 26 2018, 17:02:57) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] QGIS version: 3.4.4-Madeira Madeira, exported

Python Path:

/Applications/QGIS3.app/Contents/MacOS/../Resources/python
/Users/petersharris/Library/Application Support/QGIS/QGIS3/profiles/default/python
/Users/petersharris/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins
/Applications/QGIS3.app/Contents/MacOS/../Resources/python/plugins
/Library/Frameworks/SQLite3.framework/Versions/D/Python/3.6
/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
/Users/petersharris/Library/Application Support/QGIS/QGIS3/profiles/default/python
/Users/petersharris/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins
/Users/petersharris/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/mmqgis/forms

The error is related to reading macrons in the about.html doc
about_html=file.read()

This is only related to osx. It has not be reported by the many linux and windows users (it would be, does not let the user use the plugin)

Solutions:

• have an alternative doc without macrons that is read when this fails
  • But what about macrons in the capacities doc?
• Review encoding (this was changed with the switch to python3).