esri / insights-scripting-guide Goto Github PK

Hi,

I'm trying to set up the scripting environment for Insights desktop, I followed the instructions in the scripting guide but when I try to connect o the Python kernel I get the message "Unable to use Insights data or bring in data from Jupyter Kernel Gateway to Insights. One or more libraries are missing from your Jupyter Kernel Gateway. Review the ArcGIS Insights Scripting Guide for more information". I checked that the dependencies are all installed, but I keep getting the same message. When I try to connect to the R kernel is all good however. Do you know what might be going on?

Thank you!

Hello there!,

I am trying to test scripting on ArcGIS Insights in the Enterprise version, but when I try to connect to the kernel it never starts, it keeps loading and in the command prompt there is no answer from the gateway. Also it does not show any error.

I´ve already tested my conection using ArcGIS Insights Local and it works correctly, its only on the portal were I find this issue.
Attached you can see a screenshot of the Command Prompt when I try to start the kernel. From there it doesn´t do anything else.

Thanks for any help

hi there,

I am using the magic command %insights_return(df) to add an Pandas dateframe but I received this error after running the cell:
NameError: name '__get_column_types' is not defined

What am I doing wrong?

Thanks very much

My container gateway fails to connect while my anaconda gateway connects fine.

After comparing the installed modules in these two python environment, I find the version of the jupyter gateway in anaconda is 2.1.0 , but in the container the version is 2.4.0.

A simple change in the dockerfile to force pip to install 2.1.0 seems fix the connection problem.

Is it an isolated case, or it is true that the latest Jupyter gateway v2.4.0 does have issue when connecting from Insights

Is it possible to define a customized graph labeling that the label of a vertex is sum of the weights of incident edges and visualize the induced label by the size of the vertices?

Moreover, I would like to know some basics information about a graph, such as eccentricity of a vertex, edge-cut, spanning tree of a graph.

Is there any tools/ script can solve these problems?

Hi there,

We're trying to test scripting on Insights, but when we try to connect to the kernel it never starts, it keeps loading and in the command prompt there is no answer from the gateway. Also it does not show any error. Similar to another issue here, but we're running local.

Following the guide, we have used the following ssl config:

[dn] CN=BC.localhost.com [req] distinguished_name=dn [EXT] subjectAltName=DNS:BC.localhost.com keyUsage=digitalSignature extendedKeyUsage=serverAuth

And created with
openssl req -x509 -days 365 -out BC.crt -keyout BC.key -newkey rsa:2048 -nodes -sha256 -subj "/CN=BC.localhost.com" -extensions EXT -config insightsgw.cnf

Then started kernel
jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=C:\insightsgw\BC.crt --keyfile=C:\insightsgw\BC.key

Checking in browser seems ok?

But connecting in Insights starts the loading screen which runs indefinitely

Not sure what we're doing wrong? Any help would be much appreciated. Thanks

I am running ArcGIS Insights 2022.1.x on Windows, I already have ArcGIS Pro installed, and primarily use the Python installed with ArcGIS Pro. It seems like the instructions here may require a separate Python installation to create the Jupyter Kernel Gateway even though ArcGIS Pro uses conda (not Anaconda...I know there is a difference, but I am not completely sure if that applies here). I am possibly missing something here, but is there a way to just connect ArcGIS Insights to the ArcGIS Pro Python installation? Seems like that would be the easiest implementation as I would expect most Insight users to also have ArcGIS Pro. Thanks.

Github projects can have Releases.

It is possible to wire certain actions, notifications, mechanisms, etc. to these releases.

Could you please set up releases for this Github repository?

Using the insights-2022.yml will consume all the RAM memory of the PC and the process will never complete.

The latest version of the .yml file works fine, but we are unable to install the 2022 version.

When use the container option, it fails to connect from Insights, while the anaconda option works.

I compared the two options, and found the cause:

The version of the jupyter gateway in anaconda is 2.1.0 , but in the pip repository, the version is 2.4.0, which will be installed when building the docker image.
So change the dockerfile to force pip to install gateway 2.1.0, it will be working.

Hello,

I need to use ArcGIS Insights using Python, and it has been decided to use Jupyter's Kernel Gateway via Docker.

Docker is installed and configured (a laptop using Windows 10), has been tested, and works normally. The python script selfsign.py has been created to create the scripts (by the way, I would make a change because as it is, it generates the certificate with the hostname and not the FQDN).

In the gateway folder, there is a Dockerfile, the insights-latest.yml (I assume it is not used), the README.md, the script selfsign.py, and the public key as a private key of the certificate (server.crt and server.key).

At this point, I have a doubt.
5. Create a data folder within gateway and put your data files there.

Which files should be copied there? could you please be more explicit?
In this case, I have created the data folder and copied the same files mentioned before.

When executing it, I have errors when creating the docker image.

docker build -t insights-gateway .

ERROR: failed to solve: process "/bin/bash -o pipefail -c conda install --quiet --yes 'rpy2=2.8*' 'r-base=3.4.1' 'r-irkernel=0.8*' 'r-plyr=1.8*' 'r-jsonlite=1.5*' 'r-data.table=1.11.4*' 'r-devtools=1.13*' 'r-tidyverse=1.1*' 'r-shiny=1.0*' 'r-rmarkdown=1.8*' 'r-forecast=8.2*' 'r-rsqlite=2.0*' 'r-reshape2=1.4*' 'r-nycflights13=0.2*' 'r-caret=6.0*' 'r-rcurl=1.95*' 'r-crayon=1.3*' 'r-randomforest=4.6*' 'r-htmltools=0.3*' 'r-sparklyr=0.7*' 'r-htmlwidgets=1.0*' 'r-hexbin=1.27*' && conda clean -tipsy && fix-permissions $CONDA_DIR && fix-permissions /home/$NB_USER" did not complete successfully: exit code: 1

At least in my case, R is not needed so I have modified the Dockerfile.

# Jupyter image w/ Python
FROM jupyter/scipy-notebook

LABEL maintainer="Insights"

USER root

WORKDIR /data

COPY server.crt /etc/ssl/certs/
COPY server.key /etc/ssl/certs/

# uncomment to copy data to the container
COPY /data/* /data/

USER $NB_UID

# install the kernel gateway, arcgis Python API, geopandas, msgpack, shapely, and requests
RUN pip install --quiet jupyter_kernel_gateway==2.1.0 arcgis geopandas msgpack shapely requests

# environment settings for kernel gateway
ENV KG_ALLOW_ORIGIN='*'
ENV KG_LIST_KERNELS='True'

# run kernel gateway on container start, not notebook server
EXPOSE 9999

# set tls (https) on the kernel gateway
CMD ["jupyter", "kernelgateway", "--KernelGatewayApp.ip=0.0.0.0", "--KernelGatewayApp.port=9999", "--KernelGatewayApp.allow_credentials='*'", "--KernelGatewayApp.allow_headers='*'", "--KernelGatewayApp.allow_methods='*'", "--KernelGatewayApp.certfile=/etc/ssl/certs/server.crt", "--KernelGatewayApp.keyfile=/etc/ssl/certs/server.key"]

I rebuild the Docker image, and no error messages.

docker build -t insights-gateway .

[+] Building 205.5s (11/11) FINISHED docker:default => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.02kB 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/jupyter/scipy-notebook:latest 1.0s => [1/6] FROM docker.io/jupyter/scipy-notebook@sha256:fca4bcc9cbd49d9a15e0e4df6c666adf17776c950da9fa94a4f0a045d5 0.0s => [internal] load build context 0.0s => => transferring context: 4.49MB 0.0s => CACHED [2/6] WORKDIR /data 0.0s => CACHED [3/6] COPY server.crt /etc/ssl/certs/ 0.0s => CACHED [4/6] COPY server.key /etc/ssl/certs/ 0.0s => [5/6] COPY /data/* /data/ 0.1s => [6/6] RUN pip install --quiet jupyter_kernel_gateway==2.1.0 arcgis geopandas msgpack shapely requests 202.3s => exporting to image 2.0s => => exporting layers 2.0s => => writing image sha256:e5fe6b9d09c9e3351cb86ad713959ce633ffc0714d29950f15131ad37598433e 0.0s => => naming to docker.io/library/insights-gateway 0.0s

What's Next? View summary of image vulnerabilities and recommendations → docker scout quickview_

I start the Kernel Gateway.

docker run -p 9999:9999 insights-gateway

[KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

The container is created in Docker.

But when accessing.

http://localhost:9999/
https://localhost:9999/
http://server.domain.name:9999/
https://server.domain.name:9999/
http://10.1.1.2:9999/
https://10.1.1.2:9999/

I should receive a response in JSON.
{"reason": "Not Found", "message": ""}

But in this case, I have the log in Docker.

[KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999
[W 240319 16:05:05 iostream:1382] SSL Error on 8 ('172.17.0.1', 49888): [SSL] internal error (_ssl.c:1006)
[W 240319 16:05:05 iostream:1382] SSL Error on 9 ('172.17.0.1', 49890): [SSL] internal error (_ssl.c:1006)
[W 240319 16:05:05 iostream:1382] SSL Error on 8 ('172.17.0.1', 49892): [SSL] internal error (_ssl.c:1006)
[W 240319 16:05:05 iostream:1382] SSL Error on 9 ('172.17.0.1', 49894): [SSL] internal error (_ssl.c:1006)
[W 240319 16:05:06 iostream:1382] SSL Error on 8 ('172.17.0.1', 49896): [SSL] internal error (_ssl.c:1006)
[W 240319 16:05:06 iostream:1382] SSL Error on 9 ('172.17.0.1', 49898): [SSL] internal error (_ssl.c:1006)

Any idea / recommendation to solve the issue?

Regards.

Try to get the data from the pane.
But I have an error:
Error in eval(expr, envir, enclos): object 'points.salesamtho' is not found.
(I added the sample geojson to workbook, that data set name is "points".)

Can support for the Plotly package for R be added? It looks like the chart_studio module is supported. Currently the R scripting environment lets you execute code to create a plotly chart, but the output is empty (and the image card).

It would be great to have the full functionality of Plotly in Insights :)

This is what I get:

SpecsConfigurationConflictError: Requested specs conflict with configured specs.
requested specs:
- rpy2=2.8
pinned specs:
- python=3.7
Use 'conda config --show-sources' to look for 'pinned_specs' and 'track_features'
configuration parameters. Pinned specs may also be defined in the file
/opt/conda/conda-meta/pinned.

Context

• Subject : Solving dependencies unsuccessful
• Resume : The creation of the Python virtual environment is unsuccessful and remains stuck on resolving dependencies.

Environment

• ArcGIS Enterprise : 11.1 (without "badpatch")
  • Machine 1 : Portal / WebServer
  • Machine 2 : Server
• ArcGIS Insights : 2023.3
• Strategy : Co-location on ArcGIS Server on machine 2

Steps

Create project and gateway directory into :

D:\poc-insights-gateway\gateway

Copy ./gateway/selfsign.py and .gateway/insights-latest.yml

Edit insights-latest.yml to remove unnecessary r-packages :

name: insights-latest
channels:
  - conda-forge
dependencies:
  - python=3.9
  - jupyter_kernel_gateway=2.5.1
  - pandas=1.5.1
  - shapely=1.8.5
  - requests
  - msgpack-python=1.0.4
  - matplotlib
  - geopandas=0.11.1
  - pyspark=3.3.1
  #- r-itertools
  #- r-essentials

Create new conda env from ArcGIS Server conda.exe :

# Load
C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\Scripts>proenv.bat

# Init env into project dir
(arcgispro-py3) C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\Scripts>conda env create -f D:\poc-insights-gateway\gateway\insights-latest.yml -p D:\poc-insights-gateway\gateway\env\ags111insights20233

# ...

At this point, solving environment takes forever. Stop after 24h running.

According to the documentation, segregate the creation of env. :

# Load
C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\Scripts>proenv.bat

# create new env
(arcgispro-py3) C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\Scripts>conda create -p D:\poc-insights-gateway\gateway\env\ags111insights20233 -c conda-forge python=3.9

# load new env
(arcgispro-py3) C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\Scripts>activate D:\poc-insights-gateway\gateway\env\ags111insights20233

# move to directory
(D:\poc-insights-gateway\gateway\env\ags111insights20233) C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\Scripts> cd D:\poc-insights-gateway\gateway

# Install step 1
(D:\poc-insights-gateway\gateway\env\ags111insights20233) D:\poc-insights-gateway\gateway>conda install -c conda-forge jupyter_kernel_gateway=2.5.1 pandas=1.5.1 shapely=1.8.5 msgpack-python=1.0.4 matplotlib

# [...]

At this point, it works after ~1H.

Install geopandas

(D:\poc-insights-gateway\gateway\env\ags111insights20233) D:\poc-insights-gateway\gateway>conda install -c conda-forge geopandas=0.11.1

#[ ...]

Stuck here with solving dependencies. Stop after ~2H.

Can you please fix all the package versions used in the yml?

Thank you in advance

My ArcGIS Enterprise is using Integrated Windows Authentication, it seems the python gateway container can't handle the windows authentication, as a result, it can't access the data sources from the ArcGIS Enterprise (Portal).

It does sound odd to let a Linux container handle windows authentication... but I am no expert in this area.

Are there any work around for this?