Giter VIP home page Giter VIP logo

iot-edge-sqlite's Introduction

Archived

This repository is archived and no longer being maintained.

Note

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments

Azure IoT Edge SQLite Module GA

Using this module, developers can build Azure IoT Edge solutions with capability to access SQLite databases. The SQLite module is an Azure IoT Edge module, capable of executing sql queries sent from other modules, and return result to the senders or to the Azure IoT Hub via the Edge framework. Developers can modify the module tailoring to any scenario.

There are prebuilt SQLite module container images ready at here for you to quickstart the experience of Azure IoT Edge on your target device or simulated device.

Visit http://azure.com/iotdev to learn more about developing applications for Azure IoT.

Azure IoT Edge Compatibility

Current version of the module is targeted for the Azure IoT Edge GA.
If you are using the obsolete v1 version of IoT Edge (previously known as Azure IoT Gateway), please use v1 version of this module, all materials can be found in v1 folder.

Find more information about Azure IoT Edge at here.

Target Device Setup

Platform Compatibility

Azure IoT Edge is designed to be used with a broad range of operating system platforms. SQLite module has been tested on the following platforms:

  • Windows 10 Enterprise (version 1809) x64
  • Windows 10 IoT Core (version 1809) x64
  • Linux x64
  • Linux arm32v7

Device Setup

Build Environment Setup

SQLite module is a .NET Core 2.1 application, which is developed and built based on the guidelines in Azure IoT Edge document. Please follow this link to setup the build environment.

Basic requirement:

  • Docker CE
  • .NET Core 2.1 SDK

HowTo Build

In this section, the SQLite module we be built as an IoT Edge module.

Open the project in VS Code, and open VS Code command palette, type and run the command Edge: Build IoT Edge solution. Select the deployment.template.json file for your solution from the command palette.
Note: Be sure to check configuration section to properly set each fields before deploying the module.

In Azure IoT Hub Devices explorer, right-click an IoT Edge device ID, then select Create deployment for IoT Edge device. Open the config folder of your solution, then select the deployment.json file. Click Select Edge Deployment Manifest. Then you can see the deployment is successfully created with a deployment ID in VS Code integrated terminal. You can check your container status in the VS Code Docker explorer or by run the docker ps command in the terminal.

Configuration

Before running the module, proper configuration is required. Here is a sample configuration for your reference.

Container Create Option

To persist the database file, you will need to bind a host folder to container folder. If you are using "Docker for Windows" to run linux container, you will need to propertly configure docker file sharing to enable volume binding.

{
  "HostConfig": {
    "Binds": [
      "<Host Path>:/app/db"
    ]
  }
}

Database Configuration in Module Twin Settings

{
  "SQLiteConfigs":{
    "Db01":{
      "DbPath": "/app/db/test.db",
      "Table01":{
        "TableName": "test",
        "Column01":{
          "ColumnName": "Id",
          "Type": "numeric",
          "IsKey": "true",
          "NotNull": "true"
        },
        "Column02":{
          "ColumnName": "Value",
          "Type": "numeric",
          "IsKey": "false",
          "NotNull": "true"
        }
      }
    }
  }
}

Meaning of each field:

  • "SQLiteConfigs" - Contains one or more SQLite databases' configuration. In this sample, we have "Db01":
    • "Db01" - User defined names for each SQLite database, cannot have duplicates under "SQLiteConfigs".
      • "DbPath" - The absolute path to the db file.
      • "Table01" - User defined names for each table, cannot have duplicates under the same db.
      • "TableName" - Table name of Table01.
        • "Column01", "Column02" - User defined names for each column, cannot have duplicates under the same table.
          • "ColumnName": Column name of Column01, Column02.
          • "Type" - Data type of the column
          • "IsKey" - Is key property of the column
          • "NotNull" - Is notnull property of the column

[NOTE] Please notice that /app/db folder in the sample configuration is created only via binding with permissions allowing module user (non-root) to write. If you don't want to persist the database file on host (not applying above container create option), you will need to specify an existing folder with module user read/write permission, for example "/tmp" . You cannot specify a non-existing folder to store the database file (/app/db folder does not exist unless via folder binding). If you see following error in container log :

Exception while opening database, err message: SQLite Error 14: 'unable to open database file'.
Check if the database file is created or being mounted into the conainter correctly

Then it's probably because you store the database file at an non-existing folder, or the folder you specify has not write permission to module user.

Module Endpoints and Routing

There are two endpoints defined in SQLite module:

  • "sqliteOutput": This is an output endpoint for the result of sql queries.
  • "input1": This is an input endpoint for sql queries.

Input/Output message format and Routing rules are introduced below.

Send SQL Queries to SQLite

SQLite module use input endpoint "input1" to receive commands. Note: Currently IoT Edge only supports send messages into one module from another module, direct C2D messages doesn't work.

Command Message

The content of command must be the following message format.

Message Properties:

"command-type": "SQLiteCmd"

Message Payload:

{
    "RequestId":"0",
    "RequestModule":"filter",
    "DbName":"/app/db/test.db",
    "Command":"select Id, Value from test;"
}

Route from other (filter) modules

The command should have a property "command-type" with value "SQLiteCmd". Also, routing must be enabled by specifying rule like below.

{
  "routes": {
    "filterToSQLite":"FROM /messages/modules/filtermodule/outputs/output1 INTO BrokeredEndpoint(\"/modules/sqlite/inputs/input1\")"
  }
}

Receive Result from SQLite

Result Message

Message Properties:

"content-type": "application/edge-sqlite-json"

Message Payload:

[
  {
    "PublishTimestamp":"2018-09-04 04:16:47",
    "RequestId":"0",
    "RequestModule":"filter",
    "Rows":[
      [
        "1",
        "20"
      ],
      [
        "2",
        "100"
      ]
    ]
  }
]

Route to IoT Hub

{
  "routes": {
    "sqliteToIoTHub":"FROM /messages/modules/sqlite/outputs/sqliteOutput INTO $upstream"
  }
}

Route to other (filter) modules

{
  "routes": {
    "sqliteToFilter":"FROM /messages/modules/sqlite/outputs/sqliteOutput INTO BrokeredEndpoint(\"/modules/filtermodule/inputs/input1\")"
  }
}

HowTo Run

Run as an IoT Edge module

Please follow the link to deploy the module as an IoT Edge module.

iot-edge-sqlite's People

Contributors

microsoft-github-policy-service[bot] avatar microsoftopensource avatar msftgits avatar yphuangms avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

iot-edge-sqlite's Issues

Special characters issue at received result message from SQLite

Characters like ö and ü are received from SQLite messages as question mark characters. I checked the hex value that is received in the message payload that contains these special characters and it seems to be 3F (which is decimal 63 which is ascii character '?').
Any special characters I tested are received from SQLite as question marks.
I checked and can confirm that all characters are correctly stored in SQLite (no question mark characters there).
Is there any way to correctly store, send & receive special characters?

what is the real-world use-case for this module

This is to confirm my understanding of this modules(sorry if this is naive). What I understand from the devdoc folder, this module basically offers an SQLite database to other modules. Other modules could execute SQL queries on need. To rephrase, this module avoids the need to link to SQLite library in other modules. Other modules still need to frame the appropriate SQL queries. Nothing more. Nothing less.

I was looking for a module which offers disk persistence( azure-iot-sdk-c offers in-memory network retries, but data is lost on power loss ). If I have to re-use the functionality in this module, then my Modbus module should have SQL queries to INSERT and my iothub module must have queries to SELECT/DELETE(cleanup) from this module. Is that a fair understanding?

Pull of mcr.microsoft.com/azureiotedge/sqlite:1.0 fails

When deploying the module using the Azure IoT Edge Marketplace the pull of 'mcr.microsoft.com/azureiotedge/sqlite:1.0' fails (No such image... 404).

I tried to pull it by hand:

C:\Users\sande>docker pull mcr.microsoft.com/azureiotedge/sqlite:1.0
1.0: Pulling from azureiotedge/sqlite
no matching manifest for unknown in the manifest list entries

And the Docker Hub refers to 'docker pull mcr.microsoft.com/azureiotedge/sqlite' which is failing too.

Is this module still available?
If so, what is the right container address?
If so, will it work on Windows 10 IoT Enterprise 1809 with Windows containers?

I think this needs to be fixed both in the Marketplace and on Hub.docker.com

How to run the module on Windows 10 IoT Enterprise (Win 10)

I want to run the sqlite module with Windows 10 IoT Enterprise (Win 10 1809 LTS). I make use of Windows containers.

I tried with URI:

mcr.microsoft.com/azureiotedge/sqlite:1.0.5-win10-amd64-1809

Container create options:

{
  "HostConfig": {
    "Binds": [
      "C:\\iotsqlite:c:\\app"
    ]
  }
}

(With full access for everyone to the exiting folder on host)

and desired properties:

"properties": {
  "desired": {
    "SQLiteConfigs": {
      "Db01": {
        "DbPath": "test.db",
       ....

(so there is no path in here)

But this set of settings (and many other combinations) all failed.

Can you provide the correct Windows containers on Windows LTS 1809 setup for the sqlite module which is also available in the iot edge module store?

Regards,

Sander

Jetson_Nano: Could not pull image mcr.microsoft.com/azureiotedge/sqlite:1.0

Hi
I am trying to run a sqlite module on the Jetson Nano & it's failing with the following msg. Need some directions on how to resolve this.

<6> 2021-01-29 13:29:09.439 +00:00 [INF] - Executing command: "Command Group: (\n [Create module SQLiteJetson]\n [Start module SQLiteJetson]\n)"
<6> 2021-01-29 13:29:09.439 +00:00 [INF] - Executing command: "Create module SQLiteJetson"
<3> 2021-01-29 13:29:26.735 +00:00 [ERR] - Executing command for operation ["create"] failed.
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module SQLiteJetson: Could not create module SQLiteJetson
caused by: Could not pull image mcr.microsoft.com/azureiotedge/sqlite:1.0
caused by: no matching manifest for linux/arm64/v8 in the manifest list entries, StatusCode:500, at: at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 232
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 143 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.CreateModuleAsync(ModuleSpec moduleSpec) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 100 at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 59 <3> 2021-01-29 13:29:26.740 +00:00 [ERR] - Executing command for operation ["Command Group: (\n [Create module SQLiteJetson]\n [Start module SQLiteJetson]\n)"] failed. Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module SQLiteJetson: Could not create module SQLiteJetson caused by: Could not pull image mcr.microsoft.com/azureiotedge/sqlite:1.0 caused by: no matching manifest for linux/arm64/v8 in the manifest list entries, StatusCode:500, at: at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 232 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 143
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.CreateModuleAsync(ModuleSpec moduleSpec) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 100
at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 59
at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 59
<3> 2021-01-29 13:29:26.744 +00:00 [ERR] - Step failed in deployment 56, continuing execution. Failure when running command Command Group: (
[Create module SQLiteJetson]
[Start module SQLiteJetson]
). Will retry in -03s.

Also, tried the docker pull
$ docker pull mcr.microsoft.com/azureiotedge/sqlite
Using default tag: latest
latest: Pulling from azureiotedge/sqlite
no matching manifest for linux/arm64/v8 in the manifest list entries

OS info
Jetpack: 4.4.1 [L4T 32.4.4]

Your help is highly appreciated.

Sending incorrect SQL query to SQLite causes infinite crash loop

Sending an incorrect SQL query message crashes SQLite before it can send a message received confirmation to the edgeHub. The edge agent then restarts the SQLite module and the edgeHub (not having received any message confirmation) resends the same incorrect SQL query message to SQLite, making it crash again. This makes the SQLite module end up in an infinite crash loop unless the messages are cleared from edgeHub.

In my case I had a typo in the select query table name and forgot to add ';' at the end of the query.

See below image with error in edgehub
afbeelding

Filter for a specific ID

I'm trying to make a query where I need to select a specific Id, is it possible? Or when I filter data from a table I receive everything?

Something like in SQL:
select Id , Value from test where Id is 123;

Is there any documentation where I can see the commands I can make besides insert and filter? Maybe a "update" or "delete"....

Default deployment using Marketplace fails

I try to deploy the module as created by the Azure IoT Edge MArketplace deployment. This is the result:

uno2271gsv@uno2271g2sv-UNO-2271G-E2xAE:/app/db$ sudo iotedge logs -f SQLite
IoT Hub module client initialized.
Desired property change:
{"SQLiteConfigs":{"Db01":{"DbPath":"/app/db/test.db","Table01":{"TableName":"test","Column01":{"ColumnName":"Id","Type":"numeric","IsKey":"true","NotNull":"true"},"Column02":{"ColumnName":"Value","Type":"numeric","IsKey":"false","NotNull":"true"}}}},"$version":1}
Attempt to load configuration: {"SQLiteConfigs":{"Db01":{"DbPath":"/app/db/test.db","Table01":{"TableName":"test","Column01":{"ColumnName":"Id","Type":"numeric","IsKey":"true","NotNull":"true"},"Column02":{"ColumnName":"Value","Type":"numeric","IsKey":"false","NotNull":"true"}}}},"$version":1}
Exception while opening database, err message: SQLite Error 14: 'unable to open database file'.
Check if the database file is created or being mounted into the conainter correctly

I tried to create the /app/db folder using sudo rights but this did not change a thing. Neither did giving the folder more rights or creating an empty file with the right filename.

What is happening?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.