astarte-platform / astartectl Goto Github PK

See #118, this means that we want to be able to add a flag --active-until. Right now this is not possible since only the last connection and disconnection are saved and this erases the possibility of moving the activity window end.

This issue will track the implementation of the feature when Astarte APIs allow to do so.

According to the motivations provided in astarte-platform/astarte-kubernetes-operator/issues/329, since astartectl and the Astarte Operator share the same versioning scheme, strip leading zeros from version numbers in astartectl releases (starting from release 23.5.0).

AppEngine supports device metadata now, following additional features are required in astartectl:

• display device metadata
• add key value pair to device metadata
• remove key value pair from device metadata
• update value for given device metadata key

astartectl cluster install-operator is not supported anymore on Astarte 1.0, therefore it should be removed.

realm-management interfaces save command might be renamed to something like realm-management interfaces pull or ... fetch, since ... save might have an ambiguous semantic.

astartectl utils gen-jwt allows to pass custom claims, but right now it does not easily support custom combination (e.g. a claim X on Pairing API and a different claim Y on AppEngine API).

The API of the command must be rethinked to allow such use cases.

I am trying to setup a context for astartectl to use with Astarte in 5 minutes guide, but I am not sure about the correct procedure to do so. After following that guide I now have keys for housekeeping and the test realm, so this is what I am trying:

$ astartectl version
astartectl 1.0.0-beta.2
$ astartectl config clusters create testcluster --housekeeping-url http://localhost:4001/ --housekeeping-key compose/astarte-keys/housekeeping_private.pem
Cluster testcluster saved successfully
$ astartectl config contexts create testcontext --cluster testcluster --realm-name test --realm-private-key test_private.pem
warn: Error while loading configuration: Config File "astartectl" Not Found in "[/home/davide/.config/astarte]"
Context testcontext saved successfully
$ astartectl config contexts list
warn: Error while loading configuration: Config File "astartectl" Not Found in "[/home/davide/.config/astarte]"
file does not exist
$ astartectl config set-current-context testcontext
warn: Error while loading configuration: Config File "astartectl" Not Found in "[/home/davide/.config/astarte]"
file does not exist
$ astartectl version
warn: Error while loading configuration: Config File "astartectl" Not Found in "[/home/davide/.config/astarte]"
astartectl 1.0.0-beta.2

It seems there's some trouble with config files, however a couple of files were created:

$ cd $HOME && ls .config/astarte/*
.config/astarte/clusters:
testcluster.yaml

.config/astarte/contexts:
testcontext.yaml

Not sure if this is the supposed way to setup a context, maybe issue #107 is related.

Right now it's using the async version, and sometimes this leads to (silent) failure in the installation of some interfaces

Cannot manage a device with UUID starting with '-' char.
Ctl returns : unknown shorthand flag: '_' in -_cy******************

The upcoming release of Astarte v1.1 will include the new resource type of Trigger Delivery Policies.
A set of CRUD APIs are available and existing policies can also be referenced by Triggers.
Supporting these operations on astartectl is desirable.

Further references on Trigger Delivery Policies:

1. The JSON Schema, as per Astarte v1.1.0-alpha.0, for how Trigger Policies can be defined: astarte-platform/astarte#554 (comment)
2. The Swagger file that defines the REST APIs available to manage Trigger Policies: https://github.com/astarte-platform/astarte/blob/master/apps/astarte_realm_management_api/priv/static/astarte_realm_management_api.yaml
3. Documentation about the concepts of Triggers / Trigger Policies: https://docs.astarte-platform.org/snapshot/060-triggers.html#trigger-delivery-policies
4. Documentation about using Triggers / Trigger Policies: https://docs.astarte-platform.org/snapshot/060-using_triggers.html#trigger-delivery-policies

In Astarte 0.11 mappings have two new fields:

• datatabase_retention_policy: possible values "no_ttl" and "use_ttl", default if not present: "no_ttl".
• datatabase_retention_ttl: possible values integers >= 0.

Right now astartectl does not support them, and since interfaces are deserialized and re-serialized with astartectl realm-management interfaces install, the result is that those fields are silently dropped.

A standard release cycle for the Astartectl must be defined.

Given the existing correlation with the Astarte Operator, Astartectl must share both the versioning scheme and the release dates with the Operator. The full specification can be found here.

Where can I find the configuration options for the config file? Trying to replace this command with a config

astartectl realm-management triggers install my_trigger.json --realm-management-url http://localhost:4000/ -r test -k test_private.pem

so options like the realm-management-url, housekeeping-url, etc.

Housekeeping API now allows, when explicitly permitted, to delete realms (see astarte-platform/astarte#435).
Add housekeeping realms delete command to exploit this new feature.

This might come in handy for testing purposes, e.g. if a self-signed certificate is used.

I tried to install astartectl on a Ubuntu 14.04 machine, but didn't succeed. After executing the installation steps describe on the README.md, I got the following message:

builder@builder:~$ echo $GOPATH

builder@builder:~$ vim ~/.bash_profile
builder@builder:~$ source ~/.bash_profile 
builder@builder:~$ go get github.com/astarte-platform/astartectl
package github.com/astarte-platform/astartectl
	imports code.cloudfoundry.org/bytefmt: unrecognized import path "code.cloudfoundry.org/bytefmt"
package github.com/astarte-platform/astartectl
	imports github.com/araddon/dateparse
	imports github.com/dgrijalva/jwt-go
	imports github.com/google/uuid
	imports github.com/jedib0t/go-pretty/table
	imports github.com/go-openapi/strfmt
	imports github.com/asaskevich/govalidator
	imports github.com/go-openapi/errors
	imports github.com/mitchellh/mapstructure
	imports go.mongodb.org/mongo-driver/bson
	imports github.com/go-stack/stack
	imports go.mongodb.org/mongo-driver/bson/bsontype
	imports go.mongodb.org/mongo-driver/bson/primitive
	imports github.com/mattn/go-runewidth
	imports github.com/spf13/cobra
	imports github.com/spf13/pflag
	imports github.com/spf13/viper
	imports github.com/fsnotify/fsnotify
	imports golang.org/x/sys/unix
	imports math/bits: unrecognized import path "math/bits"
builder@builder:~$ astartectl
astartectl: command not found
builder@builder:~$ echo $GOPATH
/home/builder/go

Right now astartectl has the astartectl appengine devices credentials inhibit command but there's no way to reenable the credentials using it.

The API should look something like this
astartectl devices list --active-since=2020-11-01
This is already possible, the client side code should look something like this (assuming t as the "since" timestamp):

• Retrieve the detailed devices list
• Filter out all devices that have connected: false
• Filter out all devices that have a disconnection < t

When a value is sent on an integer mapping with the appengine devices send-data command, astartectl incorrectly parses it as int64, resulting in the following error:

> astartectl appengine devices send-data $DEVICE_ID $SERVER_OWNED_INTERFACE /an/integer/path 10
Value int64 for mapping does not match type restrictions for integer

The --payload-type flag does not fix the issue.
Astarte integers are int32, while longintegers are int64.

We're moving all Astarte components towards using EC cryptography, astartectl should do that too

astartectl appengine devices get-samples fails on object aggregated interfaces with the following error

json: cannot unmarshal object into Go value of type []client.DatastreamAggregateValue

As the title says, pretty much.

Unsetting a server-owned property seems to be currently impossible.

After the rework of the astarte-go client (see astarte-platform/astarte-go/issues/33) the creation of a realm is not possible when a replication strategy is explicitly set. The issue is due to an extra layer of quotes in the json payload which is sent to Astarte.

This behavior affects astartectl v22.11.

It looks like that devices list is missing some devices, that might be caused by buggy usage of paginated lists.

Probably useful:
https://github.com/moul/http2curl

The command astartectl appengine devices send-data when invoked with a non-existing interface prints the error "send-data makes sense only for server-owned interfaces", while it should print something that makes the user understand the actual problem.

Adding support to "save CSV to S3 bucket" when downloading data from Astarte would be quite useful for data science scientists.

Expected behavior: astartectl would read data chunk by chunk from Astarte API and save them to a remote storage, therefore downloaded data might exceed local storage size.

This addition would also allow implementing naive export cronjobs.

Running the astartectl appengine device data-snapshot command on a registered device to show the summary of its properties the following error occurred:

{
  "errors": {
    "detail": "Not found"
  }
}

Looking at the docker-compose logs it seems there is an error in the following HTTP request:

astarte-appengine-api_1         | level=info ts=2023-05-12T14:30:02.268+00:00 msg="Got client request." method=GET request_path=/v1/rust/interfaces/org.astarte-platform.rust.examples.individual-properties.ServerProperties module=PlugLoggerWithMeta function=call/2 request_id=F15ro4erGdJN5ToAADlI tag=got_client_req
astarte-appengine-api_1         | level=info ts=2023-05-12T14:30:02.268+00:00 msg="Sent reply." status_code=404 elapsed=163µs module=PlugLoggerWithMeta function=call/2 request_id=F15ro4erGdJN5ToAADlI tag=sent_reply

I got the error while trying the astarte-device-sdk-rust running the example individual_properties with the org.astarte-platform.rust.examples.individual-properties.DeviceProperties interface

Astartectl fails to parse datastream if value path is like /%{SensorSN}/%{OutID}/value

Json from appenine API:
{
"data": {
"sensorserial1": {
"output1": {
"value": {
"value": 12341,
"timestamp": "2019-11-29T14:27:03.456Z",
"reception_timestamp": "2019-11-29T14:27:03.456Z"
}
}
}
}
}

$ ❯❯❯ astartectl appengine devices data-snapshot grNRc5DGSmCjAZqQs12ipg
Using config file: /Users/___/.astartectl.yaml
Error: Unable to parse Datastream

The shorthand -p should be employed so as to point to a path where a PEM encoded public key is. The option is mentioned in astartectl examples:

Usage:
  astartectl housekeeping realms create <realm_name> [flags]

Examples:
  astartectl housekeeping realms create myrealm -p /path/to/public_key

It is not recognised, though:

Error: unknown shorthand flag: 'p' in -p

--realm-public-key may be used as workaround for the time being, but -p should be considered as a valid shorthand.

When creating a new realm using a context config, the protocol is forced to https, making the housekeeping call fail if https is not supported.

Steps to reproduce:

• create a http only deployment
• save the cluster config (eg: astartectl cluster instances get-cluster-config astarte)
• create a new realm with astartectl housekeeping realms create test

After #120 gets merged the filtering infrastructure will be there, so we can add this functionality that the dashboard already has

When brew is installing astartectl I get many errors:

"/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/syscall_darwin.1_13.go:25:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.1_13.go:27:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.1_13.go:40:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.go:28:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.go:43:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.go:59:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.go:75:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.go:90:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.go:105:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.go:121:3: //go:linkname must refer to declared function or variable

/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_darwin_amd64.go:121:3: too many errors"

This is happening on multiple machines. Is there an issue with the install file?

Thank you

Astarte already has implicit regex delimiters since 0.10 (see astarte-platform/astarte#37), so there's no need to add another pair of delimiters when creating JWT claims.

I am running astartectl 1.0.0-beta.2 with an astarte 1.0.0 cluster.
When viewing samples from a device on an object aggregated interface with astartectl appengine devices get-samples, only a single mapping gets printed, instead of all of them.
Moreover, if that mapping's value is missing in other samples to be shown, a null pointer error happens:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x98 pc=0x8f1f52]

goroutine 1 [running]:
github.com/jedib0t/go-pretty/table.isNumber(0x0, 0x0, 0x182fd5c)
	/home/runner/go/pkg/mod/github.com/jedib0t/[email protected]+incompatible/table/util.go:20 +0x42
github.com/jedib0t/go-pretty/table.(*Table).analyzeAndStringify(0xc0001b6000, 0xc0003a2280, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0xc00053b270, 0x4e7728, ...)
	/home/runner/go/pkg/mod/github.com/jedib0t/[email protected]+incompatible/table/table.go:309 +0x616
github.com/jedib0t/go-pretty/table.(*Table).initForRenderRowsStringify(0xc0001b6000, 0xc0004f4c00, 0x3, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc0003ac5b0, 0xc000314158, ...)
	/home/runner/go/pkg/mod/github.com/jedib0t/[email protected]+incompatible/table/table.go:624 +0xf6
github.com/jedib0t/go-pretty/table.(*Table).initForRenderRows(0xc0001b6000)
	/home/runner/go/pkg/mod/github.com/jedib0t/[email protected]+incompatible/table/table.go:596 +0xbb
github.com/jedib0t/go-pretty/table.(*Table).initForRender(0xc0001b6000)
	/home/runner/go/pkg/mod/github.com/jedib0t/[email protected]+incompatible/table/table.go:520 +0xa7
github.com/jedib0t/go-pretty/table.(*Table).Render(0xc0001b6000, 0x7f1d08752e98, 0x0)
	/home/runner/go/pkg/mod/github.com/jedib0t/[email protected]+incompatible/table/render.go:22 +0x40
github.com/astarte-platform/astartectl/cmd/appengine.renderOutput(0x1ad79a0, 0xc0001b6000, 0x1599780, 0xc0003a2320, 0x183341b, 0x7)
	/home/runner/work/astartectl/astartectl/cmd/appengine/device.go:1038 +0x1a5
github.com/astarte-platform/astartectl/cmd/appengine.devicesGetSamplesF(0x265a0c0, 0xc000214d80, 0x3, 0x9, 0x0, 0x0)
	/home/runner/work/astartectl/astartectl/cmd/appengine/device.go:780 +0x1a1a
github.com/spf13/cobra.(*Command).execute(0x265a0c0, 0xc000214cf0, 0x9, 0x9, 0x265a0c0, 0xc000214cf0)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0x2658680, 0x44276a, 0x25f4cc0, 0xc000074778)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
github.com/astarte-platform/astartectl/cmd.Execute()
	/home/runner/work/astartectl/astartectl/cmd/root.go:52 +0x31
main.main()
	/home/runner/work/astartectl/astartectl/main.go:20 +0x20

This relates to #598 (and possibly #574) on the astarte repo.

Rationale

Currently, astartectl requires a tedious process of setting URLs/configuration flags everytime, or a configuration file mechanism which is not widely used due to the tedious process of setting it up.

Solution

Switch to a kubectl-like context mechanism, which allows for multiple configuration switching at runtime and easy loading of different cluster/realm configuration.

Tasks

• Create a base structure for context-based configuration. Take something like https://github.com/fnproject/cli/tree/master/config as inspiration
  • Rather than having a huge YAML file such as kubectl, keep a clusters/ and contexts/ directory and single yaml files for each of them (Cluster == astarte cluster, Context == Astarte Cluster+Astarte Realm)
  • Create a YAML struct to ensure we can validate contexts easily and import/export them properly
• Add environment variables/config switches to allow replacing current context
  • --context, ASTARTE_CONTEXT to switch current context
  • --config-dir, ASTARTE_CONFIG_DIR to use custom configuration directory
• Port existing configuration to use the new structure
• astartectl housekeeping realm create should automatically create a context entry in astartectl config
• astartectl cluster deploy should automatically create a cluster entry in astartectl config
• astartectl cluster get-cluster-config, to fetch cluster configuration from the current kubectl context
• Allow for astartectl config import and astartectl config export to bundle configuration into a single YAML file to pass it around easily
• astartectl config {get,set,rename,delete}-{context,cluster}, astartectl config use-context, kubectl-style, to define default cluster and context for interaction

The current Long help does not explain the new realm creation mechanism (i.e. the one that automatically creates and saves the keys) and the example contains a flag (-p) that is not valid anymore (replaced by --realm-public-key).

The help implies that it should be possible to pass an interface to the astartectl appengine devices data-snapshot command to restrict the results to that specific interface.

However, doing so returns an empty result.

The ultimate goal is to replicate all interfaces installed on cluster on a different cluster.
This can be done in 2 steps: with a "download all interfaces" sub-command, and using the already existing sync sub-command.

astartectl supports a --to-curl option for commands such as publishing data on an Astarte interface.
However, the generated command suffers from several issues.

For instance, an astartectl command to set a property on a server-owned properties interface would look like this:

astartectl appengine devices send-data mydeviceid properties.interface.Name /interfacePath 3.3 --payload-type double -r test -t "$TOKEN" -u http://api.astarte.localhost

When adding the --to-curl option, astartectl outputs the following curl instead of directly setting the property:

curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" \
        -H "User-Agent: astarte-go" \
        -H "Authorization: Bearer $TOKEN" \
        "https://$ASTARTE_BASE_URL/appengine/v1/$REALM/devices/$DEVICE_ID/interfaces/$INTERFACE/$PATH
        --data '{"data" : $DATA}'

This has several issues:

1. The HTTP verb should be PUT, not POST
2. It assumes that some environment variables are populated, but if someone populates the PATH env variable then the terminal wouldn't be able to locate executables anymore and wouldn't recognize the astartectl binary. More specific variable could be used instead, such as INTERFACE_PATH
3. Using single quotes on '{"data" : $DATA}' means that the content is taken as-is and there is no substitution for $DATA. It should be "{\"data\" : $DATA}" instead.

Astarte dashboard needs a token with authorizations for Astarte Appengine. Astarte Channels and Astarte Realm Management. Currently it is not possible to create it using astartectl.

We could probably add astartectl utils gen-jwt dashboard to create such token, but we should also give users a way to create tokens with specific authorizations across multiple Astarte APIs.

Is there a way to delete a realm using astartectl?

The last release deb fails to install if the configuration directory is missing

user@host:~/Downloads$ sudo dpkg -i astartectl_22.11.00_linux_x86_64.deb 
(Reading database ... 217462 files and directories currently installed.)
Preparing to unpack astartectl_22.11.00_linux_x86_64.deb ...
Unpacking astartectl (22.11.0) over (22.11.0) ...
Setting up astartectl (22.11.0) ...
warn: Error while loading configuration: stat /root/.config/astarte: no such file or directory
warn: Error while loading configuration: stat /root/.config/astarte: no such file or directory

Creating the directory fixes the issuse

Areas which need improvement

• utils gen-jwt is completely missing documentation
• cluster should not show Astarte API-specific parameters as a suggestion

When this command it's executed

astartectl --appengine-url "https://api.your.astarte.com/appengine" -k test_private.pem -r test appengine devices data-snapshot

It crashes with

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7c9c35]

goroutine 1 [running]:
github.com/astarte-platform/astartectl/client.(*RealmManagementService).GetInterface(0x0, 0x7fff81f2cff6, 0x3, 0xc0005dc150, 0x25, 0x1, 0xc00012a000, 0x319, 0x0, 0x0, ...)
        /home/runner/work/astartectl/astartectl/client/realm_management.go:71 +0x65
github.com/astarte-platform/astartectl/cmd/appengine.devicesGetSamplesF(0x2200040, 0xc000158ab0, 0x3, 0x9, 0x0, 0x0)
        /home/runner/work/astartectl/astartectl/cmd/appengine/device.go:560 +0x1b66
github.com/spf13/cobra.(*Command).execute(0x2200040, 0xc00030f380, 0x9, 0xc, 0x2200040, 0xc00030f380)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0x21fe9c0, 0x43d54a, 0x219e8a0, 0xc000000180)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
github.com/astarte-platform/astartectl/cmd.Execute()
        /home/runner/work/astartectl/astartectl/cmd/root.go:51 +0x31
main.main()
        /home/runner/work/astartectl/astartectl/main.go:20 +0x20

astartectl should, instead, check that also --realm-management-url is present (or omit Realm Management checks) when --appengine-url is passed.

We should probably discuss about special-casing the localhost host so that astartectl can easily be used with the Docker Compose Astarte instance.

When astartectl fails to parse data from the server, the following error is returned

invalid character '<' looking for beginning of value

for example by using the wrong URL in the parameters

astartectl housekeeping realms list --housekeeping-url https://api.example.com/wrong/path

Returning a more user friendly message would help identifying the problem.

Allow util commands (eg: astartectl utils gen-jwt) to retrieve informations from the config context