strmprivacy / cli Goto Github PK

This is the STRM Privacy Command Line Interface, to define and manage your privacy streams, data schemas, event contracts and much more.

Home Page: https://strmprivacy.io

License: Apache License 2.0

Makefile 0.28% Shell 0.67% Go 98.71% Python 0.34%

cli data privacy data-privacy data-privacy-compliance data-processing data-pipeline

cli's People

Contributors

Stargazers

Watchers

Forkers

astronomous erdal-pb

cli's Issues

Flag completions don't always work

Describe the bug
Pressing after a flag which has a completion function, doesn't always work.

To Reproduce
Try: strm create kafka-exporter a_stream --cluster <tab>

Expected behavior
For the example, cluster names should be presented as tab completions.

Cause
Many completion functions contain the following logic:

if len(args) != 0 {
	// this one means you don't get two completion suggestions for one stream
	return nil, cobra.ShellCompDirectiveNoFileComp
}

Which causes the completion options not to appear.

Distribute the CLI through package managers

Error when refresh token is expired

When the refresh token is expired, the user is presented with the error:

strm auth login
Error: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_grant","error_description":"Offline user session not found"}

Refresh of credentials is never done

Remove person in descriptions

Often, descriptions in the CLI include a person, i.e. Do we want to export the keys stream. The subject we should be omitted, to make these descriptions only represent a fact. For example:

Do we want to export the keys stream -> Export encryption keys stream
Do we want to include all existing events -> Include all existing events in export

Create file based way to create resources

For example, a YAML file to create a stream, with the appropriate config.

Add the feature to list schemas

Automatically install shell completions?

@ariebovenberg shared an example for automatically installing shell completions with Homebrew. Let's update our Homebrew formula so that shell completions are installed automatically and we can remove the instructions from the caveats section.

If public API is not reachable, stacktrace is thrown

❯ strm auth login
Enter your Stream Machine portal email address:
Please enter your Stream Machine portal password:
Exception in thread "main" java.util.concurrent.ExecutionException: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 3 column 2 path $
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at com.github.kittinunf.fuel.core.requests.CancellableRequest.get(CancellableRequest.kt)
	at io.streammachine.api.cli.commands.Login.run(Authentication.kt:109)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:180)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:189)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:189)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:17)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:396)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:393)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:411)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:436)
	at io.streammachine.api.cli.StrmKt.main(Strm.kt:21)
Caused by: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 3 column 2 path $
	at com.google.gson.JsonParser.parseReader(JsonParser.java:66)
	at com.google.gson.JsonParser.parseString(JsonParser.java:47)
	at io.streammachine.api.cli.common.CommonKt.asPrettyJson(Common.kt:114)
	at io.streammachine.api.cli.commands.Login$run$1.invoke(Authentication.kt:105)
	at io.streammachine.api.cli.commands.Login$run$1.invoke(Authentication.kt:47)
	at com.github.kittinunf.fuel.core.DeserializableKt$response$2.invoke(Deserializable.kt:108)
	at com.github.kittinunf.fuel.core.DeserializableKt$response$2.invoke(Deserializable.kt)
	at com.github.kittinunf.fuel.core.DeserializableKt$response$asyncRequest$2$1.invoke(Deserializable.kt:217)
	at com.github.kittinunf.fuel.core.DeserializableKt$response$asyncRequest$2$1.invoke(Deserializable.kt)
	at com.github.kittinunf.fuel.core.RequestExecutionOptionsKt$sam$java_lang_Runnable$0.run(RequestExecutionOptions.kt)
	at com.github.kittinunf.fuel.core.DefaultEnvironment$callbackExecutor$1.execute(Environment.kt:16)
	at com.github.kittinunf.fuel.core.RequestExecutionOptions.callback(RequestExecutionOptions.kt:44)
	at com.github.kittinunf.fuel.core.DeserializableKt$response$asyncRequest$2.invoke(Deserializable.kt:216)
	at com.github.kittinunf.fuel.core.DeserializableKt$response$asyncRequest$2.invoke(Deserializable.kt)
	at com.github.kittinunf.fuel.core.requests.RequestTaskCallbacks.call(RequestTaskCallbacks.kt:30)
	at com.github.kittinunf.fuel.core.requests.RequestTaskCallbacks.call(RequestTaskCallbacks.kt:20)
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 3 column 2 path $
	at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1564)
	at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1405)
	at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:543)
	at com.google.gson.stream.JsonReader.peek(JsonReader.java:426)
	at com.google.gson.JsonParser.parseReader(JsonParser.java:61)
	... 21 more

Clarify fish completion install

Nice setup. A small thing I encountered. The instructions to install completions for the fish shell seem outdated. (source "strm/path.fish.inc")

There seems to be a strm completion fish command so it seems you already have something better in mind :)

strm sim run-random should give preference to options instead of saved-entities

Describe the bug
When I pass the --client-id and --client-secret options to the CLI command sim run-random, it first tries to find the saved entity with the stream-name. And only if it is not found, it uses the options. It should be the other way around and give preference to the options, and only use the saved entity if none are entered.

See: https://github.com/streammachineio/cli/blob/master/pkg/sim/randomsim/randomsim.go#L25

No error on missing properties when creating a derived stream

strm create stream derived-stream --derived-from banana -> missing consent level type and consent levels, however, no error is returned to the user.

Use zap log instead of logrus for logging

See https://github.com/uber-go/zap for more info. Much better performance!

keyExports

exposes functionality for sending encryption keys to buckets

Help text `strm config info` is wrong

strm context info
Error: accepts 1 arg(s), received 0
Usage:
strm context info [flags]

Flags:
-h, --help   help for info

Global Flags:
--api-auth-url string      User authentication host (default "https://accounts.streammachine.io")
--api-host string          API host and port (default "apis.streammachine.io:443")
--events-auth-url string   Event authentication host (default "https://auth.strm.services")
-o, --output string            Output format [json, json-raw, table, plain] (default "table")
--token-file string        Token file that contains an access token (default is $HOME/.config/stream-machine/strm-creds-<api-auth-url>.json)
--web-socket-url string    Websocket to receive events from (default "wss://out.strm.services/ws")

Error: accepts 1 arg(s), received 0

strm context info <saved-entity> is the correct help text

Reduce overused pointers and dereferences

While adding new features to the CLI, I've noticed that we overuse pointers. We don't need this many pointers. When should we use pointers?

While mutating an element passed as a parameter to a function
When you need to differentiate between nil and empty/default
When handling large structs, it might be useful

I propose to simplify the source code, by removing pointers where possible.

Information references:

Add command for printing strm context

For example, this could be an output:

Config directory: /path/to/config
Contents of strm config:

$ cat /path/to/config

Saved entities:
- Stream/my-first-stream
- KafkaUser/a-kafka-user
- etc

Integrate with Linear 3

Typos

strm get schema-code streammachine/demo/1.0.0

--output-file -- tar file name -> should be zip, but better to actually omit the file type.

Context info lacks required (entity)/(name) argument in Usage attribute

strm context info --help
Show the stored information for a saved entity

Usage:
strm context info [flags]

CLI should not autocomplete on files when not appropriate

strm create stream should not autocomplete on files

Change module path up to first slash to include a dot (Go convention)

See https://go.dev/ref/mod#go-mod-file-ident:

The leading path element (up to the first slash, if any), by convention a domain name, must contain only lower-case ASCII letters, ASCII digits, dots (., U+002E), and dashes (-, U+002D); it must contain at least one dot and cannot start with a dash.

permissions credentials file too loose

should be rw for the user only

Create file based way to create resources

For example, a YAML file to create a stream, with the appropriate config.

Extract clickstream schema stubs into separate repo

Auto generate documentation

See kubectl on how they generate docs:

Ensure tests work in CI

Change the way create event-contract works

As the create event-contract command is quite confusing (since it expects a json representation of the proto message for EventContract, without a wrapping eventContract), let's change the command to work as follows:

strm create event-contract "bla/example/1.3.0" --schema-ref "streammachine/demo/1.0.2" --public --definition-file /path/to/definition.json

--schema-ref and --definition-file are mandatory. --public is a bool flag to make the contract public or not. The definition.json is a JSON file (maybe also support YAML? Or the ability to read from stdin?) containing the following:

{
    "keyField": "consistentValue",
    "piiFields": {
        "consistentValue": 2,
        "someSensitiveValue": 3,
        "uniqueIdentifier": 1
    },
    "validations": [
        {
            "field": "consistentValue",
            "type": "regex",
            "value": "^.+$"
        }
    ]
}

This way, we split versioning logic and a link to the actual schema (which is metadata in this case), from the actual content (the definition).

Print formatting options

Let's print non-verbose info by default (e.g. when listing streams, only list the names and maybe whether they're a source/derived stream).
Flags instruct the CLI to print more, like:

--output table, prints all information as a table (see kubectl get pods as an example), where e.g. stream type (source / derived) is also shown as a column.
--output plain, prints only reference names as plain text
--output json-raw, prints all information in a raw unformatted JSON
--output json, prints all information pretty

--output short would be -o.
The default should be configurable through the config file. The default if no preference is given, then the fallback is table.

Simulator does not log errors

Describe the bug
When I run strm simulate random-events with an invalid "events-api-url" (for example, pointing it to localhost), I don't see an error. This is very confusing.

Integrate to Jira

https://media1.giphy.com/media/iCeY8KoAQxYsLG453e/giphy.gif?cid=ecf05e471dhlp8tahabaw7uc3qv0yud20er777b1g7drca1g&rid=giphy.gif&ct=g

Not all autocompletes seem to work

For example on strm create stream derived-stream --derived-from banana --levels 0 --consent-type CUMULATIVE, the --consent-type autocomplete doesn't work.

Use Wire as a replacement for global mutable shared state (e.g. common.BillingId)

At the moment, we have a global, mutable, shared variable for the BillingId. The initialization process of setting this variable is not transparant.
This is where dependency injection could come to play, which Golang offers (officially built): https://github.com/google/wire

Let's explore if Wire could help us remove the current way common.BillingId works.