bitspark / slang Goto Github PK

Background
Right now, we don't know which builtin operators are really necessary or how we should name them. Maybe we need to split them up in the future or give them a different behavior. However, until then these builtin operator will be baked into many many other operators so we cannot simply change their names or behavior.

Proposed solution
Register the builtin operators under a slib-like name such as slib.loop instead of just loop. This would allow us to implement the loop operator differently by simply removing the builtin operator entirely and write a slib operator with the same signature. No operator using slib.loop would behave differently because it will simply use the slib operator instead.

It would seemlessly work with our current implementation because we always check if there exists a builtin operator with that name (regardless whether it contains a . dot or not).

• loop -> slib.loop
• merge -> slib.merge
• fork -> slib.fork
• agg -> slib.agg
• reduce -> slib.reduce
• const -> slib.const
• eval -> slib.eval

After #4 we need a persistent storage of objects. This is the goal of this issue.

Connect the DB layer to a MongoDB server. Also provide the means to configure the connection in a separate JSON file (e.g. database.json).

The reason why we want a different JSON file is that we need a separation of logic and system configuration. Everything contained in the program_xyz.json file should be completely independent from the environment and potentially run everywhere. That is clearly not the case for database settings.

currently generics are just map of PortDefs. This works for us but a specific typo Generics would be fine

Don't just push infinitely many items to out, but rather think of implementing a pull-based approach. Only when an operator pulls the operator should serve an element.

Wait is just a passer which delays redirecting.
It has two inputs (time and item) and one output.

• Use InstanceDef per TestCase
• Allow for specifying delegate data input and output per delegate
• Make it possible to test builtins through TestCases this way

Investigate & fix

There might occur problems when a trigger is listening on a map or a stream if only sub or subs is connected. In this case the trigger will never fire.

Possible solution

1. Map
   When connecting a map with a trigger you could either ask the user to connect with a map sub port instead or arbitrarily descend and do it automatically.

2. Stream
   This one is more difficult. One solution would be to remember the source port and descend. When pulling from this port, it would suffice if an appropriate marker (ideally BOS) appears.

#116 <--

Currently there is no way to specify another working directory in slang. That means, requesting or loading custom OperatorDefs will only works from slang-deamons working directory.
We need the possibility for specifying different working directories within environments, from which operators can be loaded.

I might suggest following changes:

• Introduce environment struct (SlangEnv, Env, Environment, ->Environ<-) to package "api"

package "api"
type Environ struct {
    paths []string // paths[0] is always working dir
}

• Make all api functions methods of Environ which need a correct working-dir
• Enjoy the possibilities

Until now, Slang applications are tiny and there is no need for heavy debugging or performance optimization tools.

However, in the future we might want to have

• heatmaps which connections transport how much data
• how long which operator takes to calculate things
• live obervations of a Slang applications allowing to watch how data is being processed

Extend the Port structure with two methods for pushing and pulling complex objects.

• move builtin_function_test.go into slang/builtin/
• Rename builtin.MakeOperator to builtin.NewOperator
• Don't use builtin.MakeOperator within builtin operator tests
• Rename constructor functions beginning with Make... into New...
• Move all ...Def structs in separate file slang/op/def.go
• Extend Assertions by Assertions.PortPushes
• Rename ParseConnection into ParsePort
• Move functions Parse...Def from package slang/op into slang
• Rename slang/op into slang/core

@jm9e

Implement URL mapping of URLs to operators. This is needed to create an API server or a simple web server. Take the request method into consideration.

This issue includes configuration (ideally part of the program JSON configuration file) and implementation. Implementation includes starting an HTTP server and redirecting requests to the correct operators / returning response back to the client following the mapping configuration.

Mapping should allow for adding placeholders which produce values that are inserted into the according operators (e.g. /findPerson/:firstName/:lastName will produce firstName and lastName as stream outputs that can be connected to operator input ports).

Important: Do not forget that we want to be able to use foreign operators inside other operators. HTTP mapping must be an addition, not a replacement for automatic API generation.

Also think about a similar mapping for WebSockets (implementation not part of this issue).

readLibPathsFromEnvironment adds slangd's workingdir. Since slangd will run as a server, that workingdir is not working for us anymore

• Have a look at govaluate tokens

... to allow for propagation of properties through operators.

It would also allow for specifying constant values by wrapping an oberator together with several const operatorts which take their constant value from the properties map (currying).

Implement automatic generation of a Port from a JSON schema.

Names in map ports cannot be changed easily one they are being used.
Therefore we should choose good names and put more thought into them.

Implement rudimentary database features and provide an API for them. The API shall be later used within object and class operators.

It should provide

• JSON schema management and validation
• class and object management

After implementation one should be able to

• use the DB layer from class and object operators
• store and retrieve objects
• update objects
• delete objects

Later, it should be connected to a MongoDB server. For now, a temporary RAM solution is enough to close this issue.

Trivial case:

{
  "name": "test",
  "in": {"type":"number"},
  "out": {"type":"number"},
  "operators": {},
  "connections": {
    "test.in": ["test.out"]
  }
}

Invalid cases:

• missing or empty "in"
• missing or empty "out"
• missing or empty "connections"

More complex example:

{
 "name": "test2",
 "in": {"type":"map", "map":{"a": {"type": "number"}, "b": {"type": "number"}}},
 "out": {"type":"number"},
 "operators": {
   "add1": {
     "type": "function",
     "properties": {
       "function": "x+y"
     }
   }
 },
 "connections": {
   "test2.in[a]": ["add1.in[x]"],
   "test2.in[b]": ["add1.in[y]"],
   "add1.out": ["test2.out"]
 }
}

• Move tests out of tests package
• tests package becomes a package for test utilities (old tests/assertions package which can be removes)
• Revise all tests
• Check test coverage
• Think about a concise and consistent naming schema for test
  • When _ as separator?
  • When __ as separator?
  • What prefixes to use, what suffixes, general phrasing, ...
• Maybe even write a Wiki page about Slang testing conventions
• Add separate subdirectories for operator tests

• Add test coverage to CI pipeline
• Allow for browsing an HTML page with coverage visualization for every build
• Require a minimum coverage of X % (TBD) to accept a pull request

• don't use global path

Think about a basic set of operators needed to construct arbitrary programs. Also think about operators which can push and pop streams from the stream stack (i.e. start new streams or "gulp" streams / generators and aggregators).

Implement these basic operators and test them thoroughly.

See #82 and #83

Proposal for new operator definitions

in:
  type: number
out:
  type: number
delegates:
  delegate1:
    in:
      type: number
    out:
      type: number
properties:
  - prop1

operators:
  op1:
    operator: opName1
  op2:
    operator: opName2
    properties:
      prop: $prop1

connections:
  (:
    - (op1
  op1):
    - (delegate1
  delegate1):
    - (op2
  op2):
    - )

Changes

• Add delegates for wiring out delegate ports
• Add properties for allowing to specify properties when instancing
• Allow usage of property values prefixed with $ to indicate usage of property variables specified in properties

... such as

• Streamify()
• PrimitivePortDef
• NumberPortDef
• ...

Some operators have to delegate data processing to another operator / series of operators (aka pipeline).

Examples include reduce, loop, agg, take, ...

Currently, in and out ports are used for that purpose. But conceptionally this doesn't seem right. in ports should be regarded as ports which take one logical element and output another logical element emerging from it. This is currently not the case with loop and others.

Instead, introduce delegates. Each delegate belongs to an operator and consists of an in and an out port.

Example: loop

Think of a loop instance looper. In the following, there is a comparison between the ports (descended to the maximum) now and then.

Now:

1. init(looper
2. iteration.state(looper
3. iteration.continue(looper
4. looper)end
5. looper)state

Then:

1. (looper
2. state(looper.iterator
3. continue(looper.iterator
4. looper)
5. looper.iterator)

Regrouped by delegates:

• (looper
• looper)
• state(looper.iterator
• continue(looper.iterator
• looper.iterator)

Another big advantage would be that many pointless port names are rendered unnecessary this way (here: iteration, init, end).

Also allow for testing of generic operators.

Name Delegate.op same as Port.operator

Change in and out to generic types in sllib/

• File system
  • read file
  • read directory
• Parsing
  • csv, json, xml, yaml, ...
• Compression
  • zip, bz2, gzip, ...
• Cryptography
• Network
  • TCP, UDP
  • HTTP server & client
  • Send email
  • DNS lookup
• Math
  • sum, avg, min, max, ...
• DS/Algos
  • sort
  • find
• OS
  • exec
  • environment
• Date & Time
  • now

Think about a migration concept. Migrations could probably be implemented as "disposable operators" that are executed once in the migration train, just like ordinary migrations known from Rails, Django, ...

Make Port of type="Any" allow passing items of any streams, maps or primitives.

Currently type="Any" just means a primitive of any datatype

• detection of duplicate structures and regrouping them into operators
• detection of separable structures and grouping them automatically

[ ] allow for arbitrary in ports
[ ] rename to eval