Giter VIP home page Giter VIP logo

zen's Introduction

Zen meta store & schema

Clojars Project

zen-docs

Motivation

Often application can be split into two parts declarative models (DSLs) and imperative interpreter (engines).

Models can be expressed with data - Data DSLs. Why? Because data is easily composable, regular and introspectable.

There are a lot of successful Data DSLs in clojure ecosystem. But often they do not play together :(.

Zen is a framework to unify Data DSLs, make them composable and take model driven design to the next level.

Model Project

Zen separate models source code from interpreters code, but keep it in source path using same layout as clojure.

Models are grouped into namespaces and namespaces are stored in edn files.

Namespaces can be published as reusable packages.

Namespace

Each namespace contains one or multiple models described with data.

Namespace is a map (in terms of edn) with two special symbol keys - 'ns and 'import

  • ns - defines name of namespace
  • imports - imported namespaces

Namespaces may refer other namespaces by imports. Core zen namespace is imported implicitly.

That's how starting from one entry point namespace, your project can import only used modules and models from other packages.

Rest of symbolic keys in namespace define models.

Just like in clojure namespace you may refer one model from another located in one namespace by short name (symbol) and refer between namespaces by full name - (namespace.name/symbol)

Example namespace:

{ns myapp.module ;; namespace name
 imports #{http pg model auth} ;; imports - TODO: think about aliases
 
 db
 {:zen/tags #{pg/config}
  :connection {:host "..." :port "..."}
  :storages #{model/patient-store}}

 ;; model
 web 
 {:zen/tags #{http/server} ;; tags set
  :port 8080
  :workers 8
  :db db
  :apis #{api http/admin-api}} 
 
 api 
 {:zen/tags #{http/api}
  :zen/desc "API definition"
  :middleware [{:type http/cors :allow #{"https://myapp.io"}}
               {:type auth/authorize}]
  :routes {
    "Patient" {:post {:op create-pt}
               :get  {:op search-pt}}
    "meta" {:get {:op http/meta}}}}

 create-pt 
 {:zen/tags #{http/create}
  :operation http/create
  :schemas #{model/patient}
  :response {
    201 {:confirms #{model/patient}}
    422 {:confirms #{http/error}}}}

 search-pt 
 {:zen/tags #{http/search}
  :operation http/search
  :store #{model/patient-store}
  :params {:query {:name {:zen/desc "Search by name" 
                          :engine http/fhir-search-param
                          :type "string" :expression "Patient.name"}
                   :ilike {:engine http/sql-search-param
                           :query "resource::text ilike {{param.value}}"}}}
  :response { 
    201 {:confirms #{model/search-bundle}} 
    422 {:confirms #{model/search-errors}}}}}

Tags

Instead of introducing any kind of types and type hierarchies, zen uses tag system to classify models.

You may think about tag system as non-hierarchical multidimensional classification (just like java interfaces). Or as a function of meta store - get all models labeled with specific tag.

Store

Model project may be loaded into store. You start loading from entry point namespace. All imports will be resolved, validated and loaded into store.

Store functions:

  • get-symbol ns/sym
  • get-tag tag/name
  • read-ns my/ns

Envs

Some data can be moved from edn to envs using special readers:

  • #env - read string
  • #env-integer - read integer
  • #env-symbol - read symbol
  • #env-keyword - read keyword
  • #env-number - read number
  • #env-boolean - read boolean

Before reading with envs - zen context should be initialized with envs: (zen.core/new-context {:env {:ENV_NAME "env-value"}})) or system env variables should be set - export MY_VAR='value'.

{ns test-env

 schema
 {:zen/tags #{zen/schema zen/tag}
  :type zen/map
  :keys {:string {:type zen/string}
         :int    {:type zen/integer}
         :sym    {:type zen/symbol}
         :key    {:type zen/keyword}
         :num    {:type zen/number}
         :bool   {:type zen/boolean}}}

 model
 {:zen/tags #{schema }
  :string #env ESTR
  :int    #env-integer EINT
  :sym    #env-symbol ESYM
  :key    #env-keyword EKEY
  :num    #env-number ENUM}
  :bool   #env-boolean BOOL
  ;; or with defaults
  :string #env [ESTR "Default"]
  }

Schema

Zen has built-in schema engine deeply integrated with meta-store.

The key features of zen/schema is:

  • open world design - i.e. each schema validates only what it knows
  • support of RDF inspired property schema - i.e. schema attached to key (only namespaced keys) - not to a key container
{ns myapp

 Contact 
 {:zen/tags #{zen/schema}
   :type zen/map
   :keys 
   {:system {:type zen/string 
             :enum [{:value "phone"} 
                    {:value "email"}]
     :value  {:type zen/string}}}

 Contactable 
 {:zen/tags #{zen/schema}
   :type zen/map
   :keys {:contacts {:type zen/vector 
                     :every {:type map :confirms #{Contact}}}}}

 User 
 {:zen/tags #{zen/schema}
   :type zen/map
   :confirms #{Contactable}
   :require #{:id :email}
   :keys {
     :id {:type zen/string}
     :email {:type zen/string :regex ".*@.*"}
     :password {:type zen/string }}}

 ;; example of property schema
 
 human-name 
 {:zen/tags #{zen/property zen/schema}
   :type zen/map
   :keys {:family {:type zen/string} 
          :given {:type zen/vector :every {:type zen/string}}}}}

 instance
 {:id "niquola"
  :myapp/human-name {:given ["Nikolai"] :family "Ryzhikov"}
  :password "secret"}

Schema Specification

Schema statements are described with maps. Map may have a :type key, which defines how this map is interpreted. For example :type zen/string will check for string, zen/map describes map validation.

Here is list of built-in types:

  • primitives
    • zen/symbol
    • zen/keyword
    • zen/string
    • zen/number
    • zen/integer
    • zen/boolean
    • zen/date
    • zen/datetime
  • collections
    • zen/vector
    • zen/set
    • zen/map
    • zen/list
  • zen/case

You can get all schema types by query meta-store for zen/type tag. User can extend schema with new types - TBD

All schema maps may have common a keys:

  • :confirms - set of other schemas to confirm (this is not inheritance!)
  • :enum - polymorphic enumeration of possible values (TODO: think about terminology - reference semantic?)
  • :constant - polymorphic fixed value validation
  • :valuesets - like enum, but using valueset zen protocol (TBD)

Depending on type schema map may have type specific keys. For example :minLength and :regex for zen/string or :keys for zen/map.

zen/case

zen/case is alternative to union type, it is more advanced and may be applied to different maps

path
{:zen/tags #{'zen/schema}
 :type 'zen/vector
 :every
 {:type 'zen/case
  :case [{:when {:type 'zen/string}}
         {:when {:type 'zen/map}
          :then {:type 'zen/map 
                 :require #{:name} 
                 :keys {:name {:type 'zen/string}}}}]}}

zen/symbol

  • :tags - set of symbols - check symbols refers models with

zen/map

For example zen/map type defines following validation keys:

  • :values schema - schema to apply to all values
  • :keys { key: schema } - enumeration of keys and schema for each key
  • :require #{:key,...} - list of required keys in map
  • :schema-key {:key :some-key } - key to resolve schema from data on fly

zen/vector

Apply clojure.spec regular expressions for collections!!!!

  • :every schema - apply schema to every element in collection
  • :nth {integer: schema} - apply schema to nth element
  • :minItems & :maxItems - min/max items in collection
  • :filter - TODO: apply filter to collection, then apply schema to

zen's People

Contributors

aitem avatar apricotlace avatar carbon-hvze avatar ddegr avatar genarazmakhnin avatar ir4y avatar islambegk avatar ivannikovg avatar kenichsberg avatar kgoh avatar krevedkokun avatar mput avatar niquola avatar octoshikari avatar panthevm avatar qdzo avatar rublag avatar spicyfalafel avatar tentoshka avatar veschin avatar vganshin avatar xamelon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zen's Issues

Review & refactor zen errors

Current state:

  • some errors are difficult to understand what they mean and to what they relate to
  • errors are not strictly categorized

Desired state:

  • as a developer I want to see human readable error description, it's location and understand how to fix it
  • as a developer I want to be able to differentiate between types of errors (i.e. data validation, schema validation, namespace reading, etc)
  • as a user I want to see errors with my language localization

TODO:

  • list all possible errors produced by zen
  • design solution
  • decide on categories

zen package archive

Introduce package archive (aka jar) to distribute zen modules efficiently.
zen archive (zar) is zip archive with an index (metadata) entry and namespaces as other entries
encoded by nippy or EDN?

  • compare nippy and EDN performance
  • index contains deps and a list of namespaces (or use entry for that)

Introduce :zen/type

Users are not good with :zen/tags - we can introduce :zen/type to simplify relationships.
:zen/type should be searchable by zen-tags (probably materialized into zen-tags)

Support zen/op for zen/system

start/stop protocol is good for stateful components of the systems, but there a lot of only init functions (like seeds etc) and this functions is zen/op too.

In system start - analyze the engine of the component - if it is zen/op - just call it on start, passing config as :params.

Add ability to create zen types with zen

I want to define DSL which contains entity definition (zen/schema) and custom metadata. And it would be nice to have some feature - that helps avoid redundancy

Example:

MyDoc
{:zen/tags #{zen/schema zen/tag}
 :type zen/map
 :keys {:a {:control input/text
                 :text "My new input"
                 :type zen/string}}}

Here I define entity and form which gather that entity, but I need to specify both :control 'input/text' and :type 'zen/string'. I want to find a way to make that input/text define type of value - zen/string - and don't need to repeat it several times.

As I know it's possible by creating new :type - but this option requires to create validation functions in clojure. and also implement every constraint for the type - which can be error prone.

It's would be very usefull to have some zen constructions to combine such aspects of data.

urgency: high

Error when using regex in zen/string

Steps to reproduce:

  1. Create namespace somens.edn:
{ns somens

  some-tag {:zen/tags #{zen/schema}
          :type     zen/string
          :regex   #".*"}
  1. Load this namespace to the meta store:
(ns core
  (:require [zen.core :as zen])

(def ctx (zen/new-context))

(zen/read-ns ctx 'somens)
  1. Observe this error:
Unhandled clojure.lang.ExceptionInfo
   Regex not allowed. Use the `:regex` option
   {:type :edamame/error, :row 5, :col 23}

Zen schemas sourcing and dependency loading

Currently I have to solve a problem of distributing my zen schemas. It would be good if zen offered tools or guidelines.

Use-case:

  • I created a set of zen namespaces, I need to deliver it to my working environments
  • I implemented a model defined in zen and now want others to supply configs using my model. My model is a dependency for their configs and must be supplied with their configs for them to be valid. Currently I have to share a copy of the my namespace file and share it to my users
  • I developed a library with, let's say json-rpc engine. I want others to use my library and write own json-rpcs using my json-rpc tag. Currently they have to copy my namespace with tag and put into their zrc
  • Zen delivery is done at the configuration time, but I have to implement zen schemas delivery in my project. Zen could do that and I could use zen to achieve this

Urgency: medium
Currently I already implemented solutions for all of these problems in one of my projects, but in my upcoming project I will have to solve all of this problems again

No matching method valueOf found taking 1 args exception.

Encountered this error. Looks like there are some stale or broken code

No matching method valueOf found taking 1 args java.lang.IllegalArgumentException: No matching method valueOf found taking 1 args
 at clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:154)
    clojure.lang.Reflector.invokeStaticMethod (Reflector.java:332)
    zen.v2_validation$fn__8257$fn__8258.invoke (v2_validation.clj:393)
    zen.v2_validation$compile_schema$compiled_sch__8136.invoke (v2_validation.clj:162)
    zen.v2_validation$get_cached$fn__8146.invoke (v2_validation.clj:182)
    zen.v2_validation$fn__8309$keys_sch__8316.invoke (v2_validation.clj:514)
    zen.v2_validation$compile_schema$compiled_sch__8136.invoke (v2_validation.clj:162)
    zen.v2_validation$fn__8309$keys_sch__8316.invoke (v2_validation.clj:514)
    zen.v2_validation$compile_schema$compiled_sch__8136.invoke (v2_validation.clj:162)
    zen.v2_validation$fn__8335$fn__8339$err_fn__8341.invoke (v2_validation.clj:547)
    clojure.lang.ArrayChunk.reduce (ArrayChunk.java:58)
    clojure.core.protocols$fn__8244.invokeStatic (protocols.clj:136)
    clojure.core.protocols/fn (protocols.clj:124)
    clojure.core.protocols$fn__8204$G__8199__8213.invoke (protocols.clj:19)
    clojure.core.protocols$seq_reduce.invokeStatic (protocols.clj:31)
    clojure.core.protocols$fn__8236.invokeStatic (protocols.clj:75)
    clojure.core.protocols/fn (protocols.clj:75)
    clojure.core.protocols$fn__8178$G__8173__8191.invoke (protocols.clj:13)
    clojure.core$reduce.invokeStatic (core.clj:6886)
    clojure.core$reduce.invoke (core.clj:6868)
    zen.v2_validation$fn__8335$fn__8339.invoke (v2_validation.clj:561)
    zen.v2_validation$compile_schema$compiled_sch__8136.invoke (v2_validation.clj:162)
    zen.v2_validation$fn__8309$keys_sch__8316.invoke (v2_validation.clj:514)
    zen.v2_validation$compile_schema$compiled_sch__8136.invoke (v2_validation.clj:162)
    zen.v2_validation$get_cached$fn__8146.invoke (v2_validation.clj:182)
    zen.v2_validation$fn__8381$fn__8383.invoke (v2_validation.clj:674)
    zen.v2_validation$compile_schema$compiled_sch__8136.invoke (v2_validation.clj:162)
    zen.v2_validation$get_cached$fn__8146.invoke (v2_validation.clj:182)
    zen.v2_validation$fn__8362$confirms_sch__8366.invoke (v2_validation.clj:618)
    zen.v2_validation$compile_schema$compiled_sch__8136.invoke (v2_validation.clj:162)
    zen.v2_validation$fn__8309$keys_sch__8316.invoke (v2_validation.clj:514)
    zen.v2_validation$compile_schema$compiled_sch__8136.invoke (v2_validation.clj:162)
    zen.v2_validation$get_cached$fn__8146.invoke (v2_validation.clj:182)
    zen.v2_validation$_STAR_validate_schema.invokeStatic (v2_validation.clj:219)
    zen.v2_validation$_STAR_validate_schema.invoke (v2_validation.clj:209)
    zen.v2_validation$validate_schema.invokeStatic (v2_validation.clj:228)
    zen.v2_validation$validate_schema.doInvoke (v2_validation.clj:221)
    ```

Circular dependency tag ns -> tagged ns -> tag ns doesn't return errors of tagged ns when tag ns is read

Steps to reproduce. After reading 'a ns which imports 'b, 'b/s is doesn't return errors until 'b ns is explicitly read

  (def memory-store
    {'b '{:ns b
          :import #{a}
          s {:zen/tags #{a/t}}}

     'a '{:ns a
          :import #{b}
          t {:zen/tags #{zen/tag zen/schema}
             :type zen/map
             :require #{:foo}}}})

  (def z (zen.core/new-context {:memory-store memory-store}))

  (zen.core/load-ns z (get memory-store 'a))
  ;; => [:resources-loaded 1]

  (zen.core/errors z)
  ;; => []

  (keys (:ns @z))
  ;; => (zen a b)
  (remove #(clojure.string/starts-with? % "zen") (keys (:symbols @z)))
  ;; => (a/t b/s)

  (def z-before-b-load @z)

  (zen.core/load-ns z (get memory-store 'b))
  ;; => [:resources-loaded 1]

  (zen.core/errors z)
  ;; => [{:type "require", :message ":foo is required", :path [:foo], :schema [a/t :require], :resource b/s}]

  (keys (:ns @z))
  ;; => (zen a b)
  (remove #(clojure.string/starts-with? % "zen") (keys (:symbols @z)))
  ;; => (a/t b/s)

  (= (select-keys z-before-b-load [:ns :symbols :tags])
     (select-keys @z [:ns :symbols :tags]))
  ;; => true

  #_(butlast (clojure.data/diff z-before-b-load @z))

Exception on validating array instead of map

Got an exception while validating such data:

 {:serviceProvider
  [{:id "8cd75b6e", :resourceType "Organization"}]}]

with this schema
https://github.com/zen-fhir/hl7-fhir-r4-core/blob/4e8244048fe4327e1d6675b2ccbc50973dc9acb9/zrc/hl7-fhir-r4-core/Encounter.edn#L80

java.lang.IllegalArgumentException: Key must be integer
 at clojure.lang.APersistentVector.invoke (APersistentVector.java:297)
    clojure.core$some.invokeStatic (core.clj:2718)
    clojure.core$some.invoke (core.clj:2709)
    zen.misc$fn__8610$fn__8612.invoke (misc.clj:12)
    zen.v2_validation$compile_schema$compiled_sch__8151.invoke (v2_validation.clj:162)
    zen.v2_validation$fn__8318$keys_sch__8325.invoke (v2_validation.clj:508)
    zen.v2_validation$compile_schema$compiled_sch__8151.invoke (v2_validation.clj:162)
    zen.v2_validation$get_cached$fn__8161.invoke (v2_validation.clj:182)
    zen.v2_validation$_STAR_validate_schema.invokeStatic (v2_validation.clj:219)
    zen.v2_validation$_STAR_validate_schema.invoke (v2_validation.clj:209)
    zen.v2_validation$validate.invokeStatic (v2_validation.clj:247)
    zen.v2_validation$validate.doInvoke (v2_validation.clj:231)

Mutually recursive schema-key definitions lead to StackOverflowError

Mutually recursive schema-key definitions lead to StackOverflowError.
Example:

(do                                                                                                                                                                                          
  (require '[zen.core :as zen])                                                                                                                                                              
  (def ztx (zen/new-context))                                                                                                                                                                
  (def mns                                                                                                                                                                                   
    '{ns mns                                                                                                                                                                                 
                                                                                                                                                                                             
      a {:zen/tags #{zen/schema zen/tag}                                                                                                                                                     
         :schema-key {:key :a}                                                                                                                                                               
         :type zen/map                                                                                                                                                                       
         :keys {:a {:type zen/symbol}}}                                                                                                                                                      
                                                                                                                                                                                             
      b {:zen/tags #{zen/schema zen/tag}                                                                                                                                                     
         :schema-key {:key :b}                                                                                                                                                               
         :type zen/map                                                                                                                                                                       
         :keys {:b {:type zen/symbol}}}                                                                                                                                                      
                                                                                                                                                                                             
      c {:zen/tags #{a}                                                                                                                                                                      
         :a b                                                                                                                                                                                
         :b a}})                                                                                                                                                                             
  (zen/load-ns ztx mns))

Zen *key words* should be keywords instead of symbols

There are few reserved symbols in zen namespaces: ns, import and alias
Because of this user can not define own schemas with these names. Frequent use-case looks like this:

{ns my-operations
 
 export {:zen/tags #{my-op}}
 import {:zen/tags #{my-op}}}

Zen will return error that a #{} expected as a value of import since it is used to resolve namespace deps.

Probably ns, import and alias should be keywords like :zen/tags, and user definitions would be any symbol e.g.:

{:zen/ns my-operations
 :zen/import #{my-deps}
 
 export {:zen/tags #{my-op}}
 import {:zen/tags #{my-op}}}

Urgency: low
This is a minor inconvenience for me, but it looks like a language design flaw

Zen modules dependency aliases

After #27 will be implemented there will be increasing chance of namespace collisions.

Problem example:

  • package1: defines namespaces A, B. A depends on B here
    A.edn
    {ns A
     import #{B}
     sym {:zen/tags #{B/tag}}}
  • package2 defines namespaces B and C. C depends on B here
    C.edn
    {ns A
     import #{B}
     sym {:zen/tags #{B/tag}}}
  • I'm interested in package1.A/sym namespace and in package2.C/sym
  • I import both packages as my dependency.
  • Namespaces collision by the name B will break one of these packages

Possible solutions:

  • suggest user to copy and patch names in the lib with name clash manually
  • add local custom package prefixes. When specifying a dependency could/should specify also a prefix which will prepend ALL of the symbols in this dependency

Urgency: low
This may be important architecture decision, i.e. we may want to decide to force all deps to have local alias

Add support of :engine or :class as an alternative to :zen/tags

The tag's system is flexible, but sometimes it is looks too complicated and simple :class or :engine pattern may work:

{ns myapp
 db {:engine pg/db :user "..."}
 db {:zen/tag #{pg/db} :engine pg/db ...}}

:engine or :zen/engine may be indexed as a tag - i.e. (get-tag ztx 'pg/db) will find it

Exclusive keys error

Steps to reproduce:
  1. Create namespace some-ns.edn:
{ns some-ns

 some-tag {:zen/tags       #{zen/schema}
           :type           zen/map
           :exclusive-keys #{:key-1 :key-2}
           :keys           {:key-1 {:type zen/string}
                            :key-2 {:type zen/string}}}}
  1. Load this namespace to the meta store:
(ns core
  (:require [zen.core :as zen])

(def ctx (zen/new-context))

(zen/read-ns ctx 'some-ns)
  1. Print errors from the meta store
(clojure.pprint/pprint (:errors @ctx))
[{:message "Expected type of 'set, got keyword",
  :type "type",
  :path [:exclusive-keys 0],
  :schema [zen/schema :schema-key zen/map :exclusive-keys :every],
  :resource some-ns/some-tag}
 {:message "Expected type of 'set, got keyword",
  :type "type",
  :path [:exclusive-keys 1],
  :schema [zen/schema :schema-key zen/map :exclusive-keys :every],
  :resource some-ns/some-tag}]

`zen/schema` traversing

Validation implements zen/schema traversing. There are more applications that need to traverse zen/schema besides validation

They can be split in 3 categories. With data and without.

With data -- use cases alternative to validation. We need to compile schema into a function and apply compiled function to a data:

  • Semantic diff (data is some DSL expression, compiled schema is DSL definition)
  • Breaking changes analysis
  • Generate :text property for some resource

We also may want to transform data:

  • Enriching data by setting default values
  • Fixing data by type coercing (e.g.: replace arrays with sets if schema says so)

Without data -- we need to compile the whole schema into something else:

  • Generating type/object schema for programming languages (e.g.: typescript)
  • Generating snapshot schema (super-schema, with symbol definitions inlined in-place of symbols)
  • Generating zen/schema viewer & generative documentation
  • Generative test data based on a schema

Some may be done in either way:

If we want to implement any of above, then we need to reimplement all traversing for all zen/schema syntax available and then keep updating this implementation with support of new features that may be introduced into zen/schema that may alter traversing.

I suggest that we need to come up with separate namespace with sole purpose of traversing, which then will be used in zen.v2-validation and in other features we want to implement.

Current zen.walk just uses validation

Severity: high (we need to describe transformations)

Add fixed-point precision (or format) validation to zen/number

It is suitable to have constraints on decimal numbers like:

  • precision (number of digits in the value)
  • scale (number of digits after comma)

Sources:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/toPrecision
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/toFixed

Or maybe add format like in "##.#"

https://docs.oracle.com/javase/7/docs/api/java/text/DecimalFormat.html

Add ability to suppress symbol resolution

Problem:

Sometimes it's usefull to place some 'code like' DSL inside zen definitions.

Example

{:zen/tags #{extract}
:extract {:patient-name (get-in [:name :given 0])
               :user-id (dsl/ctx :user)}}

But every symbol needs to be resolved and zen will fail in loading ns with sutch definitions.

Potentially we can get around this problem by writing empty symbol definitions,

But in the long run it's not an good solution.

Zen processing tools

There's often a need to process similar zen expressions, like resolving :confirms or traversing a zen definition to do symbols definitions lookup. These procedures are repeatedly implemented in-place from scratch, this results in a lot of boiler-plate during work with zen processing.

Typical use-case:

  • Processing a zen schema for a some kind of structure description
  • Processing user definition which confirms to one of your tags when this tag requires a symbol to be tagged with another of your tags
  • Also check zen-lsp for use cases

It would be good if zen could offer something else besides get-symbol and get-tag. Maybe get-snapshot returning a definition with all symbols and :confirms included

Urgency: medium
This is an inconvenience. I can keep reimplementing and testing ad-hoc :confirms lookups and other zen processing stuff

NullPointerException in zen.v2_validation/get-cached

It looks like there are some concurrency issue with schema compilaiton.

In one of the test we encounter such issue when calling zen.v2_validation/validate.

#error {
 :cause Cannot invoke "clojure.lang.IFn.invoke(Object, Object, Object)" because "v" is null
 :via
 [{:type java.lang.NullPointerException
   :message Cannot invoke "clojure.lang.IFn.invoke(Object, Object, Object)" because "v" is null
   :at [zen.v2_validation$get_cached$fn__429 invoke v2_validation.clj 166]}]
 :trace
 [[zen.v2_validation$get_cached$fn__429 invoke v2_validation.clj 166]
  [zen.v2_validation$_STAR_validate_schema invokeStatic v2_validation.clj 204]
  [zen.v2_validation$_STAR_validate_schema invoke v2_validation.clj 194]
  [zen.v2_validation$validate invokeStatic v2_validation.clj 232]
  [zen.v2_validation$validate doInvoke v2_validation.clj 216]

When running test 2nd time - everything is ok.
But after recreating zen-context - fail repeats.

For now I failed in making simple reproducible test case.

Zen paths priority

There's no defined behaviour on loading the same namespaces from different paths

Possible solutions:

  • Add some strength marker to a path
  • Merge namespaces, make it in a way that later namespaces are merged on top
  • Throw an error, forbid collisions

Urgency: low

:zen/tags validation logic hardcoded into zen.store/validate-resource

Zen symbol validation can not be reproduced with zen.validation or zen.v2-validation code:
:zen/tags are only validated on load here:

(mapv (fn [res] (validate-resource ctx res)))))))

zen/src/zen/store.clj

Lines 54 to 55 in 19e6508

(defn validate-resource [ctx res]
(let [tags (get res :zen/tags)

(let [{errs :errors} (v2/validate ctx schemas res)]

It looks like :zen/tags should be validated in zen.v2-validation/zen.validation and zen.store/validate-resource should just call validate function without handling :zen/tags explicitly

Steps to reproduce:

  • Prepare ztx:
    (def my-zrc
      {'myns '{ns myns
    
               tag {:zen/tags #{zen/schema}
                    :type zen/map
                    :require #{:foo}
                    :keys {:foo {:type zen/any}}}
    
               sym {:zen/tags #{tag}}}})
    
    (def ztx (zen.core/new-context {:memory-store my-zrc}))
    
    (zen.core/load-ns ztx (get my-zrc 'myns))
  • (zen.core/errors ztx) will show 2 errors
    [{:type "require",
      :message ":foo is required",
      :path [:foo],
      :schema [myns/tag :require],
      :resource myns/sym}
     {:message
      "Expected symbol 'myns/tag tagged with '#{zen/tag}, but only #{zen/schema}",
      :type "symbol",
      :path [:zen/tags 0],
      :schema [myns/tag :property :zen/tags :every 0 :tags],
      :resource myns/sym}]
  • But (zen.core/validate ztx #{'zen/schema} (zen.core/get-symbol ztx 'myns/sym)) will show only one error:
    {:errors
     [{:message
       "Expected symbol 'myns/tag tagged with '#{zen/tag}, but only #{zen/schema}",
       :type "symbol",
       :path [:zen/tags 0],
       :schema [zen/schema :property :zen/tags :every 0 :tags]}],
     :warnings [],
     :effects []}
    

Urgency: very low

zen/string :length is defined for zen/schema but has no interpreter implemented

zen/length is defined here:

zen/pkg/zen.edn

Lines 163 to 166 in 6529623

length
{:zen/tags #{schema is-key}
:for #{string}
:zen/desc "lenght of string"}

but defmethod zen.v2-validation/compile-key :length is missing

Execute this to reproduce:

  (def ztx (zen.core/new-context))
  (zen.core/load-ns ztx '{ns myns mystr {:zen/tags #{zen/schema}, :type zen/string, :length 10}})
  (zen.core/errors ztx)
  (zen.core/validate ztx #{'myns/mystr} "hello")
  ;; => {:errors [], :warnings [], :effects []}

Urgency: low
Just noticed this issue, don't use :length myself

Inconsistent tag behavior

When symbol a is tagged with symbol b which has any tags except zen/tag, or symbol a is tagged with symbol a, there is no error.

  1. Error
    (do (require '[zen.core :as zen])
        (def ztx (zen/new-context))
        (def myns '{ns testns
    
                    testnottag
                    {:zen/tags #{zen/schema}}
    
                    testsym
                    {:zen/tags #{testnottag}
                    :a "b"}})
        (zen/load-ns ztx myns)
        (zen/errors ztx))
  2. No error
    (do (require '[zen.core :as zen])
        (def ztx (zen/new-context))
        (def myns '{ns testns
                    testsym
                    {:zen/tags #{testsym}
                    :a "b"}})
        (zen/load-ns ztx myns)
        (zen/errors ztx))
  3. No error
    (do (require '[zen.core :as zen])
        (def ztx (zen/new-context))
        (def myns '{ns testns
    
                    testnottag
                    {:a :b}
    
                    testsym
                    {:zen/tags #{testnottag}
                    :a "b"}})
        (zen/load-ns ztx myns)
        (zen/errors ztx))

Zen generation tools

There often is a need to generate zen definitions, i.e. namespaces and symbols. Currently each such task implements a zen namespace building and spitting

It would be good if zen provided functions like:

  • define-namespace
  • define-symbol
  • spit-namespace/spit-namespaces (create new files with selected zen namespaces)
  • update-namespace/update-namespaces (append new parts to some existing file)

Use-cases:

  • generate zen schemas
  • generate zen definitions using some custom tag (i.e. generate http routes)

Urgency: low
This is just an inconvenience. I can keep reimplementing spit-namespace

zen properties are being validated on any map

For some reason zen tries to valiadate properties on all data coming in, this can be illustrated on a map:
I expect to get error that :zen/tags is not a valid key defined for my map, but I get no errors instead. After I can pass a hash set of symbols it is apparent that it is being validated by the 'zen/tags schema.

  (def ztx (zen.core/new-context {}))

  (zen.core/load-ns ztx '{:ns myns foo {:zen/tags #{zen/schema}
                                        :type zen/map
                                        :keys {:foo {:type zen/string}}}})

  (zen.core/validate ztx #{'myns/foo} {:zen/tags "foo"})
  ;; => {:errors [], :warnings [], :effects []}

  (zen.core/validate ztx #{'myns/foo} {:zen/tags #{'myns/foo}})
  ;; => {:errors
  ;;     [{:message
  ;;       "Expected symbol 'myns/foo tagged with '#{zen/tag}, but only #{zen/schema}",
  ;;       :type "symbol",
  ;;       :path [:zen/tags myns/foo],
  ;;       :schema [myns/foo :property :zen/tags :every myns/foo :tags]}],
  ;;     :warnings [],
  ;;     :effects []}

Value sets

I need to specify that some value may be not just a :enum but some reusable and bigger set of possible values, size of a such value set may reach up to a million values.

Use-case:
A key with possible values from ICD-10

diagnosis-concept
{:zen/tags #{zen/schema}
 :value-set {:sym icd10, :keys #{:code :display}} 
 :type zen/map
 :keys {:code {:type zen/string}
        :display {:type zen/string}}}

diagnosis-code
{:zen/tags #{zen/schema}
 :value-set {:sym icd10, :value :code} 
 :type zen/string}

Urgency: high
Currently I need to implement this as a zen extension making my zen schemas not shareable

zen generator

Generate samples based on zen schema.
Custom generators configuration.

Add key-case instruction

to validate value based on key schema
for example for route-map

 api
 {:zen/tags #{zen/tag zen/schema}
  :type zen/map
  :keys {:apis   {:type zen/symbol :tags #{api}}
         :GET    {:type zen/symbol :tag #{op}}
         :POST   {:type zen/symbol :tag #{op}}
         :PUT    {:type zen/symbol :tag #{op}}
         :DELETE {:type zen/symbol :tag #{op}}
         :PATCH  {:type zen/symbol :tag #{op}}}
  :key-case {:when {:type zen/string}  :then {:confirms #{api}}
             :when {:type zen/vector } :then {:confirms #{api}}}
  :values {:type zen/any}
  }

Forbid a key

This is a form of constraint like require, user would like to forbid for a key to be present in a map

Urgency: low

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.