Giter VIP home page Giter VIP logo

pathom3's Introduction

Pathom 3 Clojars Project Test cljdoc badge bb compatible

Pathom Logo

Logic engine for attribute processing for Clojure and Clojurescript.

Pathom3 is a redesign of Pathom, but it is a new library and uses different namespaces.

Status

Alpha. Changes and breakages may occur. Recommended for enthusiasts and people looking to help, chase bugs, and help improve the development of Pathom.

Install

com.wsscode/pathom3 {:mvn/version "VERSION"}

Documentation

https://pathom3.wsscode.com/

Run Tests

Pathom 3 uses Babashka for task scripts, please install it before proceeding.

Clojure

bb test

ClojureScript

To run once

bb test-cljs-once

Or to start shadow watch and test in the browser:

bb test-cljs

pathom3's People

Contributors

calebmacdonaldblack avatar cbx avatar dehli avatar dependabot[bot] avatar eneroth avatar jj-atkinson avatar mike706574 avatar rodolfo42 avatar royalaid avatar souenzzo avatar tekacs avatar the-alchemist avatar tommy-mor avatar wilkerlucio avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pathom3's Issues

Batch error report

Hello @wilkerlucio
i think we found a problem with batch processing but not sure if i am miss-understanding something on the way batch processing should work. i’ve created this gist https://gist.github.com/jmayaalv/b8cb9b8a6e465ef591773daab0829534 which hopefully can help explain the behavior we are experiencing. Basically the batch results get lost if there are a few batch resolvers involved in the query.

(def db {:a {1 {:a/id 1 :a/code "a-1"}
               2 {:a/id 2 :a/code "a-2"}}
           :f {1 {:f/id 1 :f/code "f-1"}
               2 {:f/id 2 :f/code "f-2"}}
           :b {1 [{:f/id 1}]}
           :c {1 [{:f/id 2}]}})

(pco/defresolver a [input]
    {::pco/input  [:a/id]
     ::pco/output [:a/id :a/code]
     ::pco/batch? true}
    (mapv #(get-in db [:a (:a/id %)]) input))

(pco/defresolver f [input]
  {::pco/input  [:f/id]
     ::pco/output [:f/id :f/code]}
    (get-in db [:f (:f/id input)]))

;; this is the batched version of the resolver above
(pco/defresolver f' [input]
    {::pco/input  [:f/id]
     ::pco/output [:f/id :f/code]
     ::pco/batch? true}
    (mapv #(get-in db [:f (:f/id %)]) input))

(pco/defresolver b [input]
    {::pco/input  [:a/id]
     ::pco/output [{:a/b [:f/id]}]
     ::pco/batch? true}
    (mapv (fn [{:a/keys [id]}]
           {:a/b (get-in db [:b id])}) input))

(pco/defresolver c [input]
    {::pco/input  [:a/id]
     ::pco/output [{:a/c [:f/id]}]
     ::pco/batch? true}
    (mapv (fn [{:a/keys [id]}]
           {:a/c (get-in db [:c id])}) input))

;; non batched f resolver
(p.eql/process (pci/register [a f b c])
                 [{[:a/id 1] [:a/code {:a/b [:f/id :f/code]} {:a/c [:f/id :f/code]}]}])
 ;; => output: 
 ;; {[:a/id 1]
 ;; {:a/code "a-1"
 ;;  :a/b    [{:f/id 1 :f/code "f-1"}]
 ;;  :a/c    [{:f/id 2 :f/code "f-2"}]}}

 ;; batched f resolver
   (p.eql/process (pci/register [a f'  b c])
                 [{[:a/id 1] [:a/code {:a/b [:f/id :f/code]} {:a/c [:f/id :f/code]}]}])

 ;; => output: 
 ;; {[:a/id 1]
 ;; {:a/code "a-1"
 ;;  :a/b    [{:f/id 1 :f/code "f-1"}]
 ;;  :a/c    [{:f/id 2}]}} ;; notice the missing :f/code here

Batch resolver runs twice when inside a placeholder

Pathom version: 2021.07.23-alpha

Here the code to reproduce

(ns dev.pathom
  (:require [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.interface.eql :as p.eql]))

(pco/defresolver todo-batch-resolver
  [env items]
  {::pco/input  [:todo/id]
   ::pco/output [:todo/id :todo/name]
   ::pco/batch? true
   ::pco/cache? false}
  (println "reached")
  (mapv #(assoc % :todo/name "to do") items))


(def env (pci/register [todo-batch-resolver]))

(comment
  ;; print "reached" twice
  (p.eql/process env {:todo/id 1} [{:>/a [:todo/name]}])
  ;; print "reached" once
  (p.eql/process env {:todo/id 1} [:todo/name])
  )

Priorization bug

Following the #48 bug, this code also doesn't obey priorization:

(pco/defresolver default-namespaces [env {:keys [repl/kind]}]
  {::pco/output [:repl/namespace] ::pco/priority 0}

  (prn :PRIORITY-0))

(pco/defresolver namespace-from-editor [inputs]
  {::pco/input [{:editor/ns [:text/contents]}]
   ::pco/output [:repl/namespace]
   ::pco/priority 1}

  (prn :PRIORITY-1)
  {:repl/namespace (-> inputs :editor/ns :text/contents symbol)})

(pco/defresolver repl-kind-from-config [{:config/keys [eval-as]}]
  {::pco/output [:repl/kind] ::pco/priority 1}

  {:repl/kind eval-as})

(pco/defresolver seed-data [{:keys [seed]} _]
  {::pco/output (->> schemas/registry keys (remove #{:map}) vec)
   ::pco/priority 99}
  seed)

#_
(-> [seed-data
     repl-kind-from-config
     default-namespaces namespace-from-editor]
    indexes/register
    (plugin/register (plugins/attribute-errors-plugin))
    (assoc :seed {:repl/kind :clj
                  :editor/ns {:text/contents "some-ns"}})
    (eql/process [:repl/namespace]))

The last code prints :PRIORITY-0 first

If I remove the repl-kind-from-config resolver from the list, the priorization works correctly - even considering that this resolver is not being called on this query.

Smart Maps support in Babashka

Due to some challenges to create new types in Babashka/SCI, it's a bit difficult to implement custom map types in Babashka.

I'm waiting for a feature in process in SCI that may make this process doable: babashka/sci#540

After that, I believe we can have Smart Maps working in Babashka too!

Attribute Errors are silent when there are only "missing" errors

There is a bug in the attribute error plugin.

Reproduction example:

(pco/defresolver full-name [{:keys [first-name last-name]}]
  {:full-name (str first-name last-name)})

(-> (p.eql/process
      (-> (pci/register
            [full-name
             (pbir/constantly-fn-resolver :error #(throw (ex-info "error" {})))])
          (p.plugin/register (pbip/attribute-errors-plugin)))
      [:full-name :error])
    (pf.eql/data->query))
; => [#:com.wsscode.pathom3.connect.runner{:attribute-errors [:error :full-name]}]

In this example we get one error for :error, and also an error for :full-name because it can't be found, but if we don't request the :error attribute, the :error signal for :full-name is also gone:

(-> (p.eql/process
      (-> (pci/register
            [full-name
             (pbir/constantly-fn-resolver :error #(throw (ex-info "error" {})))])
          (p.plugin/register (pbip/attribute-errors-plugin)))
      [:full-name]) ; no :error
    (pf.eql/data->query))
; => [] ; no errors at all, it should have the error for :full-name

Batch resolvers break stuff

Repro:

(let [parents  {1 {:parent/children [{:child/id 1}]
                     :parent/plug     {:plug/id 1}}}
        children {1 {:child/ident :child/good}}
        plugs    {1 {:plug/ident :plug/temp}}
        indexes  (pci/register
                   [(pco/resolver `pc
                      {::pco/input  [:parent/id]
                       ::pco/output [:parent/children]
                       ::pco/batch? true}
                      (fn [{:keys [db]} items]
                        (mapv (fn [{:parent/keys [id]}]
                                (select-keys (parents id) [:parent/children]))
                          items)))

                    (pco/resolver `pp
                      {::pco/input  [:parent/id]
                       ::pco/output [:parent/plug]
                       ::pco/batch? true}
                      (fn [{:keys [db]} items]
                        (mapv (fn [{:parent/keys [id]}]
                                (select-keys (parents id) [:parent/plug]))
                          items)))

                    (pco/resolver `ci
                      {::pco/input  [:child/id]
                       ::pco/output [:child/ident]
                       ::pco/batch? true}
                      (fn [{:keys [db]} items]
                        (mapv (fn [{:child/keys [id]}]
                                (select-keys (children id) [:child/ident]))
                          items)))

                    (pco/resolver `pi
                      {::pco/input  [:plug/id]
                       ::pco/output [:plug/ident]
                       ::pco/batch? true}
                      (fn [{:keys [db]} items]
                        (mapv (fn [{:plug/keys [id]}]
                                (select-keys (plugs id) [:plug/ident]))
                          items)))

                    (pco/resolver `pi*
                      {::pco/input  [:parent/plug]
                       ::pco/output [:plug/id]
                       ::pco/batch? true}
                      (fn [{:keys [db]} items]
                        (mapv (fn [{:parent/keys [plug]}] plug)
                          items)))])]
    [(mapv #(p.eql/process indexes {:all-parents [{:parent/id 1}]} [{:all-parents %}])
       [[{:parent/children [:child/ident]}]
        [:plug/ident
         {:parent/children [:child/ident]}]])
     (mapv #(p.eql/process indexes {:parent/id 1} %)
       [[{:parent/children [:child/ident]}]
        [:plug/ident
         {:parent/children [:child/ident]}]])])

=>

[[{:all-parents [#:parent{:children [#:child{:ident :child/good}]}]}
   {:all-parents [{:plug/ident :plug/temp
                    :parent/children [{}]}]}]
   ;; query directly on parent works fine
   [#:parent{:children [#:child{:ident :child/good}]}
    {:plug/ident :plug/temp, :parent/children [#:child{:ident :child/good}]}]]

Expected:

[[{:all-parents [#:parent{:children [#:child{:ident :child/good}]}]}
   {:all-parents [{:plug/ident :plug/temp
                    :parent/children [#:child{:ident :child/good}]}]}]
 ;; same
]

Tested against latest master commit: cd8f4d0

Fail fast mode

Support a way for exceptions to rise up from resolvers and mutations.

Strict mode

As per discussion at #65 this will change Pathom to fail fast mode by default.

Intended goals for this change:

  • Improve developer experience: quickly find out about invalid attributes or queries, even before they run. trigger exceptions early instead of hiding in the depts of the returned data
  • For a lot of use cases this may remain the method to use in production. Different from before, this provides binary success or failure cases for the request. Not all systems will prefer this. So there is a config to revert to the previous lenient behavior.

This is a breaking change.

To keep Pathom working the same was it was doing before you must add the following config to the env:

{:pathom/lenient-mode? true}

Strict Mode consistency adjusts

One of the goals of strict mode is to give consistent query requirements between resolver inputs and eql process queries.

This issue is to fix the current things that are not:

  • In strict mode, an optional attribute missing on the index will cause a throw, while it is ignored if it were an input (was already working as expected)
  • In lenient mode, an optional attribute missing on the index adds an error, it shouldn't
  • In lenient mode, an optional unreachable attribute adds an error, it shouldn't

Exceptions happening in resolver executions will always flow up in strict mode.

Cycles on nested inputs

Pathom fails to detect cycles that happen with nested inputs.

Example broken setup:

[{::pco/op-name 'cycle-a
  ::pco/output  [:a]}
 {::pco/op-name 'cycle-b
  ::pco/input   [{:a [:b]}]
  ::pco/output  [:b]}]

Gets a stack overflow.

Lenient mode optionals

When a query item is optional it should ignore the missing error on lenient mode.

(p.eql/process
    (pci/register
      {::p.error/lenient-mode? true}
      (pco/resolver 'foo 
        {::pco/output [:bar :baz]}
        (fn [_ _] {:bar "val"})))
    [:bar (pco/? :baz)])
=>
{:bar "val",
 :com.wsscode.pathom3.connect.runner/attribute-errors {:baz {:com.wsscode.pathom3.error/cause :com.wsscode.pathom3.error/node-errors,
                                                             :com.wsscode.pathom3.error/node-error-details {1 {:com.wsscode.pathom3.error/cause :com.wsscode.pathom3.error/attribute-missing}}}}}

; this is wrong, it should be
{:bar "val"}

Foreign lenient mode

Currently, in the case of lenient mode, errors are not propagated back.

Need to explore combinations of lenient vs non-lenient parsers connecting and see how the errors should propagate.

equivalence-resolvers on p.eql/process

I was studying some pathom3 and I think that I found this bug concerning p.eql/process vs smart-maps:

(let [env (pci/register
           [(pbir/equivalence-resolver :acme.user/ip :ip)
            (pco/resolver `a {::pco/input  [:ip]
                              ::pco/output [:longitude
                                            :latitude]}
                          (fn [_ _]
                            {:longitude 1
                             :latitude  2}))
            (pco/resolver `b {::pco/input  [:longitude
                                            :latitude]
                              ::pco/output [:woeid]}
                          (fn [_ _]
                            {:woeid 3}))
            (pbir/constantly-resolver :ip "42")])]
  [((psm/smart-map env {:ip "1"})
    :woeid)
   (p.eql/process env  [:acme.user/ip
                        :woeid])
   (p.eql/process env  [{[:acme/ip "1"] [:woeid]}])
   (p.eql/process env {:acme.user/ip "1"}
                  [:woeid])])

it should gave me {:woeid 3} for the eql process with ident on top of the query, and it's returning me nothing... using constantly-resolver and equivalence-resolver .

It was with :sha "bfa1763c206de546ee8ed1796577a5981a495fc4" at the deps 😄

Optional outputs cause missing input errors

Thanks so much for Pathom, it's a beautiful idea!

I'm noticing that while it's legal to make an optional request for a field that isn't provided by a resolver, if I make an optional request for a field whose provider depends on an unprovided field it errors. For example:

(ns example
  (:require
   [com.wsscode.pathom3.connect.operation :as pco]
   [com.wsscode.pathom3.connect.indexes :as pci]
   [com.wsscode.pathom3.interface.eql :as p.eql]
   [com.wsscode.pathom3.error :as p.error]))

(pco/defresolver a->b
  [{:keys [a]}]
  {::pco/output [:b]}
  {})

(pco/defresolver b->c
  [{:keys [b]}]
  {::pco/output [:c]}
  {:c "c"})

(let [env (pci/register
           [a->b
            b->c])]

  (p.eql/process env {:a 1} [(pco/? :b)])  ; => {}
  (p.eql/process env {:a 1} [(pco/? :c)])) ; throws

   Insufficient data calling resolver 'example/b->c. Missing attrs :b
   {:required {:b {}}, :available {}, :missing {:b {}}}

In the above scenario I would expect both calls to process to return {} without error, but the latter request fails. Is this expected behaviour?

Batch pre-cache lookup is broken

When the batch is checking if there is a moment it assumes the cache is a deferrable, which is not necessary true, a custom cache store might not be deferrable.

From user report:

is resolver-cache* supposed to be derefed here? seem like the code shouldn't assume it's an atom but use the CacheStore protocol instead? https://github.com/wilkerlucio/pathom3/blob/21fad8734c7aa1f18cb02c0e2a72a5a6e1f38e91/src/main/com/wsscode/pathom3/connect/runner/async.cljc#L211

Spec for plan-cache is wrong

From slack conversation: https://clojurians.slack.com/archives/C87NB2CFN/p1628795228076000

the plan-cache spec should allow for pcache/CacheStore protocol

(ns pathom3.experiment
  (:require [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.interface.eql :as p.eql]
            [com.wsscode.pathom3.connect.built-in.resolvers :as pbir]
            [com.wsscode.pathom3.connect.planner :as pcp]
            [clojure.core.cache.wrapped :as cc]
            [com.wsscode.pathom3.cache :as pcache]))


(defrecord CoreCacheStore [cache-atom]
  pcache/CacheStore
  (-cache-lookup-or-miss [_ cache-key f]
    (cc/lookup-or-miss cache-atom cache-key (fn [_] (f))))

  (-cache-find [_ cache-key]
    (find @cache-atom cache-key)))


(pco/defresolver a []
  {::pco/output [:a]}
  {:a "hello world"})

(defn new-env []
  (-> (pci/register [a])
      (pcp/with-plan-cache (->CoreCacheStore (cc/lu-cache-factory {} :threshold 1024)))))

(comment
  (p.eql/process (new-env) {} [:a]))

Dynamic Resolvers

Dynamic resolvers are a special type of resolver that allows for "sideways batching" in Pathom.

This kind of resolvers enables efficient integration with things like Datomic, GraphQL, SQL, etc... Also a requirement to make Pathom integration with other Pathom graphs, which will enable the Maximal Graph vision.

I would say this is 80% there. The hard part of planning around dynamic resolvers has been done for a year, but given the new process design on Pathom 3, the runner part needs a rewrite. I also need to gain more confidence in the planner.

To gain confidence and consider this done, I think we need to have a few working solutions that use dynamic resolvers, this way we can test if it behaves as expected.

Plans of dynamic resolver implementations for Pathom 3:

Support custom cache key per resolver

Some resolvers may want to vary cache other than just input and params.

One use case I have is for APIs that support partial attribute loading (like Twitter or Youtube).

Another type of case is to use cache stores that don't support rich data types (like saving to the file system), in those cases it's desirable to have a simple string cache key.

A solution will involve adding a new option called ::pco/cache-key, which is a function that will receive env and input and should return the cache key value.

Performance problems with big ::pco/input size

Recently we've had a case with a resolver (here c) which take the output of two different resolvers as an input (here a and b), it looks like the compute plan step is very long to run, I didn't dive in the index composition but I suspect they aren't optimized for this use cases (big input keys cardinality)

(ns dev.pathom-playground
  (:require [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.interface.eql :as p.eql]
            [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.connect.runner :as p.runner]))

(def N 100)

(defn range-keys
  "(range-keys \"a\" 3)
  => [:a0 :a1 :a2 :a3]"
  [prefix n]
  (->> (range n)
       (mapv #(keyword (str prefix %)))))

(pco/defresolver a []
  {::pco/input  []
   ::pco/output (range-keys "a" N)}
  (into {} (map #(vector % :value) (range-keys "a" N))))

(pco/defresolver b []
  {::pco/input  []
   ::pco/output (range-keys "b" N)}
  (into {} (map #(vector % :value) (range-keys "b" N))))

(pco/defresolver c []
  {::pco/input  (into [] (concat (range-keys "a" N) (range-keys "b" N)))
   ::pco/output (range-keys "c" N)}
  (into {} (map #(vector % :value) (range-keys "c" N))))

(def env (pci/register [a b c]))

(comment
  (time (p.eql/process env
                       {}
                       (into [] (range-keys "c" N))))
  ;; "Elapsed time: 2255.596461 msecs"

  (let [{::p.runner/keys [compute-plan-run-start-ms compute-plan-run-finish-ms
                          graph-run-start-ms graph-run-finish-ms]}
        (-> (p.eql/process env
                           {}
                           (into [] (range-keys "c" N)))
            meta
            :com.wsscode.pathom3.connect.runner/run-stats)]
    {:compute-plan-run (int (- compute-plan-run-finish-ms compute-plan-run-start-ms))
     :graph-run        (int (- graph-run-finish-ms graph-run-start-ms))})
  ;; => {:compute-plan-run 2293, :graph-run 2298}
  )

Add config to make "throw on missing" optional

In the new strict mode, an attribute failure may happen because:

  1. Attribute is unknown in the index
  2. Attribute exists in the index, but available data isn't enough to reach it
  3. The resolver responsible for the attribute did throw an exception
  4. The resolver didn't return a key with that attribute (which is different from a key with a nil value, which is a valid response)

For strict mode cases, 1 to 3 will always get an exception up, but case 4 is arguable.

We can add a config option like :pathom/throw-on-attribute-missing?, which defaults to true in strict mode but can be turned off by the user.

java.lang.ClassCastException when the parent resolver of a batch resolver returns a seq.

If the parent of a batch resolver returns a seq, Pathom raises

   clojure.lang.PersistentList cannot be cast to
   clojure.lang.Associative

                   RT.java:  827  clojure.lang.RT/assoc
                  core.clj:  191  clojure.core/assoc
                  core.clj: 6169  clojure.core/assoc-in
                  core.clj: 6169  clojure.core/assoc-in
                  core.clj: 6169  clojure.core/assoc-in
                  core.clj: 6161  clojure.core/assoc-in
                 Atom.java:   65  clojure.lang.Atom/swap
                  core.clj: 2354  clojure.core/swap!
                  core.clj: 2345  clojure.core/swap!
          entity_tree.cljc:   45  com.wsscode.pathom3.entity_tree$swap_entity_BANG_$f46764__46797/invoke
          entity_tree.cljc:   32  com.wsscode.pathom3.entity_tree$swap_entity_BANG_/invokeStatic
          entity_tree.cljc:   32  com.wsscode.pathom3.entity_tree$swap_entity_BANG_/invoke
               runner.cljc:  631  com.wsscode.pathom3.connect.runner$run_batches_BANG_/invokeStatic

To reproduce:

(def db {:contracts {"contract-1" {::contract/id   1
                               ::contract/code "contract-1"
                               ::contract/name "the name"}}
         ;; if positions below, return a vec everything works as expected.
         :positions {1 '({::position/units  10
                         ::instrument/fund {::fund/id 1}}
                        {::position/units  20
                         ::instrument/fund {::fund/id 2}})}
         :funds     {1 {::fund/code "fund 1"}
                     2 {::fund/code "fund 2"}}})

(pco/defresolver contract-resolver [{::contract/keys [code]}]
  {::pco/output [::contract/id ::contract/code ::contract/name]}
  (get-in db [:contracts code]))

(pco/defresolver position-resolver [{::contract/keys [id]}]
  {::pco/output [{::contract/positions [::position/units {::instrument/fund [::fund/id]}]} ]}
  {::contract/positions (get-in db [:positions id])})

(pco/defresolver fund-resolver [input]
  {::pco/input [::fund/id]
   ::pco/output [::fund/code]
   ::pco/batch? true}
  (map #(get-in db [:funds (::fund/id %)])
       input))

(p.eql/process
  (pci/register [contract-resolver position-resolver fund-resolver])
  [{[::contract/code "contract-1"] [::contract/name {::contract/positions [::position/units {::instrument/fund [::fund/code]}]}]}])

Resolver priorization error

Hi, on my project I am trying to use the resolver priorization, but they are resolving out of order. Below, the offending code:

(ns example
  (:require [com.wsscode.pathom3.connect.indexes :as indexes]
            [com.wsscode.pathom3.interface.async.eql :as eql]
            [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.plugin :as plugin]
            [com.wsscode.pathom3.connect.built-in.plugins :as plugins]))

(pco/defresolver p2 [{:config/keys [repl-kind]}]
  {::pco/output [:repl/kind] ::pco/priority 2}
  (prn :CHECKING-PRIORITY-2))

(pco/defresolver p1 [{:config/keys [eval-as]}]
  {::pco/output [:repl/kind] ::pco/priority 1}
  (prn :CHECKING-PRIORITY-1))

(pco/defresolver seed-data [{:keys [seed]} _]
  {::pco/output [:config/eval-as :editor/filename :config/repl-kind]
   ::pco/priority 99}
  seed)

#_
(eql/process (-> [seed-data p1 p2]
                 indexes/register
                 (plugin/register (plugins/attribute-errors-plugin))
                 (assoc :seed {:config/repl-kind :cljs, :config/eval-as :clj}))
         [:config/repl-kind :repl/kind])


#_
(eql/process (-> [seed-data p1 p2 p0]
                 indexes/register
                 (plugin/register (plugins/attribute-errors-plugin))
                 (assoc :seed {:config/repl-kind :cljs, :config/eval-as :clj}))
         [:config/repl-kind :repl/kind])

The query prints:

:CHECKING-PRIORITY-1
:CHECKING-PRIORITY-2

Instead of :CHECKING-PRIORITY-2 first. I'm also evaluating the code as ClojureScript, if it helps :)

Cannot generate a plan

Discussed in https://github.com/wilkerlucio/pathom3/discussions/61

Originally posted by markaddleman June 17, 2021

(ns pathom3.experiment
  (:require [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.interface.eql :as p.eql]
            [com.wsscode.pathom3.plugin :as p.plugin]
            [com.wsscode.pathom3.connect.built-in.plugins :as pbip]))

(pco/defresolver attr-id [input]
  {::pco/input  [:portfolioKey :attribute/attribute :attribute/dataType :attribute/path]
   ::pco/output [:attribute/id]}
  (println "attr-id")
  {:attribute/id input})

(pco/defresolver entity-attributes []
  {::pco/input  []
   ::pco/output [{:entity/attributes [:portfolioKey :attribute/attribute :attribute/path :attribute/dataType]}]}
  (println "entity-attributes")
  {:entity/attributes []})

(pco/defresolver ui-attributes []
  {::pco/input  [{:entity/attributes [:attribute/id :attribute/attribute :attribute/path :attribute/dataType]}]
   ::pco/output [{:ui/all-filter-attributes [:attribute/id :attribute/attribute :attribute/path :attribute/dataType]}]}
  (println "ui-attributes")
  {:ui/all-filter-attributes []})

(defn new-env []
  (-> (pci/register [attr-id
                     entity-attributes
                     ui-attributes])
      #_(p.plugin/register (pbip/attribute-errors-plugin))))

(comment
  ; pathom successfully creates a plan
  (let [q '[{(:>/GroupBys {}) [{:entity/attributes [:attribute/id :attribute/attribute :attribute/path :attribute/dataType]}]}]]
    (p.eql/process (new-env) q)))

(comment
  ; pathom does not create a plan
  (let [q '[{(:>/GroupBys {}) [:ui/all-filter-attributes]}]]
    (p.eql/process (new-env) q)))
```</div>

Nested dynamic inputs bug

(pco/defresolver order-line-items []
  {::pco/output
   [{:order/line-items
     [:line-item/id
      :line-item/quantity
      :line-item/title
      :line-item/price-total]}]}
  {:order/line-items
   [{:line-item/id          "1628545763873-1"
     :line-item/price-total 28.8M}
    {:line-item/id          "1628545763873-2"
     :line-item/price-total 38.9M}]})

(def items-env
  (-> (pci/register
        [order-line-items])
      (assoc ::pci/index-source-id `line-items)))

(def items-request
  (p.eql/boundary-interface items-env))

(pco/defresolver order-items-total [{:keys [order/line-items]}]
  {::pco/input
   [{:order/line-items
     [:line-item/price-total]}]}
  {:order/items-total (transduce (map :line-item/price-total) + line-items)})

(def demo-env
  (-> (pci/register
        [(pcf/foreign-register items-request)
         order-items-total])
      ((requiring-resolve 'com.wsscode.pathom.viz.ws-connector.pathom3/connect-env)
       "debug")))

(comment
  (p.eql/process demo-env
    [:order/items-total]))

The foreign-ast to the order-line-items currently is:

{:children
  [{:dispatch-key :order/line-items,
    :key :order/line-items,
    :type :prop}],
  :type :root}

This is missing the sub-query part, it must include :line-item/price-total in the foreign-ast subquery.

Possible process error

This is an issue under investigation reported by Mark Addleman:


I'd love to be able to give you a small repro case but I can't figure out how to reproduce the problem under simple conditions```

Batch resolver without cache runs forever

Reported by @cyppan

(run-graph
  (pci/register
    [(pco/resolver 'batch-no-cache
       {::pco/batch? true
        ::pco/cache? false
        ::pco/input  [:id]
        ::pco/output [:name]}
       (fn [_ input]
         input))])
  {:id 1}
  [:name])

The setup presented up there will run forever.

Seeing odd resolver timing in Pathom Viz

I'm experimenting with Pathom to determine whether it might be appropriate for an upcoming project.

I've created a query that calls a resolver 4 times, and am noticing that Pathom Viz is reporting that the final trace of a resolver is taking ~200ms. (The first three times it is called, it only takes ~2ms.) I don't think this has anything to do with the resolver itself. Have you run across this elsewhere?

image

Generative Tests - nested requests

This task is to include the support to generate nested process cases for Pathom.

This is a pre-requisite to enable generative testing of other things like:

  • Batch generative testing
  • Placeholder generative testing

Generative Tests

Due to the complexity of the Pathom execution model, generative tests will add greater confidence in the process.

The idea here is to make generators capable of generating random resolvers and expected results, they will mix and match many different patterns and check if the result still as expected.

  • Flat queries

Extended dynamic input queries

Allow a user query to extend a nested input from a resolver.

In the following example scenario:

(pco/defresolver all-people3 [{:swapi.Root/keys [allPeople]}]
  {::pco/input
   [{:swapi.Root/allPeople
     [{:swapi.PeopleConnection/people
       [:swapi.types/Person]}]}]

   ::pco/output
   [{:swapi/all-people
     [:swapi.types/Person]}]}
  {:swapi/all-people
   (get allPeople :swapi.PeopleConnection/people)})

With that setup, that re-shapes data from a dynamic source, a user query like:

[{:swapi/all-people
  [:swapi.Person/name]}]

Should extend the sub-query at :swapi.PeopleConnection/people.

One point to consider is that a query input/output may have multiple nested points, and how to move from where to where needs to be defined somehow.

My first idea would be to add some parameters to mark the compatible extensions.

This needs more thought.

API namespace

I'd like to learn more about this project. However I can't continue, because the documentation says:

When you read code from the examples in this documentation they will use alias to reference namespaces, here you can find what the aliases point to:

[com.wsscode.pathom3.cache :as p.cache]
[com.wsscode.pathom3.connect.built-in.resolvers :as pbir]
[com.wsscode.pathom3.connect.built-in.plugins :as pbip]
[com.wsscode.pathom3.connect.foreign :as pcf]
[com.wsscode.pathom3.connect.indexes :as pci]
[com.wsscode.pathom3.connect.operation :as pco]
[com.wsscode.pathom3.connect.operation.transit :as pcot]
[com.wsscode.pathom3.connect.planner :as pcp]
[com.wsscode.pathom3.connect.runner :as pcr]
[com.wsscode.pathom3.error :as p.error]
[com.wsscode.pathom3.format.eql :as pf.eql]
[com.wsscode.pathom3.interface.async.eql :as p.a.eql]
[com.wsscode.pathom3.interface.eql :as p.eql]
[com.wsscode.pathom3.interface.smart-map :as psm]
[com.wsscode.pathom3.path :as p.path]
[com.wsscode.pathom3.plugin :as p.plugin]

This problem could be solved by an API namespace.

Batch resolver with parameters can't find different parameters for the batch

Provided that I have a resolver that increments a number, given a specific param (for example, the number that the parameter should be incremented):

(let [idx (connect/resolver
           'increments
           {::connect/input [:number]
            ::connect/output [:incr-number]
            ::connect/batch? true}
           (fn [env numbers]
             (map (fn [{:keys [number]}] 
                    {:incr-number (+ number (:inc (connect/params env)))})
                  numbers)))]
  (eql/process (idx/register idx)
               {:plus-1 [{:number 1} {:number 2}]
                :plus-2 [{:number 3} {:number 4}]}
               [{:plus-1 [:number '(:incr-number {:inc 1})]}
                {:plus-2 [:number '(:incr-number {:inc 2})]}]))

This EQL results in:

{:plus-1 [{:number 1, :incr-number 3} {:number 2, :incr-number 4}], 
 :plus-2 [{:number 3, :incr-number 5} {:number 4, :incr-number 6}]}

;; :plus-1 should be: [{:number 1, :incr-number 2} {:number 2, :incr-number 3}]

The batch can't "see" that params is expected to be different for inputs {:number 1}, {:number 2} and {:number 3}, {:number 4}.

What I found is that, at the time this resolver is called, there's only a single place on the env that tells me {:inc 1} exists - on key :com.wsscode.pathom3.connect.runner/root-query. After the query ran, I can also see the it on :com.wsscode.pathom3.connect.runner/resolver-cache*.

I see two possible solutions: either (connect/params) returns a list of params for each input (so we can (map ... inputs params) and see the individual params, or even group-by in some way) or batch can be called only for queries with the same params. The first solution could make batches harder to implement; at the second, batches' implementations will not change at all (but for situations were we can batch everything, and apply the params on the results - that means, the most expensive operation is not dependent on params at all, we'll lose the ability to aggressive batch).

So... WDYT?

Foreign unions

Support unions queries on foreign parsers.

Pathom should forward the union query to the foreign parser, considering the possible paths.

Exception on batch resolver + nested inputs resolver

Repro:

(let [parents  {1 {:parent/children [{:child/id 1}]}}
        children {1 {:child/ident :child/good}}
        good?    (fn [children] (boolean (some #{:child/good} (map :child/ident children))))
        indexes  (pci/register
                   [(pco/resolver `pc
                      {::pco/input  [:parent/id]
                       ::pco/output [{:parent/children [:child/id]}]
                       ::pco/batch? true}
                      (fn [{:keys [db]} items]
                        (mapv (fn [{:parent/keys [id]}]
                                (select-keys (parents id) [:parent/children]))
                          items)))

                    (pco/resolver `ci
                      {::pco/input  [:child/id]
                       ::pco/output [:child/ident]
                       ::pco/batch? true}
                      (fn [{:keys [db]} items]
                        (mapv (fn [{:child/keys [id]}]
                                (select-keys (children id) [:child/ident]))
                          items)))

                    (pco/resolver `parent-good
                      {::pco/input  [{:parent/children [:child/ident]}]
                       ::pco/output [:parent/good?]}
                      (fn [_ {:parent/keys [children]}] {:parent/good? (good? children)}))])]
    (p.eql/process indexes {:parent/id 1} [:parent/good?]))

=>

Execution error (NullPointerException) at com.wsscode.pathom3.entity-tree/reset-entity!$f77329 (entity_tree.cljc:30).
null

Expected:

{:parent/good? true}

Tested on latest master commit: 93b2558

Assert failed: Tried to remove node that still contains references pointing to it

Hi,

I'm seeing the following issue when I am trying to use two alias resolvers, but if I remove some other resolvers, it starts to work again, so the alias specifically do not appear to be the source of problem. I will try to get the minimum number of steps and resolvers in the next days (I'm with 16 in total) to make a good example, but I'm putting here the output as maybe you already have some idea of what may cause this assertion failure.

I've been able to shrink it down to the following code.

Any of the solutions make it work again:

  • Remove employer by id resolver;
  • Remove one of the keys in the EQL query (:foo.routing-number/bank-name, :foo.bank-account/routing-number or :foo.contact/email);
  • Use :foo.routing-number/routing-number instead of :foo.bank-account/routing-number.

Also when I try to use the equivalence resolver instead of the alias resolver, it does not give me an error, but it also does not return the key (but this appears to be another issue, just FYI).

I'm using version 523dab6c4d9c9c9bc2df323a86b4f4e9833f66bb.

(ns foo.issue
  (:require
   [com.wsscode.pathom3.connect.built-in.resolvers :as pbir]
   [com.wsscode.pathom3.connect.indexes :as pci]
   [com.wsscode.pathom3.connect.operation :as pco]
   [com.wsscode.pathom3.interface.eql :as p.eql]))

(pco/defresolver employer-by-id-resolver
  [{:keys [foo.employer/id]}]
  {::pco/output [:foo.employer/id
                 :foo.contact/id
                 :foo.bank-account/bank-account-id]}
  {:foo.employer/id 1
   :foo.contact/id 1
   :foo.bank-account/bank-account-id 1})

(pco/defresolver employer-by-external-id-resolver
  [{:keys [foo.employer/external-id]}]
  {::pco/output [:foo.employer/id
                 :foo.contact/id
                 :foo.bank-account/bank-account-id]}
  {:foo.employer/id 1
   :foo.contact/id 1
   :foo.bank-account/bank-account-id 1})

(pco/defresolver bank-account-resolver
  [{:keys [foo.bank-account/bank-account-id]}]
  {::pco/output [:foo.bank-account/id
                 :foo.routing-number/routing-number]}
  {:foo.bank-account/id 1
   :foo.routing-number/routing-number "number"})

(pco/defresolver contact-resolver
  [{:keys [foo.contact/id]}]
  {::pco/output [:foo.contact/id
                 :foo.contact/email]}
  {:foo.contact/id 1
   :foo.contact/email "[email protected]"})

(pco/defresolver routing-number-resolver
  [{:keys [foo.routing-number/routing-number]}]
  {::pco/output [:foo.routing-number/routing-number
                 :foo.routing-number/bank-name]}
  {:foo.routing-number/routing-number "number"
   :foo.routing-number/bank-name      "name"})

(def env
  (pci/register
   [(pbir/alias-resolver :foo.routing-number/routing-number :foo.bank-account/routing-number)
    employer-by-id-resolver
    employer-by-external-id-resolver
    bank-account-resolver
    contact-resolver
    routing-number-resolver]))

(comment

  (p.eql/process
   env
   [{[:foo.employer/external-id "Z9n7Jx7pOHl6"]
     [:foo.routing-number/bank-name
      :foo.bank-account/routing-number
      :foo.contact/email]}])

  ;; When I remove the employer by id resolver or when I remove one of the requested
  ;; keys in the EQL query or when I use original resolver instead of the alias one,
  ;; it works.

  ())
                   com.wsscode.pathom3.interface.eql/process             eql.cljc:   48
               com.wsscode.pathom3.interface.eql/process-ast             eql.cljc:   20
                 com.wsscode.pathom3.plugin/run-with-plugins          plugin.cljc:  140
              com.wsscode.pathom3.interface.eql/process-ast*             eql.cljc:   13
               com.wsscode.pathom3.connect.runner/run-graph!          runner.cljc:  782
   com.wsscode.pathom3.connect.runner/run-graph-with-plugins          runner.cljc:  767
                 com.wsscode.pathom3.plugin/run-with-plugins          plugin.cljc:  143
com.wsscode.pathom3.connect.runner/run-graph-with-plugins/fn          runner.cljc:  769
                 com.wsscode.pathom3.plugin/run-with-plugins          plugin.cljc:  143
          com.wsscode.pathom3.connect.runner/run-graph-impl!          runner.cljc:  754
            com.wsscode.pathom3.connect.runner/plan-and-run!          runner.cljc:  640
              com.wsscode.pathom3.connect.runner/run-graph!*          runner.cljc:  612
          com.wsscode.pathom3.connect.runner/process-idents!          runner.cljc:  254
                com.wsscode.pathom3.entity-tree/swap-entity!     entity_tree.cljc:   37
                                          clojure.core/swap!             core.clj: 2352
                                                         ...                           
       com.wsscode.pathom3.connect.runner/process-idents!/fn          runner.cljc:  255
    com.wsscode.pathom3.connect.runner/process-attr-subquery          runner.cljc:  207
     com.wsscode.pathom3.connect.runner/process-map-subquery          runner.cljc:  143
               com.wsscode.pathom3.connect.runner/run-graph!          runner.cljc:  782
   com.wsscode.pathom3.connect.runner/run-graph-with-plugins          runner.cljc:  772
                 com.wsscode.pathom3.plugin/run-with-plugins          plugin.cljc:  143
          com.wsscode.pathom3.connect.runner/run-graph-impl!          runner.cljc:  754
            com.wsscode.pathom3.connect.runner/plan-and-run!          runner.cljc:  632
       com.wsscode.pathom3.connect.planner/compute-run-graph         planner.cljc: 1793
       com.wsscode.pathom3.connect.planner/compute-run-graph         planner.cljc: 1807
                            com.wsscode.pathom3.cache/cached           cache.cljc:   53
                    com.wsscode.pathom3.cache/eval59266/fn/G           cache.cljc:   10
                      com.wsscode.pathom3.cache/eval59292/fn           cache.cljc:   28
    com.wsscode.pathom3.connect.planner/compute-run-graph/fn         planner.cljc: 1810
      com.wsscode.pathom3.connect.planner/compute-run-graph*         planner.cljc: 1733
                                         clojure.core/reduce             core.clj: 6827
                                                         ...                           
   com.wsscode.pathom3.connect.planner/compute-run-graph*/fn         planner.cljc: 1737
 com.wsscode.pathom3.connect.planner/compute-attribute-graph         planner.cljc: 1721
com.wsscode.pathom3.connect.planner/compute-attribute-graph*         planner.cljc: 1678
        com.wsscode.pathom3.connect.planner/compute-root-and         planner.cljc:  978
     com.wsscode.pathom3.connect.planner/compute-root-branch         planner.cljc:  872
      com.wsscode.pathom3.connect.planner/create-branch-node         planner.cljc:  779
         com.wsscode.pathom3.connect.planner/add-branch-node         planner.cljc:  759
                                         clojure.core/reduce             core.clj: 6828
                                 clojure.core.protocols/fn/G        protocols.clj:   13
                                   clojure.core.protocols/fn        protocols.clj:   75
                          clojure.core.protocols/iter-reduce        protocols.clj:   49
      com.wsscode.pathom3.connect.planner/add-branch-node/fn         planner.cljc:  761
         com.wsscode.pathom3.connect.planner/add-branch-node         planner.cljc:  764
             com.wsscode.pathom3.connect.planner/remove-node         planner.cljc:  570
java.lang.AssertionError: Assert failed: Tried to remove node that still contains references pointing to it. Move
                                the run-next references from the pointer nodes before removing it. Also check if
                                parent is branch and trying to 
                          (if node-parents (every? (fn* [p1__59506#] (not= node-id (-> (get-node graph p1__59506#) :com.wsscode.pathom3.connect.planner/run-next))) node-parents) true)

Thanks and great project, Wilker!

Nested dynamic resolver inputs

When planning nested inputs, in the case of dynamic resolvers Pathom isn't doing the right job.

Broken example:

(let [foreign (-> (pci/register
                    (pco/resolver 'n
                      {::pco/output [{:a [:b]}]}
                      (fn [_ _] {:a [{:b 1}
                                     {:b 2}
                                     {:b 3}]})))
                  (p.eql/boundary-interface))
      env     (-> (pci/register
                    [(pco/resolver 'sum
                       {::pco/input
                        [{:a [:b]}]

                        ::pco/output
                        [:total]}
                       (fn [_ {:keys [a]}]
                         (transduce (map :b) + 0 a)))
                     (pcf/foreign-register foreign)]))]
  (p.eql/process env [:total]))

Results are sometimes dropped when batched and unbatched resolvers occur in the same query

This is going to be hard to produce a minimally working example of, but perhaps we can show you live.

There are two resolvers, sub-atoms and particles. sub-atoms is batched, particles is not. When a query is run which involves both, it returns:

    :particles [{} {}],
    :sub-atoms []}

(sub-atoms should be empty here, but the particles should not be empty)

When we make particles a batch resolver, they resolve as intended.

Planner Basics Redesign

Given the evidence from #33 , this issue will be a track of the work around the changes on the planner.

The first thing I want to try is to remove all optimizations I have in the planner algorithm. At the time of building, I thought it made sense to try to re-use parts of the graph as they are built.

The generative tests were able to expose to me new cases that I hadn't thought about, and this makes me see that I underestimated the complexity of these optimizations.

The idea is not to leave the graph all filled with duplicated unnecessary nodes. Instead, I want to start building it as simple as possible (which may generate a lot of duplicated nodes, depending on query/graph) and after the fact run extra optimization steps to make the graph only complex as it needs to be.

This idea will cost more to process on the planner for sure, but I don't think it's gonna be a problem, compared to IO it's not adding that much, and plans are cacheable, so even if very large queries require some extra time to plan, a next run of the same query can load from the cache and have it instantly.

  • Experiment removing all optimizations and check the reliability of the algorithm without them.

Feasibility of more complex queries

I'm using Pathom 2 and wondering if you have plans for 'natively' handling complex queries.

There are a couple of reasons why this might make sense:

  1. Writing code to strip a 'dumb' query result of unwanted values reifies the structure of the data outside of the query. This can introduce a future disconnect between the shape of the data as requested and the shape as needed by the post-processing functions.
  2. Having a standard dialect for filtering results could allow for resolvers to take advantage of the requested filters to reduce the scope of the query, when supported by the backend. An example of this is illustrated below.

Let's say I have the following query:

[{:users/list [:user/email :user/employeeType :user/groups]}]
; yields:
{:users/list
 [#:user{:email "jdoe@domain"
         :employeeType "staff"
         :groups [{:group/email "a-group@domain"}
                  {:group/email "b-group@domain"}]}
  #:user{:email "jpublic@domain"
         :employeeType "faculty"
         :groups [{:group/email "a-group@domain"}]}]}

Well, now I'd like to formulate a query to find users who are either faculty or staff, and are members of "a-group@domain". I can solve part of this problem with parameterization on :users/list:

[{(:users/list {:filter "(|(employeeType=staff)(employeeType=faculty))"}) .....

And, once I've implemented the logic to pick that up, that's all well and good. However, the part where I restrict to users who are members of "a-group@domain" is trickier. I created a transformer to filter results based on parameters (not the same as the logic catching the above parameterization), and then I can do something like this:

[{(:users/list {:filter "(|(employeeType=staff)(employeeType=faculty))"})
  [:user/email (:user/groups {:group/email "b-group@domain"})]}]

And have the resolver for :user/groups return :no-match rather than a value when the filter is not satisfied. I then modified the Pathom parser to short-circuit when :no-match is discovered as a value for a child and immediately return the same, and also to strip vector results of that keyword. This works: the above query will then produce

{:users/list
 [#:user{:email "jdoe@domain"
         :groups [{:group/email "b-group@domain"}]}]}

However, there is no easy way to invert this query that I can think of; i.e., it is nontrivial to ask "what users that are either faculty or staff are not members of b-group@domain" - not without going and implementing very specific logic to answer parts of the question, although all of the predicates involved are general. Moreover, I'm already up to two different approaches for filtering, as each one is ad-hoc. The only real difference is that the resolver providing :users/list is capable of restricting its queries ahead of time, whereas the groups resolver does not know how to do that. However, these shouldn't ultimately be a difference in terms of interface, imo. In an ideal world, I would be able to define behavior for handling filters on a given resolver if possible given the particular backend, otherwise letting the results be resolved and filtered post-facto by Pathom.

Interested in any thoughts on how this could be approached. Of course, if I've totally missed something and there's already a better way to do this, I'd love to know.

Nested dynamic resolver params

When there is a param in a nested part of a dynamic resolver request, Pathom is losing the params during the filtering process.

Pathom should retain the params and send to the dynamic source as if it were local.

unexpected errors reported with attribute-errors-plugin and new error handling

When running with the attribute-errors-plugin invalid errors are present in the output when all attrs are resolved.

(def db {:a {1 {:a/id 1 :a/code "a-1"}
               2 {:a/id 2 :a/code "a-2"}}
           :f {1 {:f/id 1 :f/code "f-1"}
               2 {:f/id 2 :f/code "f-2"}}
           :b {1 [{:f/id 1}]}
           :c {1 [{:f/id 2}]}})

(pco/defresolver a [input]
    {::pco/input  [:a/id]
     ::pco/output [:a/id :a/code]
     ::pco/batch? true}
    (mapv #(get-in db [:a (:a/id %)]) input))

(pco/defresolver f [input]
  {::pco/input  [:f/id]
     ::pco/output [:f/id :f/code]}
    (get-in db [:f (:f/id input)]))

(pco/defresolver b [input]
    {::pco/input  [:a/id]
     ::pco/output [{:a/b [:f/id]}]
     ::pco/batch? true}
    (mapv (fn [{:a/keys [id]}]
           {:a/b (get-in db [:b id])}) input))

(pco/defresolver c [input]
    {::pco/input  [:a/id]
     ::pco/output [{:a/c [:f/id]}]
     ::pco/batch? true}
    (mapv (fn [{:a/keys [id]}]
           {:a/c (get-in db [:c id])}) input))

(-> (p.plugin/register (pbip/attribute-errors-plugin))
      (pci/register [a f b c])
      (p.eql/process [{[:a/id 1] [:a/code {:a/b [:f/id :f/code]} {:a/c [:f/id :f/code]}]}]))

=> 
{[:a/id 1]
 {:a/b [#:f{:id 1, :code "f-1"}],
  :a/c [#:f{:id 2, :code "f-2"}],
  :com.wsscode.pathom3.connect.runner/attribute-errors
  #:a{:code
      #:com.wsscode.pathom3.error{:error-type
                                  :com.wsscode.pathom3.error/attribute-unreachable},
      :b
      #:com.wsscode.pathom3.error{:error-type
                                  :com.wsscode.pathom3.error/node-errors,
                                  :node-error-details {}},
      :c
      #:com.wsscode.pathom3.error{:error-type
                                  :com.wsscode.pathom3.error/node-errors,
                                  :node-error-details {}}}}}

Batch dynamic resolvers

In the current implementation, each batch is based on a resolver and an input. As I played more, I notice this is not sufficient to handle batch in dynamic resolvers. That's because dynamic resolvers have one extra variable, which is the foreign-ast they must run.

Foreign AST's may include some input data as part of them, this means they can't be shared between instances of the call, so they must be part of the batch data to have the necessary information to run.

Resolver with nested inputs breaks stuff

Repro:

(let [parents  {1 {:parent/children [{:child/id 1}]}}
        children {1 {:child/ident :child/good}}
        good?    (fn [children] (boolean (some #{:child/good} (map :child/ident children))))
        indexes  (pci/register
                   [(pbir/single-attr-with-env-resolver :parent/id :parent/children
                      (fn [{:keys [db]} id] (get-in parents [id :parent/children])))

                    (pbir/single-attr-with-env-resolver :child/id :child/ident
                      (fn [{:keys [db]} id] (get-in children [id :child/ident])))

                    (pco/resolver `parent-good
                      {::pco/input  [{:parent/children [:child/ident]}]
                       ::pco/output [:parent/good?]}
                      (fn [_ {:parent/keys [children]}] {:parent/good? (good? children)}))

                    (pco/resolver `parent-good*
                      {::pco/input  [:parent/children]
                       ::pco/output [:parent/good?*]}
                      (fn [_ {:parent/keys [children]}] {:parent/good?* (good? children)}))])]
    (mapv #(p.eql/process indexes {:parent/id 1} %)
      [[{:parent/children [:child/ident]}]
       [:parent/good?]
       [:parent/good?*]
       [{:parent/children [:child/ident]}
        :parent/good?]
       [{:parent/children [:child/ident]}
        :parent/good?*]
       [:parent/good?
        :parent/good?*]]))

=>

[#:parent{:children [#:child{:ident :child/good}]}
 {} ;; good? should be there
 #:parent{:good?* false}
 #:parent{:children [#:child{:ident :child/good}]}
 #:parent{:children [#:child{:ident :child/good}]
          :good?* true}
 {} ;; both good? and good?* disappeared
]

Expected:

[#:parent{:children [#:child{:ident :child/good}]}
 #:parent{:good? true}
 #:parent{:good?* false}
 #:parent{:children [#:child{:ident :child/good}]
          :good?    true}
 #:parent{:children [#:child{:ident :child/good}]
          :good?*   true}
 #:parent{:good?  true
          :good?* false}]

Tested against latest commit on master: cd8f4d0

support for for short circuit processing

We have had some performance issues when a attribute is optional and can be provided by multiple resolvers. This was solved on pathom2 using the metadata ^:final. something similar is needed on pathom3. Probably it would make sense to keep the same metadata to help with the migration from pathom2.

Bug in batch + nested inputs

Reported on Clojurians, repro case:

(ns com.wsscode.pathom3.demos.repro-optional-nested
  (:require [com.wsscode.pathom3.connect.indexes :as pci]
            [com.wsscode.pathom3.connect.operation :as pco]
            [com.wsscode.pathom3.interface.eql :as p.eql]))

;; that one works fine
(pco/defresolver timezone-resolver [item]
  {::pco/input  [:timezone/id]
   ::pco/output [:timezone/label]}
  {:timezone/label (str "label for id " (:timezone/id item))})

;; that one (batched) is buggy
(pco/defresolver timezone-batch-resolver [items]
  {::pco/input  [:timezone/id]
   ::pco/output [:timezone/label]
   ::pco/batch? true}
  (mapv (fn [item]
          {:timezone/label (str "label for id " (:timezone/id item))})
    items))

(pco/defresolver item-checks [item]
  {::pco/input  [{(pco/? :item/timezone) [:timezone/id :timezone/label]}]
   ::pco/output [:item/checks]}
  {:item/checks {:timezone-id?    (some? (get-in item [:item/timezone :timezone/id]))
                 :timezone-label? (some? (get-in item [:item/timezone :timezone/label]))}})

(comment
  ;; KO, timezone/label is not resolved
  (p.eql/process
    (pci/register [timezone-batch-resolver
                   item-checks])
    {:item/name     "my item"
     :item/timezone {:timezone/id "UTC"}}
    [:item/checks])
  ;; => #:item{:checks {:timezone-id? true, :timezone-label? false}}

  ;; OK, both timezone/id and timezone/label are found
  (p.eql/process
    (pci/register [timezone-resolver
                   item-checks])
    {:item/name     "my item"
     :item/timezone {:timezone/id "UTC"}}
    [:item/checks])
  ;; => #:item{:checks {:timezone-id? true, :timezone-label? true}}
  )

Planner fails to find join

In https://gist.github.com/markaddleman/a8ecdfbc1ec71f56511f97251cc15898 there is an example of a client query that fails when a resolver requires a join as an input but succeeds when the client queries for the same join. When the planner fails, it throws an exception with the following info:
com.wsscode.pathom3.connect.planner/unreachable-paths: #:event{:id {}} com.wsscode.pathom3.error/cause: :com.wsscode.pathom3.error/attribute-unreachable com.wsscode.pathom3.error/phase: :com.wsscode.pathom3.connect.planner/plan com.wsscode.pathom3.path/path: [:workflows.workflow/events]

Parallel Process

Pathom 3 ability to run resolvers in parallel.

Compared to Pathom 2, the Pathom 3 parallel implementation is likely to be simpler. The new planner results already have all the information the runner needs to know what it can parallelize, this will reduce the large amount of complexity that was required in the parallel runner from Pathom 2.

The challenge now becomes how to do good resource management around it. Given Pathom 3 uses Promesa, the initial feeling is that we could just let it run. But in my experience, this can easily turn into high resource usage on the server, so I like to add some control mechanisms in Pathom.

Some tunning dials I like to offer in Pathom to control parallelism:

  • Maximum number of collection items to run in parallel in a given process
  • Maximum number of parallel resolvers to run sideways
  • Maximum number of threads used per request

And some current things I know are broken in parallel and need fixing (not a bug for current usage, since there is not parallel process):

  • Ensure parallel merging doesn't lose data in concurrent operation s

If anybody has experience with high concurrency on the JVM (especially using CompletableFuture), I would love to know about ideas to make an efficient engine for a parallel process.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.