Personally, I get the implied message from the opening line, that this example would be a 'follow along and do as I do' (starts with, "First, we'll...."). I came to me, that the nature of system makes it hard to give a working example out of the box (Stuff Just Works (TM)). Should we perhaps add a minor introduction about the required additional dependencies (and/or plugins) and the internal namespace (which exposes handler)? Or perhaps, due to that composable nature, just refer to the example projects (boot/lein) since those actually do contain the required (albeit perhaps slightly outdated at times, as a worse case) dependencies/plugins.

I guess what further adds to some confusion, is the later mentioning of ns front-end.webapp.handler in a somewhat different context.

Maybe it's just me, but I'm never that fond of the (go find out yourself) attitude which has hidden (phony) code references in examples (especially not on the README) that so hallmark the JS community (grab reference libraries from code, show only non-working, most trivial examples, mention but not elaborate on obscure edge-case or common use-case scenario's). Which doesn't mean I think system has that btw, I am actually wondering if there would be much to write about in the sense that it needs a manual. Most is up to the users creativity I guess, but a nice tutorial showing all steps might be a nice/useful addition.

Your thoughts?

I suggest to use Prismatic's schema for validation.

It's very well capable and trusted, and I feel it's a more elegant solution than the custom rolled assertions we're doing now.

I've started using Prismatic schema with the Mongo component.

It would be desirable to go back to components that have assertions and turn them into Schema validations. (Jetty, Http-kit, Immutant, etc.)

Contributions welcome.

This is a good task for Clojure learners, as it's a simple task (and I'll help).

I think it is a good idea to check if you started a component before to execute the code to stop it.
The point is when you are developing and trying your code you can introduce some bug that avoid the start of a component and after you fix it you are unable to restart the system because you can not stop the component that have the bug.

• ImmutableSettings was removed
• Transport API has changed

It might be prudent to create 2 ElasticSearch components, one for 1.x and one for 2.x. Thoughts, @danielsz?

Right now system uses boot.core/by-name with the files option in boot.clj. Consider adding a regex option that would use boot.core/by-re.

My use case is that I want all files within a directory to trigger a reset (I'm using weavejester's duct, and I want all endpoints to trigger a reset).

Typically, learning about the project means reading the README and studying the examples.

What kind of documentation effort do we want to undertake? What can we realistically achieve?
Would a Wiki fill in a need?

The discussion is open to everyone, especially new users since they know best about how smooth or how rugged their approach was. As long as the memory of your pain is alive, your opinion will matter more. Even if you don't intend to contribute to a documentation effort, please feel free to tell your experience on this thread.

I found a require for compojure.core but unused.
Is it necessary?

https://blog.juxt.pro/posts/component-meet-schema.html

This is very pertinent information, and if anyone has a specific idea on how to leverage this, I'm all ears.

The article is from a user perspective, who can do validations when the system is being assembled. As a library, we are doing validation on the constructor of the component. We cannot necessarily do all the validations described, am I wrong?

At any rate, this is the kind of things that would fit in a section in the README. A paragraph on the kind of validation mentioning that we do, the kind of validations that is possible to do, and a link to Juxt's article.

I'm writing a server based on the boot example, trying to learn boot and component.js simultaneously. Given the boot task:

(deftask dev
  "Run a restartable system in the Repl"
  []
  (comp
   (environ :env {:http-port 3000})
   (watch :verbose true)
   (system :sys #'dev-system)
   (repl :server true)))

And the system:

(defn dev-system []
  (component/system-map
   :web (new-web-server (Integer. (env :http-port)) handler)))

Where might I tell the system to start automatically, so that I don't have to open a repl first and run (go) to run the server?

I see there is support for hot reloading with Boot but apparently none for lein. Is this an intentional decision? Would you consider adding support for it?

My component-making function takes a parameter like so:

(defn new-system [profile] )

And I am trying to create a new system:
(system.boot/system :sys (partial make-system profile) :auto true)
This fails with: java.lang.Exception: sys argument expects a Var, eg. #'system-dev

I tried working around with:

• (defn make-system [] (partial 'new-system profile))
• with-local-vars

Is there a way to pass a parameter to the function creating the component?

Note: I would like to use system with boot in case it matters

I have some code (a ring app served with jetty) that stores state in an atom for the duration of the application, the data is not persisted anywhere. Unfortunately every time a route handler is changed the application is reloaded and my atom is trashed. I guess this can probably be worked around with a defonce for the state but that feels like a kludge.

I couldn't find anything about something like this, did I miss it or would you suggest something else?

I am a Ring/Compojure beginner and trying to use this library. In README you have this:

(defn dev-system []
  (component/system-map
   :datomic-db (new-datomic-db (env :db-url))
   :mongo-db (new-mongo-db)
   :web (new-web-server (env :http-port) handler)))

This is slightly confusing to me. Do you not need to create the handler as a
separate component? I don't see how the db(s) are injected into the handler in
the above example. (I assume the handler is what needs the dbs.)

The README defines prod-system as:

(defn prod-system []
  "Assembles and returns components for an application in production"
  []
    (component/system-map
     :datomic-db (new-datomic-db (env :db-url))
     :mongo-db (new-mongo-db (env :mongo-url))
     :web (new-web-server (env :http-port) handler)
     :repl-server (new-repl-server (env :repl-port))))

There's at least one vector form too many there :) more problematic is this part:

(defn -main
  "Start the application"
  []
  (alter-var-root #'prod-system component/start)

which if I'm not mistaken assumes prod-system is a var containing a system-map, instead of a function that creates a system-map.

The Component README recommends start/stop methods to be idempotent. Many components in system do not have idempotent start/stop methods, see for example HttpKit:

(defrecord WebServer [options server handler]
  component/Lifecycle
  (start [component]
    (let [handler (get-in component [:handler :handler] handler)
          server (run-server handler options)]
      (assoc component :server server)))
  (stop [component]
    (when server
      (server)
      component)))

If I understand correctly start will start multiple servers if called repeatedly and stop won't remove the key from the component causing the httpkit's stop function to be called repeatedly (which it might just tolerate).

@danielsz Is this something you'd like to change or is that in your experience just not very important? (I don't have much experience with component so you probably know better :)

Hi,

I am struggling to see how to use the sentes and http-kit components. Specifically where does my (ring) handler go?

An upgrade to avoid confusion might also be to change the name of the event-msg-handler that the sente component expects - it is currently 'handler' so I naively passed it my ring handler expecting it to add routes for /chsk. A 10 second perusal of the source code made it clear of course, but a different name might be wise.

I am creating the sente component but then I need to pass that to my ring (defroutes ..) handler, but where do I do that? Should I create a synthetic component which wraps my defroutes so it can access the sente component?

Help :)

Can we get a 0.1.0 provisional release in Clojars?

I think this would be helpful, it would help gain exposure and until we have this we can't easily do CI builds etc.

Having a quick peruse through the components, and it seems that you have good stop idempotency.

To do start idempotency, components should check for their stateful part (server, connection, etc.) is already created in the start part of the protocol. https://github.com/stuartsierra/component#idempotence

  (start [this]
    (if connection  ; already started
      this
      (assoc this :connection (connect host port)))

lein test currently fails apparently due to lack of a MongoDB server in my environment.

I think tests should be disabled for components that can't be tested so that having a full environment in not necessary to do a clean build/test.

Unless we do this, it is hard for the different components to be used in an "optional" way.

Please feel free to suggest alternate names.

It's not clear from the documentation how component dependencies are handled. For example, If you have a db connection component, it's not clear how you'd access that connection from a route handler.

Would you mind adding some details to the README, and making the examples a bit more robust to further illustrate (e.g. adding a db component and a single POST route called from the front end)?

The run-jetty command takes a significant number of arguments in it's config, is there any way to support them all? https://ring-clojure.github.io/ring/ring.adapter.jetty.html

We could refactor the validation slightly to allow client control of whether validation takes place, using s/with-fn-validation instead of calling s/validate ourselves

https://github.com/plumatic/schema#schemas-in-practice

That might help if we make a mistake and over-constrain validation somewhere - client could just not validate until the issue is resolved. I might add this feature and update docs before adding new validation to previously unconstrained options

Hi,

In your https://github.com/danielsz/system/blob/master/src/system/repl.clj you use

(require

instead of

(:require

While this did work technically with clojure < 1.9 it is explicitly forbidden now with clojure 1.9.x

When creating a handler with multiple endpoints, there is no way to influence the order in which they will be combined into a single ring/routes.

The current (undocumented) logic is that

• routes with their own middleware come first
  • those that share the same middleware are combined
• routes without route-specific middleware always come last

My use case is this:

(defn prod-system [{:keys [web-port]}] 
  (c/system-map
   :app-routes     (-> (new-endpoint app-routes)
                       (c/using [:middleware]))
   :webhook-routes (new-endpoint webhook-routes) 
   :middleware     (new-middleware {:middleware middleware})
   :handler        (-> (new-handler)
                       (c/using [:webhook-routes :app-routes]))
   :jetty          (-> (new-web-server web-port)
                       (c/using [:handler]))))

Most of the functionality sits in app-routes, and uses the default middleware, but a few routes are related to webhooks, and should sit outside that middleware. These do their own authentication, and so they should not care about any CSRF tokens.

Currently they would always come last (because they don't have middleware), but this means that the middleware from app-routes gets checked first. It notices a POST request without CSRF token, and returns 403. The webhook routes are never reached.

To work around this I hacked up my own version so I can order them by priority

(defn prod-system [{:keys [web-port]}] 
  (c/system-map
   :app-routes     (-> (new-endpoint app-routes)
                       (assoc :priority 20)
                       (c/using [:middleware]))
   :webhook-routes (-> (new-endpoint webhook-routes)
                       (assoc :priority 10)) 
   :middleware     (new-middleware {:middleware middleware})
   :handler        (-> (new-handler)
                       (c/using [:webhook-routes :app-routes]))
   :jetty          (-> (new-web-server web-port)
                       (c/using [:handler]))))

I'd be curious what others think of this use case, and what other approaches there could be to handle it.

https://github.com/danielsz/system/blob/master/src/system/components/http_kit.clj

:join? false 

is extra? not need?

I think it would be useful to have some way to report about the status of the components. This could then plugged into a monitoring infrastructure of some sort.

This feature (provided we agree on consider this request in this way! :) ) could alternatively be added to Sierra’s component library. Seems more appropriate here, though.

HTH

Hey again 👋 Two things that took me a while to figure out when setting everything up:

1 The system var passed to the system task can't be in boot.user otherwise nothing will be reloaded.

If this is correct maybe adding an assert that the namespace of the var isn't boot.user would be good? Happy to provide a PR if wanted.

2 The files option takes filenames and not paths.

Not sure what to do about this, perhaps a warning if the filenames contain slashes? Also a --paths option seems reasonable in that context although not super necessary.

Now that it's all working, wow is it awesome 🚀 🎉

reloaded.repl/set-init! expects a function whereas system.repl/set-init! needs a var. Given that people may be used to reloaded.repl an extra

{:pre [(var? sys)]}

might be warranted.

Adhere to a specific version scheme for better clarity.

https://github.com/ptaoussanis/encore/blob/6dbf8c970eccb6cd33427795f11c714cea128704/BREAK-VERSIONING.md

I'm getting the following error when trying to use Sente:

AssertionError Condition failed intaoensso.sente:?[pred-form, val]: [((fn* [p1__13000#] (satisfies? interfaces/IAsyncNetworkChannelAdapter p1__13000#)) web-server-adapter), {}] taoensso.encore/assertion-error (encore.clj:310)

I have followed the instructions in #14.

Any thoughts/tips?

This is meant to be the discussion for the upcoming changes. Version 0.3.0 of system will incorporate the Duct abstractions: https://github.com/weavejester/duct/wiki/Components
This will be a breaking release for users who were leveraging the app component.

A snapshot release with the proposed changes is available on Clojars.
The example projects have been updated to demonstrate usage.
Additionally, there is a new example demonstrating usage with multiple endpoints:
https://github.com/danielsz/system-duct-style-example
I welcome all comments/criticism/contributions.

Hi, I've been running into a few issues which might warrant some additional input from other folks who've been occupying themselves with system/components.

The problem I think is best described as dependency hell. Although system in this case is a great idea, there is the problem of the way components are referenced currently. That is, being defined in a project.clj key/value pair either :dependencies or under :plugins depending on use (here mostly dependencies).

The case involves the componentization of clj-webdriver. At first thought, it seems suitable for such endeavor: it has a clear start (set-driver! ...) and stop (quit) point, it has state and it requires flexible user input which may change depending on use/time/configuration (e.g. a bundle of test actions predefined to run). Furthermore it seems logical that it depends on some (currently used) components, most notably, a web server (with routes, assets hosted and other things) to be present before it should attempt to do its magic.

The problem comes from the fact that each and every single package conflicts with selenium dependencies. Its a real nightmare of exclusions. This alone would make it a hard sell to componentize in this current suite.

But the problem is not so much limited to this specific case I feel. It seems to me, a logical part of 'breaking things up in reusable pieces' involves a little less 'tweaking dependencies to be just right'.
Ideally, one would imagine a separate running virtual machine in a virtual machine, without inheriting all of the parent project tree. Pulling in dependencies isn't that hard (I use Alembic) and experiment currently a bit with a second protocol applied on top of Lifecycle (named Prerequisite) to feed it (instill [dependencies] ...) before start and stop get to run on the component.

One consequence though, is that - should one to have a pristine environment/dependency tree - any referenced symbols from other components, or basically just any dependency at all, need to be provided explicitly. For example, currently the server component in system uses the non-componentized routes and (middleware/default) wrappers from Compojure and any component can basically use them although, as usual, one would have the include them in the component namespace required symbol list of course.

Anyway I'd be interested in some thoughts, am I overthinking this, not/indeed a problem, is the 'dream' of more isolation feasible, and if so, how might one approach it?

Thanks :)

Here is a short tests log from what is visible that Mongo component dereference db variable, but do not close actual connection.

11:20:32.141 [cluster-ClusterId{value='56cbcfff4d1a1109e5b7819d', description='null'}-127.0.0.1:27017] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:57}] to 127.0.0.1:27017
...
11:20:32.234 [cluster-ClusterId{value='56cbd0004d1a1109e5b7819e', description='null'}-127.0.0.1:27017] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:58}] to 127.0.0.1:27017
...
11:20:32.269 [main] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:59}] to 127.0.0.1:27017
...
11:20:32.435 [cluster-ClusterId{value='56cbd0004d1a1109e5b7819f', description='null'}-127.0.0.1:27017] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:60}] to 127.0.0.1:27017
...
11:20:32.440 [main] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:61}] to 127.0.0.1:27017
...
11:20:32.470 [cluster-ClusterId{value='56cbd0004d1a1109e5b781a0', description='null'}-127.0.0.1:27017] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:62}] to 127.0.0.1:27017
...
11:20:32.476 [main] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:63}] to 127.0.0.1:27017
...
11:20:32.501 [cluster-ClusterId{value='56cbd0004d1a1109e5b781a1', description='null'}-127.0.0.1:27017] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:8, serverValue:64}] to 127.0.0.1:27017
...
11:20:32.505 [main] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:9, serverValue:65}] to 127.0.0.1:27017
...
11:20:32.537 [cluster-ClusterId{value='56cbd0004d1a1109e5b781a2', description='null'}-127.0.0.1:27017] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:10, serverValue:66}] to 127.0.0.1:27017
...
11:20:32.541 [main] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:11, serverValue:67}] to 127.0.0.1:27017

Ran 6 tests containing 27 assertions.
0 failures, 0 errors.

Have you considered splitting this repo in two, one for the tooling aspects (boot/lein) and one for the a la carte components? I want to use the system tasks but having to add exclusions for every dependency is a bit of a pain. :)

Support for the hara.io.scheduler scheduling library.

system.components.jdbc
thank you very much. 👍

Hello,

I'm having some trouble running the ring web app and Postgres components as part of a clojure.test suite . In particular, when I try to output the following in a function that's called as part of database setup code, run from the test suite, I get nils:

  (println "env db: " (env :database-url))
  (println "system: " (:db-spec (:db system)))

This is making me think that the system components are reloading (works in dev, is able to connect to the database in dev, but the test suite isn't getting access to the environ or system components that were set up. I'm not quite sure how to get around this -- if my code doesn't know about system then it's hard to tie the database to the web app (and indeed, I can't even get the environ ENV vars that I'm setting to appear, as seen above.) Maybe I'm just missing something conceptually here, but it seems like this just isn't working for running with clojure.test tests?

Here's my testing task in my build.boot file:

(deftask autotest
  "Run server tests"
  []
  (set-env! :source-paths #{"test" "src"})
  (comp
   (environ :env {:http-port "3000"
                  :database-url "postgres://localhost:5432/foo_test"})
   (watch :verbose true)
   (system :sys #'dev-system
           :hot-reload true
           :auto-start true
           :files ["handler.clj"])
   (test)))

Any help appreciated. Thanks!

The boot task doesn't set up tools.namespace unloading properly, instead it just reloads the affected namespaces manually using data from ns-tracker and require :reload. If I tried to use tools.namespace.repl/refresh manually, my system would be referencing non-existent vars and it would break further eval.

I worked around it locally in my build.boot script with this combo, but this seems like an issue with both the library and documentation.

(defn system-init
  "Has to be dynamic because tns unloads vars"
  []
  ((resolve 'myapp/system)
   {:port 7979}))

(defn setup-tns
  "the 'system' boot task doesn't automatically unload namespaces, so we configure 
tools.namespace here according to https://github.com/boot-clj/boot/wiki/Repl-reloading"
  []
  (apply tns-repl/set-refresh-dirs (get-env :directories)))

(deftask dev
  "Run a non-autoreload system with a REPL"
  []
  (setup-tns)
  (reloaded/set-init! system-init)
  (repl))

This is also a problem with the system task, which calls reloaded.repl itself, which transitively calls tools.namespace. I have to set up tools.namespace dirs and use a dynamic resolve in my init function.

(deftask dev-watch
  "Run a restartable system with a REPL"
  []
  (setup-tns)
  (comp
   (watch :verbose true)
   ;; :files vector will control reloading of the system state
   (system :sys system-init :auto true :files ["handler.clj" "html.clj"])
   ;; Can't launch REPL inline because it'll break watch
   (repl :server true)))

I think the tools.namespace setup can happen in the task, which shouldn't use ns-tracker, and the dynamic resolution hack should be documented somewhere, probably reloaded.repl.

Hey Daniel, I tried to analyze system.boot/system for usage with mount but of course the whole reloaded.repl harness just does not fit the no-system-instance model of mount.

However, @tolitius created this little library for reifing a system in mount: https://github.com/tolitius/yurt.

Just putting this here in order to ponder. A user could create a yurt and pass it to system, another idea is to duplicate the whole task in another mount-only library. I still need to understand which approach is the best...

In any case, good job, good job, good job!

I'm using the watcher component in development to compile SCSS files. If something causes a start to fail during a reset (i.e. syntax error causes refresh to fail), it seems that stop is called twice on the watcher. Unfortunately this throws a NullPointerException.

If you'd like, I'll make a PR to make component/stop on the watcher idempotent. If you think this would be better fixed upstream (i.e. stop-watcher should just be idempotent), let me know and I'll make the case there.

Thanks!

Support for mount and mount-lite

The grand vision is that all the components should have their mount counterparts.

https://github.com/tolitius/mount

https://github.com/aroemers/mount-lite

I had the problem that I used system in a commercial backend with Clojure 1.9 (for spec). Since some of the dependencies in the dev profile did not comply to spec, I could not load my code. I have managed to get around by forking system and putting the deps in a different profile which is not loaded when I start a REPL, and then just providing my chosen dependencies over the regular classpath. Maybe things in the dev profile for system could use :scope "provided" to just be loaded in the dev profile of system itself.

Does this make sense? I can open a pull-request, if you like :).

I'm not entirely sure how a production system could be started,
I want the new-web-server component to start after a datomic-db component.

In this case,
(def conn (:conn (:datomic-db system))
would initialise as nil!

The exceptions is in runtime.
The code compiles correctly. However, if you run (start) you get this error.

Exception in thread "main" java.lang.IllegalArgumentException: No such namespace: , compiling:(/tmp/form-init952470378900861673.clj:1:72)
at clojure.lang.Compiler.load(Compiler.java:7391)
at clojure.lang.Compiler.loadFile(Compiler.java:7317)
at clojure.main$load_script.invokeStatic(main.clj:275)
at clojure.main$init_opt.invokeStatic(main.clj:277)
at clojure.main$init_opt.invoke(main.clj:277)
at clojure.main$initialize.invokeStatic(main.clj:308)
at clojure.main$null_opt.invokeStatic(main.clj:342)
at clojure.main$null_opt.invoke(main.clj:339)
at clojure.main$main.invokeStatic(main.clj:421)
at clojure.main$main.doInvoke(main.clj:384)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.IllegalArgumentException: No such namespace:
at clojure.lang.Var.find(Var.java:141)
at clojure.core$find_var.invokeStatic(core.clj:1928)
at clojure.core$find_var.invoke(core.clj:1923)
at system.repl$init.invokeStatic(repl.clj:18)
at system.repl$init.invoke(repl.clj:15)
at system.repl$start.invokeStatic(repl.clj:23)
at system.repl$start.invoke(repl.clj:20)
at dondeestamigrua.core$_main.invokeStatic(core.clj:19)
at dondeestamigrua.core$_main.doInvoke(core.clj:14)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.Var.invoke(Var.java:375)
at user$eval5.invokeStatic(form-init952470378900861673.clj:1)
at user$eval5.invoke(form-init952470378900861673.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6927)
at clojure.lang.Compiler.eval(Compiler.java:6917)
at clojure.lang.Compiler.load(Compiler.java:7379)
... 14 more

When we modify a method of an interface with a running system, we get:

#'project.sys/dev:refreshing
Unloading: (project.sys project.handler project.api project.search)
Reloading: (project.search project.api project.handler project.sys)
Elapsed time: 0.105 sec

However, when we call any method from that interface we get the following error:
java.lang.IllegalArgumentException: No implementation of method: ...

We need to stop and start the system to make it work again.

In our build.boot we have:

(deftask dev []
  ...
  (system.boot/system :sys #'project.sys/dev
                      :auto true
                      :files ["project/sys.clj"
                              "project/handler.clj"
                              "project/search.clj"])
  ...
)

I don't have an answer for this yet, but I think it would be a good idea to somehow separate the components so that you only pull in the ones that you want. With the current project setup, the only way to do that is to manually build your own jar, or to explicitly exclude the dependencies you don't want in your project's project.clj file.

One possible solution is to use something like lein-sub to manage "sub projects" from a parent project. This would raise questions about version numbers, but one step at a time! Another solution would be to find a way to auto-detect them as Leiningen detects plugins. This may not be ideal, but certainly possible.

I got caught by surprise that set-init! isn't called when not using the :auto option. Is there any specific reasoning behind this? Maybe we could move the set-init call outside of the (when auto ...) check?

https://github.com/danielsz/system/blob/master/src/system/boot.clj#L55-L71

#24 and #29 indicate that there is some confusion as to how components should communicate with each other (via the system map). A full CRUD example may help bring clarity to the table.

The following example by the CDS project nicely fits the bill.
http://clojure-doc.org/articles/tutorials/basic_web_development.html