aantron / dream Goto Github PK
View Code? Open in Web Editor NEWTidy, feature-complete Web framework
Home Page: https://aantron.github.io/dream/
License: MIT License
Tidy, feature-complete Web framework
Home Page: https://aantron.github.io/dream/
License: MIT License
This requires (probably) patching http/af and h2. At the moment, it is possible to open connections and never send any data. The connections remain open. Such connections never even reach Dream, but linger in the lower-level HTTP libraries indefinitely. The timeout should probably cover the entire span from when the lower-level server first open the connection, to when it passes a request to Dream, to also catch trickling of data.
As soon as I submit something simple like 'a':
~/example/d-form % dune exec ./form.exe
14.05.21 15:55:15.327 Running on http://localhost:8080
14.05.21 15:55:15.327 Press ENTER to stop
14.05.21 15:55:23.131 dream.log INFO REQ 1 GET / 127.0.0.1:41998 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
14.05.21 15:55:23.131 dream.log INFO REQ 1 200 in 152 μs
14.05.21 15:55:25.484 dream.log INFO REQ 2 POST / 127.0.0.1:41998 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
14.05.21 15:55:25.484 dream.log WARN REQ 2 Aborted by: Invalid_argument("Cstruct.blit_from_string src=[81] dst=[0,81](81) src-off=13 len=81")
14.05.21 15:55:25.484 dream.http ERROR Invalid_argument("Cstruct.blit_from_string src=[81] dst=[0,81](81) src-off=13 len=81")
14.05.21 15:55:25.484 dream.http ERROR Raised at file "stdlib.ml", line 30, characters 20-45
14.05.21 15:55:25.484 dream.http ERROR Called from file "lib/cstruct.ml", line 407, characters 4-30
14.05.21 15:55:25.484 dream.http ERROR Called from file "src/cipher/cipher.ml", line 108, characters 12-50
14.05.21 15:55:25.484 dream.http ERROR Called from file "src/cipher/cipher.ml", line 38, characters 10-60
14.05.21 15:55:25.484 dream.http ERROR Called from file "src/middleware/csrf.ml", line 44, characters 8-63
14.05.21 15:55:25.484 dream.http ERROR Called from file "src/core/lwt.ml", line 2133, characters 16-20
14.05.21 15:55:25.484 dream.http ERROR Re-raised at file "src/middleware/session.ml", line 29, characters 2-91
This is with dream 1.0.0~alpha1 and master too.
http://dream.as/d-form works though
Request bodies are already mutable, because of the use of one-time callbacks/promises (rather than buffering streams). There are hidden mutable fields in requests for internal purposes, and there might be future hidden mutable caches.
Making requests fully mutable would simplify some of Dream's code, and save on allocations during immutable updates (but this can be mitigated in some other ways, as well).
I'm having a hard time seeing the use case for immutable requests — i.e. I don't think a Web app really ever needs to share them along different "execution paths."
Dream hasn't committed either way yet. It just provides an immutable-looking request API with a partially-mutable implementation.
Emit default CSP headers with:
Dream.html
Dream.static
when returning static HTML pages.frame-ancestors
It's probably best to:
Dream.html
.Some of these are already used internally by Dream, or are variants of what is already used:
See Cryptography.
These are probably best provided by separate libraries, as some of them might undergo changes as the GraphQL community develops. EDIT: It's also possible to implement simple things like batching in Dream, but note clearly that they may be experimental.
@stream
, @defer
There appears to be a gluten dependency in the e-json
example project that's not easily resolved. Trying to build results in the following:
npx esy
info esy 0.6.10 (using dream.opam) found project at ~/src/opensource/dream/
File "src/vendor/dune", line 39, characters 11-26:
39 | (modules gluten_lwt_unix tls_io ssl_io)))
^^^^^^^^^^^^^^^
Error: Module Gluten_lwt_unix doesn't exist.
error: command failed: 'dune' 'build' '-p' 'dream' '-j' '4' (exited with 1)
esy-build-package: exiting with errors above...
error: build failed with exit code: 1
esy: exiting due to errors above
I naturally tried to install the module but it has a conflict:
opam install gluten-lwt-unix
The following actions will be performed:
⊘ remove dream 1.0.0~alpha1 [conflicts with gluten]
∗ install gluten 0.2.1 [required by gluten-lwt-unix]
∗ install gluten-lwt 0.2.1 [required by gluten-lwt-unix]
∗ install gluten-lwt-unix 0.2.1
===== ∗ 3 ⊘ 1 =====
Do you want to continue? [Y/n]
Is there a good way to work through this?
Flash messages are a common way to display success or error notifications to users in server-side rendered applications.
Dream already has a middleware to support sessions, which Flash messages are a specialization of, so it would be great to have a middleware and a set of API to support flash messages.
Additionnally, helpers to add flash messages in Eml templates would be really useful. For reference, here's Phoenix documentation on this: https://hexdocs.pm/phoenix/controllers.html#flash-messages. The templates look like this:
<p class="alert alert-info" role="alert"><%= get_flash(@conn, :info) %></p>
<p class="alert alert-danger" role="alert"><%= get_flash(@conn, :error) %></p>
Thanks for writing this, it's very exciting!
It would be useful to have a middleware to strip trailing slashes, so you can define one handler for /foo/bar
and /foo/bar/
. I can't see how to write one with the currently exposed functions for request
s. Maybe more general functionality for rewriting paths would also be good.
I tried to follow the quick start install instructions. didn't work :(:
ilja@meatdripper:~/dream/example/2-middleware$ npm install esy && npx esy
[email protected] postinstall /home/ilja/dream/example/2-middleware/node_modules/esy
node -e "process.env['OCAML_VERSION'] = process.platform == 'linux' ? '4.10.1002-musl.static.flambda': '4.10.0'; process.env['OCAML_PKG_NAME'] = 'ocaml'; require('./postinstall.js')"
importing: esy-2c95689b
Done!
npm WARN saveError ENOENT: no such file or directory, open '/home/ilja/dream/example/2-middleware/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/home/ilja/dream/example/2-middleware/package.json'
npm WARN 2-middleware No description
npm WARN 2-middleware No repository field.
npm WARN 2-middleware No README data
npm WARN 2-middleware No license field.
info esy 0.6.10 (using esy.json)
info checking https://github.com/ocaml/opam-repository for updates...
info checking https://github.com/esy-ocaml/esy-opam-override for updates...
info resolving esy packages: done
info solving esy constraints: done
info resolving npm packages: done
info fetching: done
.... installing @opam/bigarray-overlap@opam:0.2.0@d8a38f4aesy: internal error, uncaught exception:
Unix.Unix_error(Unix.EACCES, "rename", "/home/ilja/.esy/source/s/yarn_pkg_config__9829fc81")
Raised at file "src/core/lwt.ml", line 2999, characters 20-29
Called from file "src/unix/lwt_main.ml", line 26, characters 8-18
Called from file "esy-lib/Cli.re", line 264, characters 9-28
Called from file "cmdliner_term.ml", line 25, characters 19-24
Called from file "cmdliner.ml", line 25, characters 27-34
Called from file "cmdliner.ml", line 116, characters 32-39
.... installing @opam/cstruct@opam:6.0.0@c77fb059ilja@meatdripper:~/dream/example/2-middleware$
...which would allow opting in to, one by one, trusting:
X-Forwarded-Host
, Forwarded
— likely needed for Dream.origin_referer_check
AJAX CSRF protection.X-Forwarded-For
X-Forwarded-Proto
— this is needed for correctly inferring cookie Secure
attributes when working behind a proxy.Dream already avoids leaking the id on its own, providing a separate label (Dream.session_label
) for identifying the session in logs and elsewhere. Furthermore, only the label makes sense for long-term session identification, in the face of future implementation of session re-keying (#13).
If the log level is set to `Debug
globally with Dream.initialize_log ~level:`Debug ()
, Caqti debug-level messages show all issued statements and their parameters — including the real session ids.
I'm not sure yet whether this should be solved in Caqti somehow, by Dream disabling logging during sensitive queries, a combination, or otherwise.
Hopefully, this isn't an immediate concern, with users not running critical apps with log level `Debug
(default level is `Info
). But it should be solved soon.
Edit: created paurkedal/ocaml-caqti#70.
See Logging.
This template will probably have to be replaced, because GraphiQL will be way too large for a practical template when bundled:
dream/src/graphql/graphiql.eml.html
Lines 11 to 24 in 0f2ffe3
Various build instructions for GraphiQL are in its repo.
Notes:
Also:
/renderExample.js
.Dream.graphql
presently does not implement streaming responses at all, arguably a bug that should be addressed in the first or one of the first releasesi-graphql
and the API docs.Also:
Hi, I'm a bit interested to use dream
with MirageOS. However, with a quick look, dream
needs to use something else than gluten
to be compatible with MirageOS (for instance, mimic). That mostly mean a deep (but not so hard and long) work on the stack.
So to preserve our times, I would like to know if you are interested about that too 👍. If it's the case, I can try to minimize as much as I can patches to be compatible with MirageOS and propose something which will surely change your dependencies graph.
I would like to say that, from my quick look, dream
seems designed as a library which does not really use Unix
stuffs (expect for the entropy) which gives to us an opportunity to be compatible with MirageOS.
Would a Phoenix LiveView-like feature best be a part of the dream framework or live as a plug-in or library?
To those unknown, LiveView is the Elixir Phoenix library that allows servers to dynamically render HTML and serve it over websockets.
A few other frameworks exist that are beginning to do the same: Ruby Hotwire and Haskell IHP.
Another option may be to create an HTMX-compatible dream middleware much like how Sihl is doing so for a more dynamic UX without heavy JS.
Is the GitHub discussion available for this repo? This isn’t really an issue, but more a discussion.
Do you have any tips on how to deploy a dream web server for real, on the web, easily?
Do you personally use AWS? google cloud? Or something like Linode? How do you scale it?
Seems like we also need to run:
git pull --recurse-submodules
The middlewares are internally quite decoupled, but then configured with defaults that make some of them rely on others. Add:
Dream.form
, Dream.multipart
~csrf:false
or ~csrf:my_function
for no or custom CSRF checking, which removes the dependency of the form reader on sessions (because the default CSRF checker uses tokens that are bound to sessions).Dream.sql_sessions
~sql:connection
for using a different database connection other than the one supplied by Dream.sql_pool
. This is probably a quite common scenario. This part probably depends on the SQL rework so the good first issue label is not about thisIt should be easy to define a describe : route list -> [some format]
function. SInce a Dream app would be mostly composed using a route list
, that should cover most use cases of introspection.
Introspecting middlewares could be done with a composition operator that expects both a middleware and a name. However, I'm not aware of a use case for this.
...which sometimes clobbers exception logging output, especially since it itself sometimes raises exceptions as part of its control flow.
Don't have an example ready ATM as I last observed this before I was using issues in the repo, will update later.
In general, look through Dream and deps for exception-based control flow and get rid of as much of it as possible.
EDIT: it may be possible to work around this by saving the backtrace before calling Printexc.to_string
.
Hi, I saw Dune on Lobste.rs and the design looks real good!
I tried to run in my machine with no success.
My project consists of:
(executable
(name main)
(libraries dream))
(lang dune 2.0)
let () =
Dream.run (fun _ ->
Dream.respond "Good morning, world!")
When I do dune build
, I get this error:
Entering directory '/Users/rodrigo/dream'
File "dream/src/vendor/h2/lib/settings.ml", line 284, characters 7-28:
284 | Angstrom.parse_string
^^^^^^^^^^^^^^^^^^^^^
Error: This function has type 'a Angstrom.t -> string -> ('a, string) result
It is applied to too many arguments; maybe you forgot a `;'.
my opam is 2.0.8, ocaml is 4.10.2, running on windows
also, opam install dream
does not work for me:
opam install dream
[ERROR] No package named dream found.
Can anyone help me? Thanks in advance.
Dream.html
.<pre><span>
markup in the API docs.8080
.is_fin:true
upstream.set_session
should probably be put_session
. a268400session_id
is too easy to confuse with session_key
, because most of the rest of the world considers the key and the id to be the same thing.
session_tag
session_label
`HTTP2
necessary as a separate error constructorDream.static
?handler
should probably be ?loader
.Dream.graphql
context-making callback probably needs a nameProbably at least the per-query information that has been parsed:
It would be really useful to have a helper val with_etag : request -> response -> response
(or similar) that computes a hash of the response body and compare it with the request's Etag
. It would transform the response to a 304 Not modified
if they match, or add the Etag
header with the computed hash if they don't
I know that I come like someone who want to push his projects but I really think that you probably want to use multipart_form
instead of multipart-form-data
when the latter does not stream contents - which limits your usage.
If you are interested by that , I can propose a PR if you want 👍.
session_key
to session_id
, because ID is the common term for it, and the term key collides with usage in cryptographic key, for example keys in TLS. OTOH the session key is generated in about the same way, using a CSRNG, so maybe this is fine. NIST uses session secret, but session identifier remains in one place. `Info
.Some combination of:
Dream.multipart
.Dream.body
, Dream.form
, etc., for local, per-call site control.Should also check HTTP header length limits implemented by http/af and h2.
See ocaml-ppx/ppxlib#221.
Dream has a cipher rotation scheme, so this should be transparent to users.
For example, should the prefixed cookie name be signed together with the value?
package-lock.json
.I get an SSL error under the below settings:
running 6-echo server with ~https:true
run ab
command with 2 or more concurrency level
$ ab -n 2 -c 2 -f TLS1.3 -T 'application/text' -p <(echo hi) https://localhost:8080/echo
error log:
15.04.21 00:55:12.132 dream.log INFO REQ 1109 POST /echo 127.0.0.1:35012 ApacheBench/2.3
15.04.21 00:55:12.132 dream.log INFO REQ 1109 200 in 40 μs
15.04.21 00:55:12.137 dream.log INFO REQ 1110 POST /echo 127.0.0.1:35014 ApacheBench/2.3
15.04.21 00:55:12.137 dream.log INFO REQ 1110 200 in 41 μs
15.04.21 00:55:12.138 dream.http ERROR HTTP (127.0.0.1:35016): SSL read() error: error:00000000:lib(0):func(0):reason(0)
15.04.21 00:55:12.138 dream.http ERROR Raised by primitive operation at Lwt_unix.shutdown in file "src/unix/lwt_unix.cppo.ml", line 1677, characters 2-38
15.04.21 00:55:12.138 dream.http ERROR Called from Lwt.Miscellaneous.wrap2 in file "src/core/lwt.ml", line 3087, characters 15-24
ETag:
.See Dream.static
?handler
(rename to ?loader
?).
Dream's API is already written as if there is backpressure (so there will be no change for users), however backpressure is not actually implemented. It is complicated by the underlying APIs. So, implement it for...
The workaround:
Lines 144 to 155 in 882a48e
is waiting for mirage/ocaml-uri#156. The workaround is itself buggy when used with other packages that also use ocaml-uri. It only avoids triggering the bug in interactions within Dream.
When the server gets Expect: 100-continue
from the client, it should be able to send a 100 Continue response and go on processing the request, eventually sending the "real" response.
AFAIK, supporting this will require patching http/af. The Dream API will look something like
Dream.send_response : request -> response -> unit promise
...where response
will then be Dream.response ~status:`Continue ""
or Dream.response ~code:100 ""
.
cc @anuragsoni
Of course, everything should be measured first.
Some early notes from a few weeks ago:
- request-id seems to take ~3 us.
- catch ~1 us or less.
- in-memory sessions 200 us on miss, 100 us on hit.
- Initial form parser 9 us on tiny input; 30 us on larger, more realistic input
- jwto CSRF tokens maybe 50 us.
It's probably enough to just ask people to press Ctrl+C to exit (it already works). As for the 1-second grace delay, it's probably unnecessary for starter code — but needed in robust Web apps, later in development. However, a robust Web app is already probably at least a medium-sized project that can afford to, and needs to, implement its own exit logic. That means the default grace delay is likely redundant in all cases.
cc @Lupus
Not supporting this is too burdensome, as it requires some kind of multiplexing abstraction to be built by the user (in the server) to support concurrent (interleaved) writing.
Flow control is still achieved by each writer waiting on the promises returned by Dream.write
and Dream.send
, so that using only one writer will still not trigger any buffering or concurrency. In other words, making this change will not affect the one-writer code that was possible before the change, and will only allow a new way of using the API.
Flow control is also likely to limit memory consumption and GC thrashing. AFAICT Dream is currently causing internal buffers to grow considerably. While writing large enough streams, it probably even exhausts the working set, causing page thrashing. These are unconfirmed hypotheses, but they likely explain the slowdown when sending enormous streams (>> 8 GB to 8 clients, for example). See example w-stress-response
.
Hi @aantron!
Thanks a lot for your amazing work on Dream, I'm really excited to see a full-featured OCaml web server and can't wait to try it for a side project 😄
I wanted to take the pulse and see how you felt about using Rock (Opium's low-level HTTP and middleware layer) in Dream.
To give a bit of context, we extracted Rock from Opium and made it as minimalist as possible in the hope that it could be re-used by other Web frameworks in the community.
Our motivation was to provide a common middleware layer to all of the frameworks so that users and frameworks could re-use them. (In the spirit of Rack from Ruby)
If that seems to make sense to you, there would be some work needed on our side, as Rock does not support Http2 and Web sockets at the moment, but that's things we wanted to work on anyways.
I've also seen #4 and that's something I wanted to explore as well. All in all, there seem to be some overlaps and I'd love to see a collaboration happen to consolidate OCaml's web ecosystem.
Cheers and thanks again for all your work!
dune build @fmt --watch
yields
ocamlformat: ignoring "src/bin/server/bin.eml.ml" (syntax error)
when using eml
do you use ocamlformat
? if so, how? perhaps https://dune.readthedocs.io/en/stable/dune-files.html#dialect is relevant?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.