Giter VIP home page Giter VIP logo

dream's People

Contributors

aantron avatar acieroid avatar artileda avatar benjamin-thomas avatar dangdennis avatar denver-s avatar dinosaure avatar hannesm avatar jsthomas avatar kit-ty-kate avatar lessp avatar mooreryan avatar oteku avatar outkine avatar paurkedal avatar persianturtle avatar pm-mck avatar rawleyfowler avatar renatillas avatar samoht avatar shawn-mcginty avatar tcoopman avatar thangngoc89 avatar thelortex avatar tmattio avatar tmcgilchrist avatar trefis avatar ulrikstrid avatar xvw avatar yawaramin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dream's Issues

Request timeouts

This requires (probably) patching http/af and h2. At the moment, it is possible to open connections and never send any data. The connections remain open. Such connections never even reach Dream, but linger in the lower-level HTTP libraries indefinitely. The timeout should probably cover the entire span from when the lower-level server first open the connection, to when it passes a request to Dream, to also catch trickling of data.

  • Ability to simply drop requests?

d-form fails with a Cstruct.blit_from_string error

As soon as I submit something simple like 'a':

~/example/d-form % dune exec ./form.exe
14.05.21 15:55:15.327                       Running on http://localhost:8080
14.05.21 15:55:15.327                       Press ENTER to stop
14.05.21 15:55:23.131       dream.log  INFO REQ 1 GET / 127.0.0.1:41998 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
14.05.21 15:55:23.131       dream.log  INFO REQ 1 200 in 152 μs
14.05.21 15:55:25.484       dream.log  INFO REQ 2 POST / 127.0.0.1:41998 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
14.05.21 15:55:25.484       dream.log  WARN REQ 2 Aborted by: Invalid_argument("Cstruct.blit_from_string src=[81] dst=[0,81](81) src-off=13 len=81")
14.05.21 15:55:25.484      dream.http ERROR Invalid_argument("Cstruct.blit_from_string src=[81] dst=[0,81](81) src-off=13 len=81")
14.05.21 15:55:25.484      dream.http ERROR Raised at file "stdlib.ml", line 30, characters 20-45
14.05.21 15:55:25.484      dream.http ERROR Called from file "lib/cstruct.ml", line 407, characters 4-30
14.05.21 15:55:25.484      dream.http ERROR Called from file "src/cipher/cipher.ml", line 108, characters 12-50
14.05.21 15:55:25.484      dream.http ERROR Called from file "src/cipher/cipher.ml", line 38, characters 10-60
14.05.21 15:55:25.484      dream.http ERROR Called from file "src/middleware/csrf.ml", line 44, characters 8-63
14.05.21 15:55:25.484      dream.http ERROR Called from file "src/core/lwt.ml", line 2133, characters 16-20
14.05.21 15:55:25.484      dream.http ERROR Re-raised at file "src/middleware/session.ml", line 29, characters 2-91

This is with dream 1.0.0~alpha1 and master too.
http://dream.as/d-form works though

Should requests be mutable or immutable?

Request bodies are already mutable, because of the use of one-time callbacks/promises (rather than buffering streams). There are hidden mutable fields in requests for internal purposes, and there might be future hidden mutable caches.

Making requests fully mutable would simplify some of Dream's code, and save on allocations during immutable updates (but this can be mitigated in some other ways, as well).

I'm having a hard time seeing the use case for immutable requests — i.e. I don't think a Web app really ever needs to share them along different "execution paths."

Dream hasn't committed either way yet. It just provides an immutable-looking request API with a partially-mutable implementation.

Default Content Security Policy

Emit default CSP headers with:


  • In particular, include a policy for frames, to mitigate clickjacking by default.
  • Add a handler for logging CSP violation reports.
  • Document everything. Link to MDN and offer basic warnings and guidance. Create a CSP tutorial or example.


It's probably best to:

  • Provide an example that shows CSP in action, as well as reporting.
  • Link to the example from Dream.html.

Add FIRE to the logs

fire-logs

Of course, this should be optional and rare.

Show in the logger example how to enable it.

Advanced GraphQL extensions

These are probably best provided by separate libraries, as some of them might undergo changes as the GraphQL community develops. EDIT: It's also possible to implement simple things like batching in Dream, but note clearly that they may be experimental.

Web protocols (h2c, HTTP/3, HTTP/2 server push)

  • h2c (HTTP/2 without HTTPS) is deliberately not supported by browsers. However, it may be useful for some command-line clients.
  • HTTP/3 is currently experimental. It runs over UDP, but this is easy to suppor with Dream's ability to share an application context between multiple interfaces (this was hidden from the API during development, and will be re-exposed).
  • HTTP/3 WebTransport.
  • HTTP/2 server push was considered for removal from Chrome. There are replacement mechanisms/improvements being discussed.

JSON example build issues

There appears to be a gluten dependency in the e-json example project that's not easily resolved. Trying to build results in the following:

npx esy            
info esy 0.6.10 (using dream.opam) found project at ~/src/opensource/dream/
File "src/vendor/dune", line 39, characters 11-26:
39 |   (modules gluten_lwt_unix tls_io ssl_io)))
                ^^^^^^^^^^^^^^^
Error: Module Gluten_lwt_unix doesn't exist.
error: command failed: 'dune' 'build' '-p' 'dream' '-j' '4' (exited with 1)
esy-build-package: exiting with errors above...
error: build failed with exit code: 1
  
esy: exiting due to errors above

I naturally tried to install the module but it has a conflict:

opam install gluten-lwt-unix
The following actions will be performed:
  ⊘ remove  dream           1.0.0~alpha1 [conflicts with gluten]
  ∗ install gluten          0.2.1        [required by gluten-lwt-unix]
  ∗ install gluten-lwt      0.2.1        [required by gluten-lwt-unix]
  ∗ install gluten-lwt-unix 0.2.1
===== ∗ 3   ⊘ 1 =====
Do you want to continue? [Y/n] 

Is there a good way to work through this?

Support flash messages [integrate #62]

Flash messages are a common way to display success or error notifications to users in server-side rendered applications.
Dream already has a middleware to support sessions, which Flash messages are a specialization of, so it would be great to have a middleware and a set of API to support flash messages.

Additionnally, helpers to add flash messages in Eml templates would be really useful. For reference, here's Phoenix documentation on this: https://hexdocs.pm/phoenix/controllers.html#flash-messages. The templates look like this:

<p class="alert alert-info" role="alert"><%= get_flash(@conn, :info) %></p>
<p class="alert alert-danger" role="alert"><%= get_flash(@conn, :error) %></p>

Middleware to strip trailing slashes

Thanks for writing this, it's very exciting!

It would be useful to have a middleware to strip trailing slashes, so you can define one handler for /foo/bar and /foo/bar/. I can't see how to write one with the currently exposed functions for requests. Maybe more general functionality for rewriting paths would also be good.

install breaks

I tried to follow the quick start install instructions. didn't work :(:
ilja@meatdripper:~/dream/example/2-middleware$ npm install esy && npx esy

[email protected] postinstall /home/ilja/dream/example/2-middleware/node_modules/esy
node -e "process.env['OCAML_VERSION'] = process.platform == 'linux' ? '4.10.1002-musl.static.flambda': '4.10.0'; process.env['OCAML_PKG_NAME'] = 'ocaml'; require('./postinstall.js')"

importing: esy-2c95689b
Done!
npm WARN saveError ENOENT: no such file or directory, open '/home/ilja/dream/example/2-middleware/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/home/ilja/dream/example/2-middleware/package.json'
npm WARN 2-middleware No description
npm WARN 2-middleware No repository field.
npm WARN 2-middleware No README data
npm WARN 2-middleware No license field.

  • [email protected]
    added 1 package and audited 1 package in 4.082s
    found 0 vulnerabilities

info esy 0.6.10 (using esy.json)
info checking https://github.com/ocaml/opam-repository for updates...
info checking https://github.com/esy-ocaml/esy-opam-override for updates...
info resolving esy packages: done
info solving esy constraints: done
info resolving npm packages: done
info fetching: done
.... installing @opam/bigarray-overlap@opam:0.2.0@d8a38f4aesy: internal error, uncaught exception:
Unix.Unix_error(Unix.EACCES, "rename", "/home/ilja/.esy/source/s/yarn_pkg_config__9829fc81")
Raised at file "src/core/lwt.ml", line 2999, characters 20-29
Called from file "src/unix/lwt_main.ml", line 26, characters 8-18
Called from file "esy-lib/Cli.re", line 264, characters 9-28
Called from file "cmdliner_term.ml", line 25, characters 19-24
Called from file "cmdliner.ml", line 25, characters 27-34
Called from file "cmdliner.ml", line 116, characters 32-39
.... installing @opam/cstruct@opam:6.0.0@c77fb059ilja@meatdripper:~/dream/example/2-middleware$

At log level Debug, with sql_sessions, Caqti will leak session ids into the logs

Dream already avoids leaking the id on its own, providing a separate label (Dream.session_label) for identifying the session in logs and elsewhere. Furthermore, only the label makes sense for long-term session identification, in the face of future implementation of session re-keying (#13).

If the log level is set to `Debug globally with Dream.initialize_log ~level:`Debug (), Caqti debug-level messages show all issued statements and their parameters — including the real session ids.

Снимок

I'm not sure yet whether this should be solved in Caqti somehow, by Dream disabling logging during sensitive queries, a combination, or otherwise.

Hopefully, this isn't an immediate concern, with users not running critical apps with log level `Debug (default level is `Info). But it should be solved soon.

Edit: created paurkedal/ocaml-caqti#70.

Logging improvements

  • Setting different levels for sub-logs.
  • Neat multi-line logging.
  • Log JSON, or write the whole log as JSON? This depends on JSON improvements.
  • Log-anything.
  • Document how to log to file or in real deployments.

See Logging.

Bundle GraphiQL into one HTML file to avoid CDN usage

This template will probably have to be replaced, because GraphiQL will be way too large for a practical template when bundled:

(* TODO Bundle rather than use CDNs. *)
let graphiql graphql_path _request =
Dream.respond @@
<!--
* Copyright (c) 2021 GraphQL Contributors
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE file in the root directory of this source tree.
-->
<!DOCTYPE html>

Various build instructions for GraphiQL are in its repo.

Notes:

Also:

  • Suppress requests to /renderExample.js.

GraphQL subscriptions

  • Dream.graphql presently does not implement streaming responses at all, arguably a bug that should be addressed in the first or one of the first releases. These are subscriptions.
  • Implement GraphQL subscriptions using protocol graphql-ws.

  • A subscriptions example, hyperlinked from i-graphql and the API docs.
  • Client verification for WebSocket upgrade GET requests. Done by ordinary authentication checks, probably — there is no CSRF possibility with WebSockets AFAIK.
  • Proper error messages sent back to the client.
  • Format of the variables JSON. Are enums possible? Input coercion here suggests that they are represented as strings in JSON, so nothing needs to be done.
  • In case of duplicate operation ids, should the connection be terminated, or should the existing id be replaced? The whole connection should be closed.
  • Should more information be passed to the context-making function, in particular the operation name, variables map, and/or query itself? Separated out into #35.
  • Probably later, upon #12: be prepared to stream large responses. Might also need settings for incoming message size limits. Moved to #12.
  • Update docs.
  • Update GraphiQL to be compatible with this; default CDN distribution does not support subscriptions at all. See also #5, which can be solved concurrently.

Also:

  • Connection setup timeout. This appears to be tied to the graphql-ws implementation. Hopefully it is prevented by authenticating GraphQL subscription requests. If not, it can easily be added later.

MirageOS support?

Hi, I'm a bit interested to use dream with MirageOS. However, with a quick look, dream needs to use something else than gluten to be compatible with MirageOS (for instance, mimic). That mostly mean a deep (but not so hard and long) work on the stack.

So to preserve our times, I would like to know if you are interested about that too 👍. If it's the case, I can try to minimize as much as I can patches to be compatible with MirageOS and propose something which will surely change your dependencies graph.

I would like to say that, from my quick look, dream seems designed as a library which does not really use Unix stuffs (expect for the entropy) which gives to us an opportunity to be compatible with MirageOS.

Phoenix LiveView-like feature inside Dream

Would a Phoenix LiveView-like feature best be a part of the dream framework or live as a plug-in or library?

To those unknown, LiveView is the Elixir Phoenix library that allows servers to dynamically render HTML and serve it over websockets.

A few other frameworks exist that are beginning to do the same: Ruby Hotwire and Haskell IHP.

Another option may be to create an HTMX-compatible dream middleware much like how Sihl is doing so for a more dynamic UX without heavy JS.

Is the GitHub discussion available for this repo? This isn’t really an issue, but more a discussion.

More optionals for decoupling middlewares

The middlewares are internally quite decoupled, but then configured with defaults that make some of them rely on others. Add:

  • Add Dream.form, Dream.multipart ~csrf:false or ~csrf:my_function for no or custom CSRF checking, which removes the dependency of the form reader on sessions (because the default CSRF checker uses tokens that are bound to sessions).
  • Add Dream.sql_sessions ~sql:connection for using a different database connection other than the one supplied by Dream.sql_pool. This is probably a quite common scenario. This part probably depends on the SQL rework so the good first issue label is not about this

Introspection

It should be easy to define a describe : route list -> [some format] function. SInce a Dream app would be mostly composed using a route list, that should cover most use cases of introspection.

Introspecting middlewares could be done with a composition operator that expects both a middleware and a name. However, I'm not aware of a use case for this.

Find and eliminate upstream S-expression exception printer

...which sometimes clobbers exception logging output, especially since it itself sometimes raises exceptions as part of its control flow.

Don't have an example ready ATM as I last observed this before I was using issues in the repo, will update later.

  • In general, look through Dream and deps for exception-based control flow and get rid of as much of it as possible.

  • EDIT: it may be possible to work around this by saving the backtrace before calling Printexc.to_string.

Trouble building a dream project with dune

Hi, I saw Dune on Lobste.rs and the design looks real good!

I tried to run in my machine with no success.

My project consists of:

  • dune
(executable
   (name main)
   (libraries dream))

  • dune-project
(lang dune 2.0)
  • main.ml
let () =
  Dream.run (fun _ ->
    Dream.respond "Good morning, world!")

When I do dune build, I get this error:

Entering directory '/Users/rodrigo/dream'
File "dream/src/vendor/h2/lib/settings.ml", line 284, characters 7-28:
284 |        Angstrom.parse_string
             ^^^^^^^^^^^^^^^^^^^^^
Error: This function has type 'a Angstrom.t -> string -> ('a, string) result
       It is applied to too many arguments; maybe you forgot a `;'.

my opam is 2.0.8, ocaml is 4.10.2, running on windows

also, opam install dream does not work for me:

opam install dream
[ERROR] No package named dream found.

Can anyone help me? Thanks in advance.

Optimize the docs site

  • Probably bundle resources into one HTML file, because a lot of load time is currently spent queued to issue requests for resources. I've used inliner for this in GraphiQL and the Playground, but it has a bug with parsing <pre><span> markup in the API docs.
  • Subset fonts, especially Iosevka. Iosevka weighs 500K, half the total unoptimized site right now. There are Iosevka custom builds and separate font subset tools.
    • When subsetting Iosevka, consider switching to a dotted zero, because slashed zero is difficult to parse in 8080.
  • Host/bundle the remaining remote font, Tenor Sans (or drop it).
  • Run highlight.js offline.

Additional streaming APIs

  • There is already file upload string streaming exposed. Dream may need to expose bigstring streaming for zero-copy uploads.
  • WebSocket frame-by-frame chunk-by-chunk streaming of large messages. Probably only a bigstring API for this. Done in #178, though not fully exposed in the API yet.
    • Streaming out messages is currently not possible, because frames are hardcoded with is_fin:true upstream.
  • Stream limits and handling in the GraphQL helpers.
  • Upload stream error handling currently raises exception on bad Content-Type.
  • iovec conversion.
  • Stream static files.

Renaming APIs

  • set_session should probably be put_session. a268400
  • session_id is too easy to confuse with session_key, because most of the rest of the world considers the key and the id to be the same thing.
    • session_tag? session_label
  • Is `HTTP2 necessary as a separate error constructor?
  • Dream.static ?handler should probably be ?loader.
  • Dream.graphql context-making callback probably needs a name. Solved with docs, a332af1.

Provide a helper to add Etag headers to responses

It would be really useful to have a helper val with_etag : request -> response -> response (or similar) that computes a hash of the response body and compare it with the request's Etag. It would transform the response to a 304 Not modified if they match, or add the Etag header with the computed hash if they don't

Session improvements: re-keying, signing, laziness, GC, etc.

  • Session re-keying
  • Sign session tokens to reduce database access by random client-generated tokens.
  • Version the token format for transparent upgrades of the session token scheme.
  • Garbage collection.
  • Lazy sessions. There is no need to persist state for a pre-session that hasn't had any data associated with it, except to reserve the key. Is that important? This can be addressed by making the session key getter evaluate to a promise. However, that would infect the CSRF generator, template helper, etc. It may be better for it to always return a promise, since then one could implement reads by always reading from persistent storage. However, at that point, someone may want a custom session manager, anyway. There is no need for duplicate checking, as a ~144-bit session ID collision, assuming a good RNG, is not worth protecting against, even for a service with billions of monthly visits, assuming each one generates a fresh session, and all the sessions are concurrent. A service with stricter requirements still, should be easily able to implement a checking session id generator.
  • Optionally preserve session data across invalidation.
  • Session reload from persistent storage for synchronization? Do we need data generation numbers, etc., to detect concurrent modifications? Or is this best left to custom session managers?
  • Expose the session back end-making interface.
  • Cross-invalidation, e.g. on login from another device.
  • Make sure client IP "locking" is easy to implement. Create an example.
  • Consider key length carefully.
  • Rename session_key to session_id, because ID is the common term for it, and the term key collides with usage in cryptographic key, for example keys in TLS. OTOH the session key is generated in about the same way, using a CSRNG, so maybe this is fine. NIST uses session secret, but session identifier remains in one place.
  • Is it safe to just print the session ID to logs? No. OWASP recommends against this. Dream already implements a separate log-safe identifier, which should probably become a hash so that there is at least a 1-way correlation.
  • Should the session ID be hashed when storing in a database?
  • Log session lookups, etc., at level `Info.
  • Derive the label from the key by hashing; in fact, this can just be what is stored in the database.
  • Remember me as an option, i.e. don't use a 2-week lifetime by default.

Request length limits

Some combination of:

  • A middleware that filters body chunks, accumulating their total length, and aborting responses if the length exceeds some limit.
  • File size and count limits for Dream.multipart.
  • Similar limit helpers on Dream.body, Dream.form, etc., for local, per-call site control.

Should also check HTTP header length limits implemented by http/af and h2.

Improvements to (and for) playground

  • Relax WebSocket 4K buffer limit (cc @ulrikstrid).
  • There will still be a message size limit, so warn about it client-side.
  • Don't run as root.
  • Don't start the container if the build failed.
  • GC old sandboxes and containers.
  • More logging for the various server-side state transitions.
  • Expose the static file loader. The playground uses it by accessing Dream internals.
  • Expose easy daemonization in Dream. Create systemd and/or other tutorials instead.
  • Separate saved, immutable sandboxes from mutable, running containers cleanly.
  • Prevent concurrent builds of the same sandbox.
  • Style improvements.
  • Mobile-friendly layout.
  • Consider linking to playground source code.
  • Store some of the examples as saved sandboxes.
  • Reason syntax support. Probably already works out of the box, for a sandbox that contains a source file with the right extension. Just link to this sandbox as the starter for Reason syntax.
  • Support TyXML, TyXML JSX, TyXML PPX.
  • Support SQLite.
  • Get rid of whatever is running Sentry.
  • Dump opam switch state and commit, together with package-lock.json.
  • Improve README.
  • Consider putting each container in its own network.
  • Consider randomizing port numbers.
  • Store more of the sandbox logs.

Get an SSL (or Lwt_unix?) error on concurrent connection

I get an SSL error under the below settings:

  • running 6-echo server with ~https:true

  • run ab command with 2 or more concurrency level

    $ ab -n 2 -c 2 -f TLS1.3 -T 'application/text' -p <(echo hi) https://localhost:8080/echo
    

error log:

15.04.21 00:55:12.132       dream.log  INFO REQ 1109 POST /echo 127.0.0.1:35012 ApacheBench/2.3
15.04.21 00:55:12.132       dream.log  INFO REQ 1109 200 in 40 μs
15.04.21 00:55:12.137       dream.log  INFO REQ 1110 POST /echo 127.0.0.1:35014 ApacheBench/2.3
15.04.21 00:55:12.137       dream.log  INFO REQ 1110 200 in 41 μs
15.04.21 00:55:12.138      dream.http ERROR HTTP (127.0.0.1:35016): SSL read() error: error:00000000:lib(0):func(0):reason(0)
15.04.21 00:55:12.138      dream.http ERROR Raised by primitive operation at Lwt_unix.shutdown in file "src/unix/lwt_unix.cppo.ml", line 1677, characters 2-38
15.04.21 00:55:12.138      dream.http ERROR Called from Lwt.Miscellaneous.wrap2 in file "src/core/lwt.ml", line 3087, characters 15-24

Expose Dream.static file loaders

  • The default file loader would probably be useful to wrap in a middleware-like fashion, so that users don't have to implement the actual file reading, but can easily add custom headers, especially ETag:.
  • Also expose a crunch-friendly loader, and add an example.
  • Simplify static loader and move it to Responses section.

See Dream.static ?handler (rename to ?loader?).

Implement backpressure for HTTP responses and WebSockets

Dream's API is already written as if there is backpressure (so there will be no change for users), however backpressure is not actually implemented. It is complicated by the underlying APIs. So, implement it for...

  • http/af
  • h2
  • websocketaf

  • Also carefully review how exceptions are passed back to stream users; the mechanism is already there, but all places that use it need a look. This is probably best done after any client code implementation, because that will force deciding which side (readers or writers) is responsible for what.

Get rid of workaround for %-encoder mutability once ocaml-uri is fixed

The workaround:

dream/src/pure/formats.ml

Lines 144 to 155 in 882a48e

(* TODO This triggers an upstream bug that causes a mutation. *)
let iri_generic =
`Custom (`Generic, iri_safe_octets, "")
let to_percent_encoded ?(international = true) string =
let component =
if international then iri_generic
else `Path
(* TODO Workaround for https://github.com/mirage/ocaml-uri/pull/156; `Path
should be `Generic. *)
in
Uri.pct_encode ~component string

is waiting for mirage/ocaml-uri#156. The workaround is itself buggy when used with other packages that also use ocaml-uri. It only avoids triggering the bug in interactions within Dream.

Support 100 Continue responses that don't end the current request

When the server gets Expect: 100-continue from the client, it should be able to send a 100 Continue response and go on processing the request, eventually sending the "real" response.

AFAIK, supporting this will require patching http/af. The Dream API will look something like

Dream.send_response : request -> response -> unit promise

...where response will then be Dream.response ~status:`Continue "" or Dream.response ~code:100 "".

cc @anuragsoni

Optimizations

  • Router: use a trie or a DFA instead of walking through all routes using CPS.
  • Logger: logging is slower than could be expected. Make it as fast and/or asynchronous as possible. In early testing, a Hello, world! app responded in ~5 microseconds on my machine, but the logger added ~150 microseconds of overhead.
  • Most web format parsers are naive and allocate a lot of intermediate strings and lists, probably contributing to GC pressure. Many of them also do multiple passes through the input. Almost all of them can be re-written into single-pass parsers that only allocate the final result.
  • (Edit 9/4/2021) Stream files from the default static loader rather than reading them into memory.
  • (12/4/2021) Minimize allocations during streaming (or measure them, including GC pressure).

Of course, everything should be measured first.

Some early notes from a few weeks ago:

  - request-id seems to take ~3 us.
  - catch ~1 us or less.
  - in-memory sessions 200 us on miss, 100 us on hit.
  - Initial form parser 9 us on tiny input; 30 us on larger, more realistic input
  - jwto CSRF tokens maybe 50 us.

Dream.run "starter" behavior: don't wait for ENTER, get rid of grace delay

It's probably enough to just ask people to press Ctrl+C to exit (it already works). As for the 1-second grace delay, it's probably unnecessary for starter code — but needed in robust Web apps, later in development. However, a robust Web app is already probably at least a medium-sized project that can afford to, and needs to, implement its own exit logic. That means the default grace delay is likely redundant in all cases.

cc @Lupus

Allow concurrent stream writing to bodies and WebSockets

Not supporting this is too burdensome, as it requires some kind of multiplexing abstraction to be built by the user (in the server) to support concurrent (interleaved) writing.

Flow control is still achieved by each writer waiting on the promises returned by Dream.write and Dream.send, so that using only one writer will still not trigger any buffering or concurrency. In other words, making this change will not affect the one-writer code that was possible before the change, and will only allow a new way of using the API.

  • WebSockets
  • Request bodies

Flow control is also likely to limit memory consumption and GC thrashing. AFAICT Dream is currently causing internal buffers to grow considerably. While writing large enough streams, it probably even exhausts the working set, causing page thrashing. These are unconfirmed hypotheses, but they likely explain the slowdown when sending enormous streams (>> 8 GB to 8 clients, for example). See example w-stress-response.

Using Rock in Dream to share middlewares [factor out sublibraries]

Hi @aantron!

Thanks a lot for your amazing work on Dream, I'm really excited to see a full-featured OCaml web server and can't wait to try it for a side project 😄

I wanted to take the pulse and see how you felt about using Rock (Opium's low-level HTTP and middleware layer) in Dream.

To give a bit of context, we extracted Rock from Opium and made it as minimalist as possible in the hope that it could be re-used by other Web frameworks in the community.
Our motivation was to provide a common middleware layer to all of the frameworks so that users and frameworks could re-use them. (In the spirit of Rack from Ruby)

If that seems to make sense to you, there would be some work needed on our side, as Rock does not support Http2 and Web sockets at the moment, but that's things we wanted to work on anyways.
I've also seen #4 and that's something I wanted to explore as well. All in all, there seem to be some overlaps and I'd love to see a collaboration happen to consolidate OCaml's web ecosystem.

Cheers and thanks again for all your work!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.