Comments (34)
I haven't thought too much about what a minimal @undici/core would contain.
I have a pretty solid idea of what I would have in core:
lib/node/*
lib/llhttp/*
lib/core/*
lib/agent.js
lib/timers.js
lib/balanced-pool.js
lib/client.js
lib/dispatcher*
lib/pool*
from undici.
In short, yes many of these issues can be solved with better/more documentation AND improved code organization (not necessarily meaning monorepo). Let's start there and iterate.
from undici.
I have experience with changesets. Do you want me to try npm+changesets or go straight to pnpm+changesets?
from undici.
Rough plan:
- Create PR that moves everything into
packages/undici
anddocumentation
- No code will split up just yet
- CI should run as normal
- Docs site should work as normal
- Other tooling should continue to work (build, commit hooks, etc.), and run from root as normal
- This should only introduce npm workspaces. Changesets should not be required.
- Even though the code will exist in
packages/undici
directory, the package should still be named and published asundici
(for now). - (In
nodejs/node
) Update Node.js -> Undici vendor process
Step 1 is simply the first step in migrating to a monorepo. This will reduce impact on contributors and should not disrupt development.
- Code split (incremental)
- Create new packages
packages/core
,packages/fetch
, andpackages/types
- Start publishing packages as
@undici/core
,@undici/fetch
, and@undici/types
- Retain
packages/undici
asundici
and make it roll-up and export all of the other packages - Introduce changesets for version and publishing management.
- (Optional) Move linting and formatting to root
- Create new packages
Step 2 serves as an incremental step to a true monorepo organization. By retaining
undici
package we don't have to affect Node.js or users. Everything will appear as normal.
-
Migrate Node.js from
undici
to@undici/fetch
- Make necessary change in
nodejs/node
to vendor@undici/fetch
instead ofundici
- Make necessary change in
-
More code splitting!
- Intentionally not-well-defined at this time, consider other parts of Undici that we can split out into its own separate packages.
from undici.
I've found more success where versions are updated by a tool such as changesets. That said, switching to pnpm has other great benefits -- and everyone can do it since it's supported in corepack. Ill make sure to present options for us to consider
from undici.
Without hearing the benefits of having a monorepo, I can't make an argument against it, or agree to having one.
A monorepo won't make undici easier to maintain, it won't make undici easier to publish, and it won't make it easier to install. Is the only benefit bundle size? If so, undici has constantly grown and now has 7.2 million downloads a week, and we have not received a single complaint about bundle size. People who use @types/node aren't complaining about the added size of undici-types, even though most of the types aren't being used.
The entirety of undici is 1.24 MB. According to npm, fetch is 277 kB, cache is 25.6, and WebSocket is 57.5. If we were to split them into separate packages, undici core will still be ~880 kB. That is a menial difference when the docs being bundled are 183 kB, types are ~80 kB, and llhttp is 283 kB. I believe Matteo prefers that docs are included for them to be easy to access (same reason we don't use esbuild on undici and put a minified file on npm), but claiming that bundle size is too big while shipping a substantial amount of markdown files doesn't make sense to me.
I am willing to change my mind on this, if there is a noticeable improvement to any area of development. Let me know 😄!
from undici.
regarding testing - what is the issue with that? too slow?
from undici.
I think @Ethan-Arrowood was managing this idea for a while, after the split of undici-types
from undici.
There's some old threads about monorepo ideas. Happy to revisit
from undici.
I think it's inevitable that we adopt a monorepo at this point, this project has grown too big, and there are separate subsystems that could be used separately.
from undici.
Use pnpm
This is also my preference. Does anyone feel extra-strongly about using npm?
pnpm is now included in corepack so it should be very easy for everyone to use it without any additional install or set up (minus a corepack enable
call).
from undici.
I will also start by experimenting with npm workspaces to start. They've supposedly come along way so maybe its worth trying again.
from undici.
I've not been able to make npm workspace work in any reasonable way for OSS development
I think the most useful part would be if somebody could figure out how to set it up with changesets so that we could ship independent versions of the packages. Alternatively, they will all have the same version.
from undici.
I'm +1
from undici.
LGTM 🚀
from undici.
I'll have a PR up for step 1 very soon. I'd like to land #2766 first as that will help simplify things.
from undici.
My experience with npm workspaces is that they are not fit for purpose. Specifically, they are missing the workspace protocol, https://pnpm.io/workspaces#referencing-workspace-packages-through-aliases, which makes it impossible to maintain an OSS library without it. (Or I couldn’t figure out how to handle versioning).
@KhafraDev wdyt?
from undici.
I dont like monorepos. They make managing code complicated. I would rather have one package released on npm with everything than having to manage many packages where each package has different versions.
from undici.
Well, next undici version should be about 150 kb smaller. We can still remove some artifacts in llhttp and save about 8 kb and maybe the 1kb in for the MIT license in the fetch folder by giving credits for fetch to @Ethan-Arrowood in the main license file.
Maybe 74 kb if we drop the non-simd-version of llhttp.
from undici.
The biggest motivation for me is simplification. We have so much going on in this library now, splitting into at least core and fetch would be a developer experience improvement imo.
I think the benchmark and documentation split outs are still valuable even if we don't reorganize code.
from undici.
fetch is contained to lib/fetch, the only difference would be that it would be moved to packages/fetch/lib and I couldn't directly require/import the files since they'd have to use workspace aliases. The code is very organized and easy to work on right now.
from undici.
Also cache
, eventsource
, and websocket
that are all related to web platform APIs too, right?
I won't open a monorepo PR just yet. I'd like to split out other pieces first like benchmarks
, docs
, and examples
. Maybe @ronag or @mcollina have other inputs regarding monorepo organization.
from undici.
yes, those are all self-contained (and small relative to package size)
from undici.
I would prefer to have all web api stuff in own thing.
from undici.
The biggest motivation for me is simplification. We have so much going on in this library now, splitting into at least core and fetch would be a developer experience improvement IMO.
+1
from undici.
how would it improve developer experience?
If developer = maintainer, the act of maintaining the tooling offsets any simplification (I don't see how adding more tooling could simplify the development process). Being able to write pure javascript that works without tooling is a huge benefit here. Even 'simple' tooling like esbuild has caused us issues - imagine more complex tools! Personally, moving to a monorepo would break my development process.
If developer = user, having to install more packages and deal with a breaking change for a benefit no one here has expanded on doesn't seem like a good idea.
I'm not looking for an essay or a research study, just anecdotal evidence would be fine.
from undici.
For me it would be easier not to deal with all the web api both in terms of browsing code and testing.
from undici.
web tests will still run in a monorepo though
from undici.
Then I would suggest to split it into two different repos. undici core and undici web.
from undici.
We can leverage a tool like turborepo to make this as efficient as possible, but since web would depend on core, any change to core would result in running web too. But like if maybe pieces like the redirect handler or mocks lived outside of core, then changing those pieces would not require running core or web. Just think of it like a directed graph. It will depend on how we organize our modules.
I think the issue some of us have is the information density of the project. Sure the code lives in separate folders, but yet we still have a lot of it. It is difficult to know where to make changes, how to add new apis, etc. Anecdotally, I was confused for a bit what the cache
directory contained, and why it wasn't a http cache solution that all of undici could use (like the idea shared in #2256).
Beyond that I have other related issues such as code quality, documentation, and testing organization that come to mind. I can empathize with @KhafraDev and @Uzlopak , and agree that just switching to a monorepo layout is not going to fix these issues directly. However, in my opinion, fixing things like that can be easier when there is less code density.
Having a minimal core package will enable us to slim down and simplify the core api to a point where it will also be simpler to document, test, type, and more. Then other pieces can build off of that and adopt the same simplicity in docs, tests, etc.
from undici.
I haven't thought too much about what a minimal @undici/core
would contain. I find the current Dispatcher
implementation to be confusing. The indirection in particular is the worst part (such as the makeDispatcher
piece in index.js
). This obviously was done for good reason (such as extending/implementing the interface in an easy way across the repo). But if for example these pieces lived in their own module, it could force us to come up with a better API for other pieces like the Mock interfaces or custom Handlers.
This is just an off-the-cuff idea, but I hope it helps provide a little more anecdotal evidence behind my support for a monorepo organization
from undici.
I think the information density is an excellent description. Something been bothering me but I couldn't describe it. I think especially for third-party library implementers (such as my own nxt-undici and others such as elastic and possibly axios, got etc...) who only actual need or care about undici core. We/They have to navigate through a huge project just for the few things we need.
from undici.
The indirection in particular is the worst part (such as the makeDispatcher piece in index.js).
makeDispatcher
is not a core thing.
from undici.
Then I would suggest to split it into two different repos. undici core and undici web.
I'd prefer this over a monorepo.
Sure the code lives in separate folders, but yet we still have a lot of it. It is difficult to know where to make changes, how to add new apis, etc.
@Uzlopak was able to add EventSource without any help from me or anyone else on how to organize code. If we're talking about the lower leveled dispatcher api/mocks/ProxyAgent/etc., that simply isn't an issue with the web stuff and wouldn't be solved by moving it. Moving the web code to their own folder (in the PR linked here) will let people use tooling to ignore those files.
Not only that, others have made breaking changes relatively quickly lately (such as the removal of some dispatcher handler options and interceptors).
possibly axios, got etc...
axios and got both use node:http and axios will use XMLHttpRequest in browsers. If axios ever moved to fetch then they likely use undici's fetch instead of undici core (or just global fetch). I know this is very shitty proof but I looked at the most downloaded dependents of undici on npm and every single one of them used fetch. I would go so far as to say that undici.fetch is just as fundamentally important as undici.request, etc., as they both serve two entirely separate use cases in a similar api.
I haven't thought too much about what a minimal @undici/core would contain. I find the current Dispatcher implementation to be confusing. The indirection in particular is the worst part (such as the makeDispatcher piece in index.js). This obviously was done for good reason (such as extending/implementing the interface in an easy way across the repo). But if for example these pieces lived in their own module, it could force us to come up with a better API for other pieces like the Mock interfaces or custom Handlers.
I'm very confused as to how you came to the conclusion that the core api being confusing can be solved by moving the web stuff. The web stuff doesn't need docs, and this seems like a documentation issue w.r.t. core.
I can't speak on information density
because I've never had issues with it honestly. Of course we could move MockAgent or ProxyAgent to their own packages, and I wouldn't have an issue with that, because they are higher leveled than even fetch, and do not rely on core.
Fetch relies on core internals and core relies on fetch internals (specifically undici.request w/ a FormData body). There is no easy way of decoupling fetch from core with a benefit that seems to boil down to '[core/web] is too [confusing/big]' ("big" in a more abstract way than package size). From what I've seen @Ethan-Arrowood mostly has an issue with documentation in core, because the web stuff simply has enough documentation on MDN or a million other places.
So I would like to leave a few questions:
- can these issues be solved with better/more documentation?
- does #2793 solve the other issues? Tooling can now more readily be used to exclude
lib/web
from code search results.
from undici.
Related Issues (20)
- Nightly tests are failing HOT 2
- Improve support for pre-shared sessions (PSK) HOT 3
- Request constructor: Failed to parse URL from ___ HOT 2
- Broken links in docs webpage HOT 2
- Failing JSON parsing in Node 22 HOT 13
- undici/types/index.d.ts:21:8: Type error TS1192: Module '"[email protected]/node_modules/undici/types/interceptors"' has no default export HOT 6
- Node 22.3.0 regression: undici sends invalid origin header in some cases HOT 2
- fetch arrayBuffer() leaks random data from process memory HOT 18
- build: husky install is deprecated HOT 1
- Ability to externalize WASM was broken HOT 13
- Measure request latency HOT 3
- dns round-robin interceptor + cache HOT 3
- async dispatch handler helper
- Cannot convert argument to a ByteString because the character at index 46 has a value of 65286 which is greater than 255 HOT 2
- Undici.Request and AbortController doesn't work well HOT 22
- The RetryHandler receives a duplicate body when the server does not support Range requests. HOT 11
- `Dispatcher.compose` doesn't return a true dispatcher HOT 9
- Back-pressure doesn't work with sub dispatches
- Add generic type for opaque object for stream() and pipeline() HOT 2
- v7 HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from undici.