alaviss / nim-sys Goto Github PK

The current version in features/exec-prototype branch is the API we eventually want, but due to the sheer size it's hard to write all the tests in one go.

So we are gonna pick small parts out and start implementing and testing those instead. Current checklist:

• findExe()
• Launcher
  • program
  • args
  • launch()
• Process
  • wait()
• ProcessStatus

The Windows version will be implemented as part of this.

std/streams is a blocking API, and asyncstreams doesn't work for byte buffers. We would need to engineer a brand new interface that can support async operations.

It would be much easier if CPS worked...

Current progress:

• Platform support
  • Windows
  • macOS
  • Linux
• Networking protocols
  • TCP
  • UDP
  • Unix (POSIX)
  • Unix (Windows)
• Address families
  • IPv4
  • IPv6: #47
  • Unix
• Features
  • Asynchronous resolver
  • Pluggable resolver
  • Listener
    • Generic listen() interface
      • Use any iterable list of endpoints for local binding
    • Dual-stack IPv4/IPv6
  • Conn
    • Generic connect() interface
      • Use any iterable list of endpoints for initiating connection
    • Timeout for I/O

At the moment, the Nim compiler remains a big obstacle to the completion of this implementation.

Hey, great work on this library! Very clean. I hope it's in Nim 2.0!

I'm currently making a PR for a setNonBlocking proc in Nim. I ran into an issue with FreeRTOS though. Its fcntl impl seems (I don't know) to be dependent on the hardware and other extensions it is built with. But so far, I don't think anything other than setting socket FDs to non-blocking works.

If the handle types were all distinct in current Nim, then I could do something like this to cover FreeRTOS:

proc setNonBlocking(fd: cint, nonBlocking = true)
when defined freertos:
  proc setNonBlocking(fd: FileHandle, nonBlocking = true)
    {.error: "Not available for FreeRTOS".}
else:
  proc setNonBlocking(fd: FileHandle, nonBlocking = true) =
    setNonBlocking(cint(fd), nonBlocking)
proc setNonBlocking(fd: SocketHandle, nonBlocking = true) =
  setNonBlocking(cint(fd), nonBlocking)

How would you handle this scenario with your API?
Maybe something like this, where it can be overriden if the user is using a FreeRTOS arch that fully implements fcntl:

# handles_posix.nim
const hasFileNonBlock {.booldefine.} = defined(posix) and not defined(freertos)

template setBlockingImpl() {.dirty.} =
  when hasFileNonBlock and not (fd is SocketFD):
    {.error: "Platform doesn't support setting non-block property of file handles.".}

Also, aren't FDs blocking by default? If so, then why have your API be setBlocking instead of setNonBlocking? Wouldn't your API require the blocking param to be set to false for the most comon use case?

stdin.setBlocking false
stdin.setBlocking true
# vs
stdin.setNonBlocking
stdin.setNonBlocking false
# I don't know, now that I see them both, maybe your way is cleaner.

One more thing: You differentiate sockets from other handles. Why not have a handle for other types like anonymous pipes, etc? This would allow handling of incomplete platform impls like the FreeRTOS example above.

TODO: Write this thing into an actual document

Thanks for taking the time to contribute :)

The project currently enforces these rules:

• Commit rule:

  • The commit message is as important as the content of the commit. It should be written in the following format:

  <component>(<subcomponent>): <summary>
  
  <detailed description of the change>
  

  For <component>, use the Nim import path syntax, eg. sys/exec, sys/pipes or sys/[exec, pipes] for changes to both at the same time. Changes affecting the entire project should use sys(treewide).
  <subcomponent> usually denotes the specific implementation of a module, for example: sys/exec(posix) for changes to the POSIX implementation, sys/pipes(windows) for changes to the Windows implementation. Changes affecting all implementations don't require this part.
  <summary> should provide a short description of what is changed and/or what is the effect of the commit.
  The first line should not exceed 72 characters, and must not exceed 80 characters.

  The detailed description should explain at least the rationale behind the change, and the change itself in details. The rationale is always required unless covered by the summary, but the details can be omitted if it's trivial or is covered by documentation within the implementation (ie. changing one line due to a typo). Detailed descriptions should be hard wrapped to 72 characters.

  All commits staged for merging must follow this rule. No exceptions.

  Some examples: 827afaf, a51a859, 6649f5c, 77a199d

  • Keep commit small if possible, and they need not to compile/pass tests perfectly, though it is preferred that they do.

  • Feel free to adjust the commit history within your own branch as needed (eg. you used temporary commits and now you want to squash them into one commit, or if you want to rebase your work on top of the current master). However, never change the history if collaboration is being done in the branch.

• PR rules:

  • Any feature addition/fixes to be merged to master must be accompanied by tests in the same PR.
  • Any introduced API, private or public, must be documented.
  • Focus on one thing at a time. If the PR bring in changes unrelated to the title, it will be discarded.
  • PRs are reviewed on a per-commit basis.

• Merge rules for maintainers:

  • Use merge commits, and the merge commit should describe the work being merged. It is fine to just use a PR title then refer to them.
  • Never alter the history of an another user unless explicit permission was given.
  • All commits must be signed.

I'm a bit stingy regarding commits, but these strict requirements will help collaboration and people who read the code a lot, trust me :)

Goals:

• A base for all path-based functionalities in the project.
• Stateless, do not depend on the file system (sys/fs will be the extension of sys/path for that purpose).

Problems:

• Does the relative vs absolute path distinction matter enough to warrant distinct types?
• Does it matter to be able to manipulate POSIX paths on Windows and vice versa?

On Windows the native encoding is UTF-16. This poses a challenge where typical Nim strings has to converted to UTF-16 before they can be processed.

Right now, the conversion is done using WideCharToMultiByte and its counterpart, which silently drops invalid characters by normalizing them into � (U+FFFD -- Unicode Replacement Character). Furthermore, Windows itself doesn't deal in strict UTF-16, but rather WTF-16 (strings of arbitrary 16-bit codepoints) as a historical artifact.

All of this means that right now, sys operations on Windows have unexpected behaviors when the input string is not well-formed UTF-8 or when the output is not well-formed UTF-16.

To handle this, add a new string type called strings.Native (name subjected to change) that is in OS encoding and provide platform-specific conversion into their native code points.

An open question is to figure out what to do about strings.Without[T]: Whether to have it dealt in strings.Native or to create a new strings.WithoutNative[T].

i think it is possible and not hard to add new ioqueue and procs on common structures which will run syscalls through io_uring

Now that sys hosts one of the only OS-agnostic async implementation based on CPS, there is a lot of demand for a socket library.

An unordered list of things to do:

• A generic Socket type that works with all sockets.
  • Use the same read(), write() interface as with files. This should allow concepts and friends to match sockets as well as files.
• Specialized distinct TCPSocket or even TCPClient types to specialize sockets into different kinds, then implement specialized operations on top.
• Figure out the best way to handle dual-stack sockets and whether they should be the default. Some operating systems like OpenBSD and Dragonfly does not support dual-stack.
• Figure out how timeout should be done. For synchronized sockets this should be very simple, the question lies on async sockets.

Open questions:

• What to do with DNS resolution? On the majority of OS this is fully synchronous and we don't have thread migration support at the moment. Might have to look into loony, but it doesn't work with ORC.

A lot more research has to be done as the topic is researched further

Target: Vista+

Problems:

• CMD and Windows has different parameter quoting algo (and all of them sucks).
• Not everything in PATHEXT can be run directly (need confirmation on whether CreateProcess can launch .bat directly or cmd.exe has to be used).
• inherits is racy on Windows since the inheritable state is global, and we need to restore it afterwards. Probably a lock is the best way atm.

Currently files only provide low-level read/write primitives. These are not buffered, so convenience like readLine are not available.

Goals:

• Pluggable buffering interface, basically so that any Stream can be plugged into BufferedStream and receive all the perks.

Goals:

• A home for the prototype NullessString in sys/exec prototype.
• Potentially a place for special string-related stuff to be added (ie. the NUL-terminated string array in sys/exec's POSIX impl).

Problems:

• Need a name.
• What should be the range of operations provided for those types?
• Should the library be a part of the public api?