unetworking / usockets Goto Github PK

To know who connected

This is needed to avoid buffering on write failures due to errors.

Hello,
I'm using your framework to rewrite an MMORPG back-end. While I do realize that there are still a few tweaks required here and there, I stumbled across the following error:

When using us_loop_free to terminate the default loop (created with us_create_loop(0, ...), an exception is thrown: https://i.gyazo.com/db87de7a1567ff6a3fa60671d83c1bb2.png

This one was triggered by executing my App::shutdown call from within my OnIdle callback (just a timer called every x ms).

The shutdown method simply closes my OnIdle timer and then calls us_loop_free.

I've also implemented a small wrapper for creating signals (just for libuv), which does actually work. However, when calling my shutdown method from within the signal callback (to clean up and do a clean shutdown of the program), I get this error: https://i.gyazo.com/459cc385f6befba569111d1457b5f3b1.png

I've tested any possible scenario, and the only working one where no error is thrown is when I terminate the program with exit() from anywhere (even inside my OnIdle callback) as long as I don't do any us_loop_free calls.

One more test I did was replacing the deprecated uv_loop_delete function with uv_loop_close and manually free'd the loop with uv__free. Also tried removing the uv_run call inside the us_loop_free function. Both tests ended up with the same error as above with the next_idle_handle being invalid.

Any solution for this? I did everything exactly as shown in your examples.
I'm running this on Windows 10 x64, Visual Studio 2017. Libuv 1.23.2.

Currently there is libuv and epoll support. ASIO/C++20 support will be added in the future. This will need to be implemented in C++ but exported as C functions.

Since API change

The API does not expose the details of an error, e.g. error codes.

Many use cases need to know the exact reason for a connection failure, for recovery and logging and to inform the user. (The library I work on uses these codes to decide whether to retry a connection. And my employer’s support team is often found poring over error logs submitted by developers using our library, to help them figure out why the client won’t connect to the 🤬 server.)

Error codes are problematic because there are multiple namespaces (errno, gethostbyname status, OpenSSL, mbedTLS, Apple OSStatus, etc.) What I tend to do is define an enumeration covering all the namespaces, then make my error type a {namespace, code} tuple. But defining your own namespace with an enum of your own error codes works too...

(This is another task I’m willing & likely to work on, if I end up choosing uWebSockets for this work project.)

I read the eventing/epoll.c.
When create fallthrough-poll, it doesn't touch loop->num_polls.
When close fallthrough-poll, it will do loop->num_polls--.
It is a designed behavior and imply fallthrough-poll may not be closed? or I miss sth?

This version has a new interface,

Should have new documentation and front page

This API will be a lot more stable and fixed than the prior, shouldn't need any big changes from now on

Would you please choose an open source license for this project, and add the text of the license to a LICENSE file? I'd love to use this package in a project I'm working on, but want to make sure it's available under a common open source license.

please create a new release including at least 824ff97 when possible'' thanks''

meanwhile (with release v0.1.2):

build-linux-x86_64/local/include/uWS/AsyncSocket.h: In member function ‘std::string_view uWS::AsyncSocket<SSL>::getRemoteAddress()’:
build-linux-x86_64/local/include/uWS/AsyncSocket.h:95:9: error: there are no arguments to ‘us_new_socket_remote_address’ that depend on a template parameter, so a declaration of ‘us_new_socket_remote_address’ must be available [-fpermissive]
         us_new_socket_remote_address(SSL, (us_new_socket_t *) this, buf, &ipLength);

as you know.

(note https://github.com/uNetworking/uSockets/archive/master.tar.gz is useless cos is mutable)

(note git modules linking is also useless cos git is not onmipresent)

(note all this started by downloading a release from https://github.com/uNetworking/uWebSockets/releases, cos uSockets folder is empty and somehow needs to be downloaded)

Instead of relying strictly on open/close events, a more dynamic resource management can be done with reference counting. Similar to OpenSSL, us_socket_up_ref could be added to ensure a different lifetime for the socket. This together with us_socket_is_closed makes it very useful in cases where you submit the socket to some async queue.

Cost of this could be 1 bit, 1 byte or 2 bytes. More than this, and something is just wrong.

Instead of having two sets of functions; us_ssl_socket_ext and us_socket_ext, etc, you could have one single:

us_socket * can be SSL or non-SSL, it is determined by us_socket_ext(SSL or not, ...)

Instead of two different sets of functions almost identical, what is not identical could be hints instead. So SSL options could be hints that can be ignored for non-SSL.

That would clear up a lot of template magic in µWS and make it a lot easier to work with.

First step: add a new layer named us_any_socket and us_any_socket_ext, etc -> have it be a set of macros to correct function.

Second step: rename everything so that _any prefix can be removed.

In the epoll backend the conversion from milliseconds to nanoseconds is wrong. This means that times of less than one second do not work.
Instead it should be something like:

    struct itimerspec timer_spec = {
        {repeat_ms / 1000, repeat_ms * 1000000 % 1000000000},
        {ms / 1000, ms * 1000000 % 1000000000}
    };

In fact sorry about posting here, but uWebSockets doesn't accept issues.
preparations

# make new machine on cloud hetzner
mkdir uws
cd uws
git clone https://github.com/uNetworking/uWebSockets --recursive
apt install -y make clang libssl-dev
cd uWebSockets
make

output

clang -flto -O3 -c -IuSockets/src uSockets/src/*.c uSockets/src/eventing/*.c
clang++ -flto -O3 -c -std=c++17 -Isrc -IuSockets/src examples/HelloWorld.cpp
clang++ -flto -O3 -s *.o -o HelloWorld
/usr/bin/ld: /usr/lib/llvm-6.0/bin/../lib/LLVMgold.so: error loading plugin: /usr/lib/llvm-6.0/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:4: recipe for target 'examples' failed
make: *** [examples] Error 1

apt install -y llvm doesn't help

clang -v

clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.3.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0
Candidate multilib: .;@m64
Selected multilib: .;@m64

It compiles and runs on Windows but I cannot connect to the echo_server in examples. I have no idea what Windows wants me to do.

FYI: I am using uWS in visual studio in a windows environment.

I was able to dig through issues and find how to stop listening so that the event loop can finish its work and attempt to exit. Ultimately leading to the application exiting.

For me, when the loop was trying to free its self using Loop::get()->free() it was actually causing another loop to be spawned (the call to get() didn't return the previously instantiated loop), then when trying to free the SECOND loop things went haywire.

Apparently the thead context for std::atexit is not as you expected in windows OS.

Anyways its a (seemingly) easy fix, remove the call to Loop::get() in the atexit lambda:

static Loop *get(void *existingNativeLoop = nullptr) {
	static thread_local Loop *lazyLoop = nullptr;
	if (!lazyLoop) {
		/* If we are given a native loop pointer we pass that to uSockets and let it deal with it */
		if (existingNativeLoop) {
			/* Todo: here we want to pass the pointer, not a boolean */
			lazyLoop = create(true);
			/* We cannot register automatic free here, must be manually done */
		} else {
			lazyLoop = create(false);
			std::atexit([]() {
					 if(lazyLoop)
						lazyLoop->free();
			});
		}
	}

	return lazyLoop;
}

It does not send as far as possible, but returns if successful.

Either switch back to SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER or make us_ssl_socket_write call SSL_write in a loop.

v0.0.5

When I try to compile by uSockets.vcxproj without LIBUS_NO_SSL. It's failed with LNK2019 unresolved symbol BIO_get_data, BIO_meth_free, BIO_meth_new....

/opt/uSockets$ make
for f in examples/.c; do gcc -flto -std=c11 -DLIBUS_NO_SSL -O3 -o $(basename "$f" ".c") -Isrc src/.c src/eventing/*.c "$f"; done
examples/echo_server.c:4:20: fatal error: helper.h: No such file or directory
compilation terminated.
examples/hammer_test.c:4:20: fatal error: helper.h: No such file or directory
compilation terminated.
examples/http_load_test.c:4:20: fatal error: helper.h: No such file or directory
compilation terminated.
examples/http_server.c:4:20: fatal error: helper.h: No such file or directory
compilation terminated.
Makefile:2: recipe for target 'default' failed
make: *** [default] Error 1

Hey there, having some trouble getting this to work on Windows. This is my error:

server.obj : error LNK2019: unresolved external symbol us_create_ssl_socket_context referenced in function "struct us_new_socket_context_t * __cdecl us_new_create_socket_context(int,struct us_loop *,int,str
uct us_new_socket_context_options_t)" (?us_new_create_socket_context@@YAPEAUus_new_socket_context_t@@HPEAUus_loop@@HUus_new_socket_context_options_t@@@Z) [endec\build\testaddon.vcxproj]

I must not be linking some .lib file that I need as a dependency that has the forward declarations. Do you know which ones I am supposed to link?

So far I am doing:

• uSockets.lib

@alexhultman, thanks for your great library!

Would you mind putting the headers into .\ include\usockets?

We want to support Usockets in VCPKG for users, we installed the headers to \inlcude\usockets, and the test project couldn’t find related headers since they doesn’t have prefix ‘usockets’ for used headers. If we put the headers directly to .\include, it works, however we have many packages installed, so this might be conflict for some packages.

unresolved external symbol us_internal_socket_is_closed referenced in function "public: void * _cdecl <lambda...............", just this one,

It does not pass hammer test as it calls wrong callbacks after adoption.

Hi Alex,

I'm not using your library yet, but am looking forward to v0.15 as it looks like just what I need. I was looking through the uSockets code at how you were doing the libuv integration (my app is heavily libuv-based for other tasks, but I want to use your library to add a bit of http serving), and I notice that you provide your own run method: us_loop_run(). Could I request that use of that method be optional, as other code I have wants to 'own' the call to uv_run?

From what I can see by quick inspection, us_run_loop() just calls us_timer_set() with some internal values and then calls uv_run(). If you could provide something like a us_loop_setup_run() as a wrapper around that us_timer_set() call and is to be called prior to uv_run(), then users who are directly using libuv can use that instead of us_loop_run().

In my application using uWebsockets.js with SSL, all sockets connected to the server will disconnect if one is closed with WebSocket.end. I believe I've narrowed the cause down to the two calls to SSL_get_error in ssl.c. After the first socket is closed, the rest will report incorrect error codes and be disconnected as well.

From the OpenSSL docs:

SSL_get_error() must be used in the same thread that performed the TLS/SSL I/O operation, and no other OpenSSL function calls should appear in between. The current thread's error queue must be empty before the TLS/SSL I/O operation is attempted, or SSL_get_error() will not work reliably.

Adding calls to ERR_clear_error before SSL_read/SSL_write fixed the issue for me. I'm submitting an issue rather than a PR because I'm not sure this was the best way to solve the problem, or if the intermediate call to us_socket_flush in us_ssl_socket_write could cause any issues.

Connections can block when doing DNS resolution,

With C11 (we already use alignas) threading is standardized and could be used to run blocking DNS lookups.

I started a user space TCP stack 2 years ago. Time to merge this work as an experimental layer here.

LIBUS_USER_SPACE_TCP can be a new experimental layer that focus on extremely small sockets.

If not useful, still a fun thing to try.

I noticed that both https://github.com/alexhultman/tcp and this project offer some kind of an experimental user-space TCP implementation. I gather that this project offers TCP with Berkeley sockets and Epoll (user-space TCP with Berkeley sockets and epoll?) while https://github.com/alexhultman/tcp does not include either one. Can you explain this further?

I completely understand that both projects are marked "experimental" but am sure we can learn a lot from what is already there.

Hello,

First of all, thank you for this work, I gave it a test and it's blazing fast.
I just have a little problem, the current proposed multithreading solution has no effect on Windows because SO_REUSEPORT is not supported and does anything, so only one thread will get connections.
Is there anyway to implement multithreading in the listener?

Thank you.

LIBUS_EXT_ALIGNMENT should apply to every user-exposed struct, except for some (if motivated).

How I can detect connect errors?

Hello Alex,

I tried to do 1 curl requst to SSLApp from examples/HelloWorlds.js:

$ curl -k https://localhost:9001/

It seems something wrong in the event loop request handling logic:

1549719277.970468 accept4(20, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = 21
1549719277.970569 epoll_ctl(3, EPOLL_CTL_ADD, 21, {EPOLLIN, {u32=4294967295, u64=1791001362431}}) = 0
1549719277.970664 epoll_ctl(3, EPOLL_CTL_DEL, 21, 0x7fffac471a20) = 0
1549719277.970690 ioctl(21, FIONBIO, [1]) = 0
1549719277.970713 epoll_ctl(3, EPOLL_CTL_DEL, 21, 0x7fffac471a40) = -1 ENOENT (No such file or directory)
1549719277.970737 setsockopt(21, SOL_TCP, TCP_NODELAY, [1], 4) = 0
1549719277.970776 accept4(20, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
1549719277.970813 epoll_ctl(3, EPOLL_CTL_ADD, 21, {EPOLLIN, {u32=21, u64=563242011197461}}) = 0
1549719277.970837 epoll_pwait(3, [{EPOLLIN, {u32=21, u64=563242011197461}}], 1024, 950, NULL, 8) = 1
1549719277.975440 recvfrom(21, "\26\3\1\0\317\1\0\0\313\3\3X\2776\256\234\356\22d\377t)\t9\\\352\216\3E\353Q\16"..., 524288, 0, NULL, NULL) = 212
...

After accept4 file descriptor 21 we do EPOLL_CTL_ADD then EPOLL_CTL_DEL twice, then accept4 again.

Please correct this behavior, this is not only waste of time. Some bigger problem inside.

I'm using the us_loop_integrate() function to use uWS with my own event loop, but during the exit sequence us_loop_free() calls uv_run(). This is a surprising behaviour -- should we really be re-running the event loop during exit (when stuff has already largely been torn down), especially if its not your own event loop?

doing make gives me this result

1 error generated.
In file included from src/socket.c:19:
In file included from src/internal/common.h:24:
src/internal/eventing/epoll.h:23:10: fatal error: 'sys/epoll.h' file not found
#include <sys/epoll.h>
^~~~~~~~~~~~~
1 error generated.
In file included from src/eventing/epoll.c:19:
In file included from src/internal/common.h:24:
src/internal/eventing/epoll.h:23:10: fatal error: 'sys/epoll.h' file not found
#include <sys/epoll.h>
^~~~~~~~~~~~~
1 error generated.
In file included from src/eventing/libuv.c:19:
In file included from src/internal/common.h:24:
src/internal/eventing/epoll.h:23:10: fatal error: 'sys/epoll.h' file not found
#include <sys/epoll.h>
^~~~~~~~~~~~~
1 error generated.
examples/http_server.c:111:51: warning: incompatible pointer types passing 'struct us_new_socket_t
*(struct us_new_socket_t *, int)' to parameter of type 'struct us_new_socket_t ()(struct
us_new_socket_t *, int, char *, int)' [-Wincompatible-pointer-types]
us_new_socket_context_on_open(SSL, http_context, on_http_socket_open);
^~~~~~~~~~~~~~~~~~~
src/libusockets_new.h:168:126: note: passing argument to parameter 'on_open' here
...int ssl, struct us_new_socket_context_t *c, struct us_new_socket_t *(*on_open)(struct us_new_...
^
1 warning generated.
make: *** [default] Error 1

Various threading features are required. Reuse port, master listener & slave worker, etc.

A complete overhaul of the interface is much needed,

us_new_socket should be renamed to us_socket and us_socket should be renamed to us_tcp_socket and be removed from public documentation.

A basic HTTP or HTTPS GET request example would be very useful for new users of this lib.

To try to make this, I took the load test from uWebsockets as that's the closest to an example I could find, and tried to modify it to send an HTTP GET request. But it just runs through with no output and event loop doesn't seem to be blocking. Clearly I suck, but a minimal working example would really help in the manual or ReadMe.

// Source: https://github.com/uNetworking/uWebSockets/blob/16bd859b0caf5fa96cf9cbf27b64ef4229629cf1/benchmarks/load_test.c
// Modified, attempting a simple GET request.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "../uSockets/src/libusockets.h"


char request[] = "GET / HTTP/1.1\r\n"
                 "Host: google.com\r\n\r\n";
const char * host = "google.com";
int port = 80;

struct http_socket
{
  /* How far we have streamed our request */
  int offset;
};

/* Loop event callbacks. We don't need any of these. */
void on_wakeup(struct us_loop *loop)
{
  //
}

void on_pre(struct us_loop *loop)
{
  //
}

/* This is not HTTP POST, it is merely an event emitted post loop iteration */
void on_post(struct us_loop *loop)
{
  //
}

struct us_socket *on_http_socket_close(struct us_socket *s)
{
  puts("on_http_socket_close was called");
  puts("Closed!");
  return s;
}

struct us_socket *on_http_socket_end(struct us_socket *s)
{
  puts("on_http_socket_end was called");
  return us_socket_close(s);
}

struct us_socket *on_http_socket_writable(struct us_socket *s)
{
  puts("on_http_socket_writable was called");

  struct http_socket *http_socket = (struct http_socket *) us_socket_ext(s);

  /* Stream whatever is remaining of the request */
  http_socket->offset += us_socket_write(s, request + http_socket->offset, sizeof(request) - http_socket->offset, 0);

  return s;
}

struct us_socket *on_http_socket_data(struct us_socket *s, char *data, int length)
{
  puts("on_http_socket_data was called");
  puts(data);

  /* Get socket extension and the socket's context's extension */
  struct http_socket *http_socket = (struct http_socket *) us_socket_ext(s);
  struct http_context *http_context = (struct http_context *) us_socket_context_ext(us_socket_get_context(s));

  /* We treat all data events as a response */
  http_socket->offset = us_socket_write(s, request, sizeof(request), 0);

  return s;
}

struct us_socket *on_http_socket_open(struct us_socket *s, int is_client)
{
  puts("on_http_socket_open was called");

  struct http_socket *http_socket = (struct http_socket *) us_socket_ext(s);

  /* Reset offset */
  http_socket->offset = 0;

  /* Send a request */
  us_socket_write(s, request, sizeof(request) - 1, 0);

  return s;
}

struct us_socket *on_http_socket_timeout(struct us_socket *s)
{
  puts("on_http_socket_timeout was called");
  puts("Socket timeout occurred.");

  us_socket_timeout(s, LIBUS_TIMEOUT_GRANULARITY);

  return s;
}

int main()
{
  /* Create the event loop */
  struct us_loop *loop = us_create_loop(1, on_wakeup, on_pre, on_post, 0);

  // Commented out because the SSL portion causes an error.
  // ("error: cannot initialize a variable of type 'struct us_socket_context *'
  //  with an rvalue of type 'struct us_ssl_socket_context *'")
  /* Create a socket context for HTTP */
  // #ifndef LIBUS_NO_SSL
  //   // HTTPS
  //   struct us_ssl_socket_context_options ssl_options = {};
  //   struct us_socket_context *http_context = us_create_ssl_socket_context(loop, 0, ssl_options);
  // #else
  //   // HTTP
        struct us_socket_context *http_context = us_create_socket_context(loop, 0);
  // #endif

  /* Set up event handlers */
  us_socket_context_on_open(http_context, on_http_socket_open);
  us_socket_context_on_data(http_context, on_http_socket_data);
  us_socket_context_on_writable(http_context, on_http_socket_writable);
  us_socket_context_on_close(http_context, on_http_socket_close);
  us_socket_context_on_timeout(http_context, on_http_socket_timeout);
  us_socket_context_on_end(http_context, on_http_socket_end);

  /* Make an HTTP connection */
  us_socket_context_connect(http_context, host, port, 0, sizeof(struct http_socket));

  us_loop_run(loop);

  return 0;
}

This redundant printf breaks console output of my uWebSockets.js application.

On macOS 1014, opening a client connection (with or without SSL) to a port with no listener (i.e. the connection will fail with ECONNREFUSED) results in a crash in gcd.c.

Steps

1. Compile http_load_test.c
2. Run http_load_test 1 localhost xxxxx where xxxxx is a port number with nothing listening

The result is basically the same whether or not the constant SSL is defined as 0 or 1.

Result

* thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x00000001011260ec libdispatch.dylib`_dispatch_lane_resume + 623
  * frame #1: 0x0000000100082dda HelloWorld`us_poll_change(p=0x000060b00000aee0, loop=0x0000608000008820, events=3) at gcd.c:84:9
    frame #2: 0x0000000100096991 HelloWorld`us_socket_write(ssl=0, s=0x000060b00000aee0, data="GET / HTTP/1.0\r\n\r\n", length=1023, msg_more=0) at socket.c:73:9
    frame #3: 0x00000001000010db HelloWorld`on_http_socket_open(s=0x000060b00000aee0, is_client=1, ip=0x0000000000000000, ip_length=0) at http_client_test.c:64:27
    frame #4: 0x00000001000925e7 HelloWorld`us_internal_dispatch_ready_poll(p=0x000060b00000aee0, error=0, events=2) at loop.c:142:17
    frame #5: 0x00000001000819bc HelloWorld`gcd_write_handler(p=0x000060b00000aee0) at gcd.c:48:5
    frame #6: 0x000000010111ef1b libdispatch.dylib`_dispatch_client_callout + 8
    frame #7: 0x00000001011222be libdispatch.dylib`_dispatch_continuation_pop + 563
    frame #8: 0x0000000101136a87 libdispatch.dylib`_dispatch_source_invoke + 2124
    frame #9: 0x0000000101126e76 libdispatch.dylib`_dispatch_lane_serial_drain + 276
    frame #10: 0x0000000101127f4e libdispatch.dylib`_dispatch_lane_invoke + 493
    frame #11: 0x0000000101131824 libdispatch.dylib`_dispatch_root_queue_drain + 334
    frame #12: 0x00000001011322da libdispatch.dylib`_dispatch_worker_thread2 + 125
    frame #13: 0x00000001011980b7 libsystem_pthread.dylib`_pthread_wqthread + 583
    frame #14: 0x0000000101197e01 libsystem_pthread.dylib`start_wqthread + 13

Configuration

uSockets built from tag v0.3.1
macOS 10.14.5
Xcode 11b4

• TCP has two separate lines: read & write, they can be separately frozen.
• TLS does not, you cannot properly freeze read without potentially freezing write.
• We want the common denominator.

us_socket_pause, us_socket_resume, us_ssl_socket_pause, us_ssl_socket_resume for freezing and resuming both lines.

Protocols like HTTP where clients can POST large streams of data are properly solved with this,

HTTP POST sends data until done, and THEN the server writes the response. Throttling upload to server is easily done with us_socket_pause, since the server won't send anything until the upload is done anyways. So it works, most web protocols are not dependent on the ability to freeze only the read or write part of the connection, that's probably why SSL works for these protocols.

Should be simple thing to remove from epoll on pause and add again on resume.

TCP-only extensions for freezing only one part can be added later if needed, as tcp-only functions.

The project I work on very much needs TLS support, but we don't want to use OpenSSL. What would it take to get uSockets to work with mbedTLS?

Looks like it's "simply" a matter of reimplementing the SSL API in internal.h, i.e. rewriting ssl.c ... ? Is that a reasonable project for someone like me who's new to the codebase, but has C++ and networking expertise? Any words of advice or warnings?

They are very similar and could use simple ifdefs where they differ.

Hi I could not find a place to quest under uWebsocket, so I post it here:
On winodws 10, while calling hub.connect to wws, it will block several seconds.
if I add predefine SOCK_CLOEXEC;SOCK_NONBLOCK; to project, error occured with identifier "accept4" not found error.

I am using the hub to connect to multi wws, so I need this "connect" function excute as nonblocking, could you please advice?
thank you!

static inline void internal_finalize_bsd_addr(struct bsd_addr_t *addr) {
    // parse, so to speak, the address
    if (addr->mem.ss_family == AF_INET6) {
        addr->ip = (char *) &((struct sockaddr_in6 *) addr)->sin6_addr;
        addr->ip_length = sizeof(struct in6_addr);
    } else if (addr->mem.ss_family == AF_INET) {
        addr->ip = (char *) &((struct sockaddr_in *) addr)->sin_addr;
        addr->ip_length = sizeof(struct in_addr);
    } else {
        addr->ip_length = 0;
    }
}

You are assuming here that sin6_addr and sin_addr from sockaddr_in6 and sockaddr_in are strings (char *).
They are however just raw bytes: struct in_addr sin_addr; /* Internet address. */
and would need to be converted to strings with inet_ntop.
Example here: https://beej.us/guide/bgnet/html/multi/getpeernameman.html

I'd suggest changing the whole thing like so:

• Remove ip and ip_length from struct bsd_addr_t.
• Remove internal_finalize_bsd_addr.
• Add bsd_addr_to_ipstr which does the addr to ip string conversion.

Like so:

diff --git a/src/interfaces/socket.h b/src/interfaces/socket.h
index e58395b..972e1bb 100644
--- a/src/interfaces/socket.h
+++ b/src/interfaces/socket.h
@@ -46,4 +46,4 @@ WIN32_EXPORT int us_socket_is_closed(struct us_socket *s);
 WIN32_EXPORT struct us_socket *us_socket_close(struct us_socket *s);
 
 /* Copy remote (IP) address of socket, or fail with zero length. */
-WIN32_EXPORT void us_socket_remote_address(struct us_socket *s, char *buf, int *length);
\ No newline at end of file
+WIN32_EXPORT int us_socket_remote_address(struct us_socket *s, char *buf, int length);
\ No newline at end of file
diff --git a/src/internal/common.h b/src/internal/common.h
index 3de04c8..92d1aa6 100644
--- a/src/internal/common.h
+++ b/src/internal/common.h
@@ -74,7 +74,7 @@ struct us_socket_context {
     struct us_socket *iterator;
     struct us_socket_context *prev, *next;
 
-    struct us_socket *(*on_open)(struct us_socket *, int is_client, char *ip, int ip_length);
+    struct us_socket *(*on_open)(struct us_socket *, int is_client, char *ip, int port);
     struct us_socket *(*on_data)(struct us_socket *, char *data, int length);
     struct us_socket *(*on_writable)(struct us_socket *);
     struct us_socket *(*on_close)(struct us_socket *);
diff --git a/src/internal/networking/bsd.h b/src/internal/networking/bsd.h
index 57166b4..cd0d31d 100644
--- a/src/internal/networking/bsd.h
+++ b/src/internal/networking/bsd.h
@@ -34,6 +34,7 @@
 #define _GNU_SOURCE
 #include <sys/types.h>
 #include <sys/socket.h>
+#include <arpa/inet.h>
 #include <netinet/in.h>
 #include <netinet/tcp.h>
 #include <netdb.h>
@@ -99,39 +100,30 @@ static inline void bsd_shutdown_socket(LIBUS_SOCKET_DESCRIPTOR fd) {
 struct bsd_addr_t {
     struct sockaddr_storage mem;
     socklen_t len;
-    char *ip;
-    int ip_length;
 };
 
-static inline void internal_finalize_bsd_addr(struct bsd_addr_t *addr) {
-    // parse, so to speak, the address
-    if (addr->mem.ss_family == AF_INET6) {
-        addr->ip = (char *) &((struct sockaddr_in6 *) addr)->sin6_addr;
-        addr->ip_length = sizeof(struct in6_addr);
-    } else if (addr->mem.ss_family == AF_INET) {
-        addr->ip = (char *) &((struct sockaddr_in *) addr)->sin_addr;
-        addr->ip_length = sizeof(struct in_addr);
-    } else {
-        addr->ip_length = 0;
-    }
-}
-
 static inline int bsd_socket_addr(LIBUS_SOCKET_DESCRIPTOR fd, struct bsd_addr_t *addr) {
     addr->len = sizeof(addr->mem);
     if (getpeername(fd, (struct sockaddr *) &addr->mem, &addr->len)) {
         return -1;
     }
-    internal_finalize_bsd_addr(addr);
     return 0;
 }
 
-static inline char *bsd_addr_get_ip(struct bsd_addr_t *addr) {
-    return addr->ip;
+static inline int bsd_addr_to_ipstr(struct bsd_addr_t *addr, char *str, int str_length) {
+    // parse, so to speak, the address
+    if (addr->mem.ss_family == AF_INET6) {
+        struct sockaddr_in6 *s = (struct sockaddr_in6 *)addr;
+        inet_ntop(AF_INET6, &s->sin6_addr, str, str_length);
+        return ntohs(s->sin6_port);
+    } else if (addr->mem.ss_family == AF_INET) {
+        struct sockaddr_in *s = (struct sockaddr_in *)addr;
+        inet_ntop(AF_INET, &s->sin_addr, str, str_length);
+        return ntohs(s->sin_port);
+    }
+    return -1;
 }
 
-static inline int bsd_addr_get_ip_length(struct bsd_addr_t *addr) {
-    return addr->ip_length;
-}
 
 // called by dispatch_ready_poll
 static inline LIBUS_SOCKET_DESCRIPTOR bsd_accept_socket(LIBUS_SOCKET_DESCRIPTOR fd, struct bsd_addr_t *addr) {
@@ -146,8 +138,6 @@ static inline LIBUS_SOCKET_DESCRIPTOR bsd_accept_socket(LIBUS_SOCKET_DESCRIPTOR
     accepted_fd = accept(fd, (struct sockaddr *) addr, &addr->len);
 #endif
 
-    internal_finalize_bsd_addr(addr);
-
     return apple_no_sigpipe(accepted_fd);
 }
 
diff --git a/src/libusockets_new.h b/src/libusockets_new.h
index 2259f49..f873f24 100644
--- a/src/libusockets_new.h
+++ b/src/libusockets_new.h
@@ -255,7 +255,7 @@ inline static int us_new_socket_is_shut_down(const int ssl, struct us_new_socket
 #endif
 }
 
-inline static void us_new_socket_remote_address(const int ssl, struct us_new_socket_t *s, char *buf, int *length) {
+inline static void us_new_socket_remote_address(const int ssl, struct us_new_socket_t *s, char *buf, int length) {
     /* There is no SSL variant of this function */
     us_socket_remote_address((struct us_socket *) s, buf, length);
 }
diff --git a/src/loop.c b/src/loop.c
index 4f1ed32..aa316ff 100644
--- a/src/loop.c
+++ b/src/loop.c
@@ -187,7 +187,9 @@ void us_internal_dispatch_ready_poll(struct us_poll *p, int error, int events) {
                         // make sure to link this socket into its context!
                         us_internal_socket_context_link(listen_socket->s.context, s);
 
-                        listen_socket->s.context->on_open(s, 0, bsd_addr_get_ip(&addr), bsd_addr_get_ip_length(&addr));
+                        char addr_str[INET6_ADDRSTRLEN];
+                        int port = bsd_addr_to_ipstr(&addr, addr_str, sizeof(addr_str));
+                        listen_socket->s.context->on_open(s, 0, addr_str, port);
                     } while ((client_fd = bsd_accept_socket(us_poll_fd(p), &addr)) != LIBUS_SOCKET_ERROR);
                 }
             }
diff --git a/src/socket.c b/src/socket.c
index 1f0b65b..e208027 100644
--- a/src/socket.c
+++ b/src/socket.c
@@ -23,14 +23,11 @@ void us_internal_init_socket(struct us_socket *s) {
     // shared nodelay here?
 }
 
-void us_socket_remote_address(struct us_socket *s, char *buf, int *length) {
+int us_socket_remote_address(struct us_socket *s, char *buf, int length) {
     struct bsd_addr_t addr;
-    if (bsd_socket_addr(us_poll_fd(&s->p), &addr) || *length < bsd_addr_get_ip_length(&addr)) {
-        *length = 0;
-    } else {
-        *length = bsd_addr_get_ip_length(&addr);
-        memcpy(buf, bsd_addr_get_ip(&addr), *length);
-    }
+    if (bsd_socket_addr(us_poll_fd(&s->p), &addr))
+        return -1;
+    return bsd_addr_to_ipstr(&addr, buf, length);
 }
 
 int us_socket_write(struct us_socket *s, const char *data, int length, int msg_more) {

I'd also love to have a function that returns the bsd_addr_t / sockaddr instead of a string so I don't have to convert the string back to a sockaddr again if I need that.
But it seems like you are trying to keep all the OS stuff separate so I guess can't return a sockaddr anywhere in the API.

This should be possible to add without changing any of the higher level interfaces. A TCP socket should look to the application exactly like a Unix socket does.

Lower priority but still perfectly in scope of this project.

uWebsockets can be built using:

make 
sudo make install

and included with

#include <uWS/uWS.h>

But uSockets doesn't have the same steps & the Makefile only builds demo applications. Can you provide guidance on to build/install uSockets so that it can be used in a user application via #include <libusockets.h> please?

I can submit your response as a PR to update the README. Thanks.

Edit: Goal is to use with a C++ project also containing uWebsockets.

Allegedly there's a bug where on_open is called followed by on_close when connect failes. Needs to be checked.

this might be of some fun for you