$ go get -u github.com/facebookgo/grace
# cd /Users/stevejiang/developments/go/src/github.com/facebookgo/grace; git pull --ff-only
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
package github.com/facebookgo/grace: exit status 1

I'm on Mac OSX

Hi first of all thanks for building grace,

I was wondering if you had though about tagging and releasing semver versions of grace ?

Shipping semver releases makes it easy for users of the library to use a specific version and not be affected by accidental breaking changes (this can always be done by using an explicit SHA, but a semver tag is arguably nicer).

I'm running into a scenario where processes are hanging around after being sent USR2. I've pinpointed why, but I'm not sure the best way to fix it. Here's what I'm seeing (this can be seen using the demo app).

Process starts, PID: 58605, PPID: 50422 (the shell)
USR2 -> 58605

Spawns 58606, PPID: 58605
Kills 58605

USR2 -> 58606

Spawns 58609, PPID: 58606
Tries to kill 58606

At this point, since the original process (58605) is gone, 58606 now has a PPID of 1. This is causing this line: https://github.com/ParsePlatform/go.grace/blob/master/grace.go#L84 to run the cleanup stuff for an init-spawned process, rather than a regular process. This of course means the listener is never close()d and the wg.Wait() in that function never returns. This leaves process 58606 hanging around. Continuing to USR2 the "current" process will continue this behavior and several processes and zombies can be left hanging around. Here's an example after USR2'ing several times:

$ ps auxw|grep demo
scott           58642   0.0  0.0        0      0 s000  Z     6:12PM   0:00.00 (demo)
scott           58628   0.0  0.0        0      0 s000  Z     6:12PM   0:00.00 (demo)
scott           58622   0.0  0.0        0      0 s000  Z     6:11PM   0:00.00 (demo)
scott           58609   0.0  0.0        0      0 s000  Z     6:06PM   0:00.00 (demo)
scott           58645   0.0  0.0 145310244   3968 s000  S     6:13PM   0:00.02 ./demo
scott           58637   0.0  0.0 145312380   4100 s000  S     6:12PM   0:00.01 ./demo
scott           58625   0.0  0.0 145312380   4100 s000  S     6:12PM   0:00.01 ./demo
scott           58619   0.0  0.0 145311356   4088 s000  S     6:11PM   0:00.01 ./demo
scott           58606   0.0  0.0 145312384   4144 s000  S     6:05PM   0:00.02 ./demo

Would it be fair for a process to check on start up whether it is init spawned and store that knowledge rather than relying on the PPID at kill time?

An example unix socket server:
https://gist.github.com/teknoraver/5ffacb8757330715bcbcc90e6d46ac74

The punch line is:

	server := http.Server{
		Handler: http.FileServer(http.Dir(root)),
	}

	unixListener, err := net.Listen("unix", os.Args[1])
	if err != nil {
		panic(err)
	}
	server.Serve(unixListener)

I tried this here and sadly our types don't match net.http's interface.
From: https://golang.org/pkg/net/http/#Server
func (srv *Server) Serve(l net.Listener) error

However here, we're asking for func Serve(servers ...*http.Server)
and then we hardcode tcp

		l, err := a.net.Listen("tcp", s.Addr)

here in listen

It seems like we should match net.http here and take a https://golang.org/pkg/net/#Listener which would allow us to grab https://golang.org/pkg/net/#Addr so we can open tcp/udp/unix as appropriate.

I could be missing something, but I didn't see anything too TCP specific in the fork/exec/inherit code.

My use case is that I want to front multiple versions of the same service with nginx, and I don't want to play games with ports.
While that particular case is admittedly a little wacky, fronting a service with nginx or the like isn't too unreasonable (say for handling auth somewhere else).

As far as I can tell, there's no mechanism that allows anybody outside of the program to track the server's current PID so that upstart/init.d can know which process to send the signal to.

I could a PR that allows for a hook to fire (eg, so we could write a pidfile) when the new server is accepting requests, but I feel like I'm missing something obvious (or perhaps missing from non-systemd based systems)

Steps to reproduce (Go 1.1.2, OS X 10.8.5):

1. gracedemo
2. curl 'http://localhost:48567/sleep/?duration=30s'
3. kill -USR2 $GRACEDEMOPID
4. observe proper graceful restart; curl 'http://localhost:48567/sleep/?duration=1s' works fine
5. kill $GRACEDEMOPID_NEW
6. gracedemo process still alive; curl 'http://localhost:48567/sleep/?duration=1s' hangs

grace now works with listening sockets. Is it possible to make it work with connections as well. Are there some pitfalls?

After received USR2, the new processes will run in background, in that case, supervisord (A Process Control System) can't track the new processes. Is there any chance to add an option to force the new processes run in foreground?

Running a go get github.com/facebookgo/grace/gracehttp gives me:

# github.com/facebookgo/grace/gracehttp
c:\gobits\src\github.com\facebookgo\grace\gracehttp\http.go:101: undefined: syscall.SIGUSR2
c:\gobits\src\github.com\facebookgo\grace\gracehttp\http.go:111: undefined: syscall.SIGUSR2
c:\gobits\src\github.com\facebookgo\grace\gracehttp\http.go:151: undefined: syscall.Kill

It will be nice to be able to configure StopTimeout and KillTimeout for gracehttp: https://github.com/facebookgo/grace/blob/master/gracehttp/http.go#L41.

I didn't notice anything specific in the code that would guarantee that the exiting process would wait for the goroutines to be done before exiting.
Did I miss something or is it not currently handled by the library?

So far, this lib does exactly what I need and it was really easy to migrate my project to it, however I use Go for web services and some endpoints start goroutines (to upload assets to S3 for instance) and I'm afraid that go.grace wouldn't let these goroutines finish before existing.

Thanks

At the StartProcess function, while Listener' s File() is called, this will puts the Listener fd into blocking mode. This should lead to Accept() can't return even if Listener.Close() is called.
Listener File() --> (fd *netFD) dup()

func (fd *netFD) dup() (f *os.File, err error) {
    ns, err := dupCloseOnExec(fd.pfd.Sysfd)
    if err != nil {
        return nil, err 
    }   

    // We want blocking mode for the new fd, hence the double negative.
    // This also puts the old fd into blocking mode, meaning that
    // I/O will block the thread instead of letting us use the epoll server.
    // Everything will still work, just with more threads.
    if err = fd.pfd.SetBlocking(); err != nil {
        return nil, os.NewSyscallError("setnonblock", err)
    }   

    return os.NewFile(uintptr(ns), fd.name()), nil 
}

So, User either use Shutdown to close socket, or use syscall.SetNonblock to put the Listener fd into the nonblock.

Notably investigate:

https://github.com/daaku/go.grace/blob/master/grace.go#L97
https://github.com/daaku/go.grace/blob/master/grace.go#L113
https://github.com/daaku/go.grace/blob/master/grace.go#L126

Could you please indicate this software license please?

How to use https api ListenAndServeTLS

I think it'd be very useful to be able to have gracehttp.Serve(servers...) to take an additional argument for the custom net.Listener to use for handling low-level network connections. The method signature can look like:

func ServeWith(l net.Listener, servers ...*http.Server) {
  // Implementation details.
}

I can make a PR for this if this sounds like an entertaining idea.

Gracedemo process only restarts once:

1. gracedemo
2. curl 'http://localhost:48567/sleep/?duration=30s'
3. kill -USR2 $GRACEDEMOPID
4. observe proper graceful restart; curl 'http://localhost:48567/sleep/?duration=1s' works fine
5. kill -USR2 $GRACEDEMOPID_NEW
6. gracedemo process terminates instead of restarting again

Macbook Pro 10.11 El Capitan, Go 1.5.1

I've using grace for my project together with echo (v1). It works great and I love it 😃
Since v2, echo add support for fasthttp.
Switching between standard net/http and fasthttp is easy, but grace won't work with fasthttp.
Is there any plan to support fasthttp?

Would like to hear your feedback :)

It would be an improvement in my opinion to expose the logger object such that it's prefix can be customized for the application including gracehttp.

Hi,

We are using grace to good effect over here so first of all, thanks!

https://github.com/whosonfirst/go-whosonfirst-pip/

However when we try to bundle grace (and its friends) as vendored dependencies they don't appear to be being saved:

https://github.com/whosonfirst/go-whosonfirst-pip/tree/master/vendor/src/github.com/facebookgo

Is this is a limitation that FB has imposed on these libraries or am I just doing it wrong (always possible...) ?

Hi， I have a question. How could I grace restart the websocket server without disconnection the connection.

I need some help, Thx~

Is there any reason SIGINT is not also captured, with the same behavior as SIGTERM?
Or no ability to set which signals you want to capture?

Hi there. Gracehttp is restarting beautifully, but I can't figure out a way to make it work with Upstart. Once the original process stops, Upstart loses track of the process and reports it as 'not working'. I've spent a fair few hours meddling with settings, and I've tried calling start-stop-daemon (with --name and --startas instead of --pidfile), but to no avail.

Do you have any recommendations, or do you know anyone who's got it working?

Thanks

What's the prefered way to set the TermTimeout value when using grace/gracehttp? The defaultProcess var is unexported, so I'm having trouble figuring out how to set it without writing my own http wrapper.

This is related to #17 and #12, but wasn't specifically addressed there. Could you add a PIDFile that users could point systemd or other at to deal with spawning off new processes?

after graceNet.StartProcess(), process may exec some cmd. Carrying env variable LISTEN_FDS are unnecessary.
In my process, the cmd is start another grace net process which cause error ERROR: error inheriting socket fd 3: file filetnet listener: getsockopt: socket operation on non-socket

When I first restart although it can behave normally, but the binary content is empty. After restart twice
an error has occurred "fork/exec /var/folders/5q/_pswr22n39b4xj740dc41y2c0000gn/T/go-build783500014/b001/exe/demo: no such file or directory"。

Is it possible to configure an HTTPS web service with your packages?

I follow the steps from readme but it get:

SIGSYS: bad system call
PC=0x1056f03 m=0 sigcode=0

goroutine 0 [idle]:
runtime.mach_semaphore_timedwait(0x1300000e03, 0x3b9ac413, 0x7fff5fbfe554, 0x13, 0x3b9ac4135fbfe570, 0x13f95a0, 0x7fff5fbfe588, 0x1051443, 0x4a817c213, 0x103230a, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/sys_darwin_amd64.s:425 +0x13
runtime.semasleep1(0x4a817c213, 0x103230a)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/os_darwin.go:402 +0xe1
runtime.semasleep.func1()
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/os_darwin.go:432 +0x33
runtime.systemstack(0x13f9500)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/asm_amd64.s:327 +0x79
runtime.mstart()
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/proc.go:1132

goroutine 3 [syscall]:
runtime.notetsleepg(0x13f8d98, 0x4a817c213, 0x16)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/lock_sema.go:257 +0x4b fp=0xc420031f60 sp=0xc420031f20
runtime.timerproc()
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/time.go:209 +0x327 fp=0xc420031fe0 sp=0xc420031f60
runtime.goexit()
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420031fe8 sp=0xc420031fe0
created by runtime.addtimerLocked
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/time.go:116 +0xed

goroutine 1 [select]:
github.com/facebookgo/grace/gracehttp.(*app).run(0xc420078230, 0x3, 0x3)
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:165 +0x403
github.com/facebookgo/grace/gracehttp.Serve(0xc4200e24c0, 0x3, 0x3, 0xc420084d50, 0x0)
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:193 +0x4d
main.main()
/Users/sheerun/go/src/github.com/sheerun/demo/main.go:28 +0x24e

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/asm_amd64.s:2197 +0x1

goroutine 20 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/sigqueue.go:116 +0xff
os/signal.loop()
/usr/local/Cellar/go/1.8.3/libexec/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/usr/local/Cellar/go/1.8.3/libexec/src/os/signal/signal_unix.go:28 +0x41

goroutine 21 [select]:
github.com/facebookgo/httpdown.(*server).manage(0xc4200bc4d0)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:204 +0xa2b
created by github.com/facebookgo/httpdown.HTTP.Serve
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:96 +0x3df

goroutine 22 [IO wait]:
net.runtime_pollWait(0x1619038, 0x72, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420078308, 0x72, 0x0, 0xc420144000)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420078308, 0xffffffffffffffff, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4200782a0, 0x0, 0x13ca380, 0xc420144000)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42009c020, 0xc420042eb0, 0x11f680e, 0x1052350)
/usr/local/Cellar/go/1.8.3/libexec/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42009c020, 0x12c2660, 0xc420140000, 0x13ce4c0, 0xc42010a0c0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4200bc2c0, 0x13cdd80, 0xc42009c020, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:2643 +0x228
github.com/facebookgo/httpdown.(*server).serve(0xc4200bc4d0)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:277 +0x83
created by github.com/facebookgo/httpdown.HTTP.Serve
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:97 +0x401

goroutine 23 [select]:
github.com/facebookgo/httpdown.(*server).manage(0xc4200bc580)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:204 +0xa2b
created by github.com/facebookgo/httpdown.HTTP.Serve
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:96 +0x3df

goroutine 24 [IO wait]:
net.runtime_pollWait(0x1618f78, 0x72, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420078378, 0x72, 0x0, 0xc42000c4e0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420078378, 0xffffffffffffffff, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420078310, 0x0, 0x13ca380, 0xc42000c4e0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42009c028, 0xc42001c3c0, 0x125fb00, 0x13ef9b0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42009c028, 0xc42001c390, 0x125fb00, 0x13ef9b0, 0x1287f80)
/usr/local/Cellar/go/1.8.3/libexec/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4200bc370, 0x13cdd80, 0xc42009c028, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:2643 +0x228
github.com/facebookgo/httpdown.(*server).serve(0xc4200bc580)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:277 +0x83
created by github.com/facebookgo/httpdown.HTTP.Serve
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:97 +0x401

goroutine 25 [select]:
github.com/facebookgo/httpdown.(*server).manage(0xc4200bc630)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:204 +0xa2b
created by github.com/facebookgo/httpdown.HTTP.Serve
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:96 +0x3df

goroutine 26 [IO wait]:
net.runtime_pollWait(0x1618eb8, 0x72, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4200783e8, 0x72, 0x0, 0xc42000c500)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4200783e8, 0xffffffffffffffff, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420078380, 0x0, 0x13ca380, 0xc42000c500)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42009c030, 0xc42001c4b0, 0x125fb00, 0x13ef9b0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42009c030, 0xc42001c480, 0x125fb00, 0x13ef9b0, 0x1287f80)
/usr/local/Cellar/go/1.8.3/libexec/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4200bc420, 0x13cdd80, 0xc42009c030, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:2643 +0x228
github.com/facebookgo/httpdown.(*server).serve(0xc4200bc630)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:277 +0x83
created by github.com/facebookgo/httpdown.HTTP.Serve
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:97 +0x401

goroutine 27 [semacquire]:
sync.runtime_Semacquire(0xc42007f0bc)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc42007f0b0)
/usr/local/Cellar/go/1.8.3/libexec/src/sync/waitgroup.go:131 +0x7a
github.com/facebookgo/grace/gracehttp.(*app).wait(0xc420078230)
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:88 +0x115
github.com/facebookgo/grace/gracehttp.(*app).run.func1(0xc42007cba0, 0xc420078230)
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:162 +0x51
created by github.com/facebookgo/grace/gracehttp.(*app).run
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:163 +0x1e1

goroutine 28 [chan receive]:
github.com/facebookgo/grace/gracehttp.(*app).signalHandler(0xc420078230, 0xc42007f0b0)
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:106 +0x127
created by github.com/facebookgo/grace/gracehttp.(*app).wait
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:79 +0x81

goroutine 29 [chan receive]:
github.com/facebookgo/httpdown.(*server).Wait(0xc4200bc4d0, 0x12c2c80, 0xc42007f0b0)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:283 +0x5b
github.com/facebookgo/grace/gracehttp.(*app).wait.func1(0xc42007f0b0, 0xc420078230, 0x13cc280, 0xc4200bc4d0)
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:83 +0x5b
created by github.com/facebookgo/grace/gracehttp.(*app).wait
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:86 +0xdc

goroutine 30 [chan receive]:
github.com/facebookgo/httpdown.(*server).Wait(0xc4200bc580, 0x12c2c80, 0xc42007f0b0)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:283 +0x5b
github.com/facebookgo/grace/gracehttp.(*app).wait.func1(0xc42007f0b0, 0xc420078230, 0x13cc280, 0xc4200bc580)
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:83 +0x5b
created by github.com/facebookgo/grace/gracehttp.(*app).wait
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:86 +0xdc

goroutine 34 [select, locked to thread]:
runtime.gopark(0x12c2bc0, 0x0, 0x12b3268, 0x6, 0x18, 0x2)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc420132f50, 0x0, 0x18)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc420132f50)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/signal_unix.go:434 +0x265
runtime.goexit()
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/asm_amd64.s:2197 +0x1

goroutine 31 [chan receive]:
github.com/facebookgo/httpdown.(*server).Wait(0xc4200bc630, 0x12c2c80, 0xc42007f0b0)
/Users/sheerun/go/src/github.com/facebookgo/httpdown/httpdown.go:283 +0x5b
github.com/facebookgo/grace/gracehttp.(*app).wait.func1(0xc42007f0b0, 0xc420078230, 0x13cc280, 0xc4200bc630)
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:83 +0x5b
created by github.com/facebookgo/grace/gracehttp.(*app).wait
/Users/sheerun/go/src/github.com/facebookgo/grace/gracehttp/http.go:86 +0xdc

goroutine 50 [sleep]:
time.Sleep(0x4a817c800)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/time.go:59 +0xf9
main.newHandler.func1(0x13cdf80, 0xc420156000, 0xc42014e000)
/Users/sheerun/go/src/github.com/sheerun/demo/main.go:39 +0xf6
net/http.HandlerFunc.ServeHTTP(0xc4200e2400, 0x13cdf80, 0xc420156000, 0xc42014e000)
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:1942 +0x44
net/http.(*ServeMux).ServeHTTP(0xc420084c90, 0x13cdf80, 0xc420156000, 0xc42014e000)
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:2238 +0x130
net/http.serverHandler.ServeHTTP(0xc4200bc2c0, 0x13cdf80, 0xc420156000, 0xc42014e000)
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:2568 +0x92
net/http.(*conn).serve(0xc420140000, 0x13ce400, 0xc420138080)
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:1825 +0x612
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:2668 +0x2ce

goroutine 51 [IO wait]:
net.runtime_pollWait(0x1618df8, 0x72, 0x7)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42013a068, 0x72, 0x13cb7c0, 0x13c8428)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42013a068, 0xc4201380d1, 0x1)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc42013a000, 0xc4201380d1, 0x1, 0x1, 0x0, 0x13cb7c0, 0x13c8428)
/usr/local/Cellar/go/1.8.3/libexec/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42013e000, 0xc4201380d1, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/net.go:181 +0x70
net/http.(*connReader).backgroundRead(0xc4201380c0)
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:656 +0x58
created by net/http.(*connReader).startBackgroundRead
/usr/local/Cellar/go/1.8.3/libexec/src/net/http/server.go:652 +0xdf

rax 0xe
rbx 0x13
rcx 0x7fff5fbfe528
rdx 0x3b9ac413
rdi 0xe03
rsi 0x13
rbp 0x7fff5fbfe560
rsp 0x7fff5fbfe528
r8 0xc420001860
r9 0x1c9
r10 0xa30261364
r11 0x202
r12 0x539da43c93d68
r13 0x54f42816f8ac0
r14 0x14c2cc2a3b89b800
r15 0xf3
rip 0x1056f03
rflags 0x202
cs 0x7
fs 0x0
gs 0x0

Steps to reproduce

Consider following code

package main

import (
	"flag"
	"fmt"
	"github.com/facebookgo/grace/gracehttp"
	"io/ioutil"
	"net/http"
)

var cmd = flag.String("cmd", "", "cmd")

func serverCmd() {
	mux := http.NewServeMux()
	mux.HandleFunc("/",
		func(w http.ResponseWriter, r *http.Request) {
			w.Write([]byte("hello world\n"))
		})

	server := &http.Server{
		Addr:    "127.0.0.1:9228",
		Handler: mux,
	}

	if err := gracehttp.Serve(server); err != nil {
		fmt.Printf("error serve: %v\n", err)
	}
}

func clientCmd() {
	client := http.Client{}
	for {
		request, _ := http.NewRequest("POST", "http://127.0.0.1:9228", nil)
		//request.Close = true
		resp, err := client.Do(request)
		if err != nil {
			fmt.Printf("http.Post error: %v", err)
			return
		}
		ioutil.ReadAll(resp.Body)
		resp.Body.Close()
	}
}

func main() {
	flag.Parse()
	switch *cmd {
	case "server":
		serverCmd()
	case "client":
		clientCmd()
	default:
		fmt.Printf("unknown cmd %v", *cmd)
	}
}

Start server:

go build -o main
./main -cmd=server

And client:

go run main.go -cmd=client

And now make a couple of server reloads:

kill -SIGUSR2 $(pgrep -f "main -cmd=server")

It will cause client failures with one of the following errors:

Post http://127.0.0.1:9228: read tcp 127.0.0.1:35022->127.0.0.1:9228: read: connection reset by peer

or

Post http://127.0.0.1:9228: EOF

Environment

go version go1.9.1 linux/amd64

Notes

I manage to reproduce it with perl client and wrk as well, so this is not a client issue.

Adding request.Close = true in client code fixes the issue, so probably it is related to persistent connections.