e.g. error messages signal: killed and exit status 1 (iirc)

Why do we even get them again?

Does it affect sources/sinks too?

• Put source/sink name in screen name
• Put screen PID in log messages
• Log message shows wrong screen name

autotee has been used in production more than once now. It's time to stabilize it. I need to think about what I want to have in a 1.0 release (and maintain forever).

https://deferpanic.com/blog/understanding-golang-memory-usage/

restart_when_sink_dies was true

Currently it's implemented like this in src/autotee/util_cmd.go:

// Graceful termination. Blocking!
func (c *Cmd) End() error {

    // Ignore errors; process may already be dead
    _ = syscall.Kill(c.cmd.Process.Pid, syscall.SIGTERM)
    time.Sleep(250 * time.Millisecond)
    _ = syscall.Kill(c.cmd.Process.Pid, syscall.SIGKILL)
    return <-c.WaitChannel()
}

The hardcoded sleep time that can't finish early is not optimal.

A better way would be to wait for either the timeout or a SIGPIPE, then wait4 in the signal handler. There are a couple of difficulties to overcome though:

• There'd have to be only one goroutine doing all the wait4-ing because otherwise other goroutines doing graceful termination might miss their notification.
• I think (I'm not sure atm) we also couldn't use normal Wait() anywhere anymore. Can we guarantee that no library ever Wait()s?
• There is no good way to wait until the process and all of its children have died. Is it ok if we just ensure the parent process dies?
• Does Wait() in go do more than the syscall? Is it even ok if we don't wait via gos Wait() function?

Or is there a better (safe!) way to implement graceful termination?

Currently, all buffers in the buffer pool are fixed size. Maybe we could allocate variable-size buffers?

We'd have to be very careful not to harm performance (e.g. due to fragmentation, or allocation overhead, or deallocation cost / garbage collection). Maybe a custom allocator (e.g. a buddy allocator) would help, or maybe we should try using Gos allocator. We'd definitely have to do benchmarks.

a) we don't implement the server but the client
b) it's not even a fully featured client, it just gets the list of streams
c) doesn't fit in well with stream lists that arent retrieved from servers

=> something like "StreamList" would be more appropriate

Question: does killing the screen cause correct restarts?

This should be documented.

autotee has been written very, very carefully, knowing that it'd be used in production and must never crash.

However manual testing can only go so far.

Depends on #1

Currently, only if an entire parameter equals {stream}, it's replaced by the stream name.

Though maybe we should switch to something like ${stream} then, while we're at it?

That might make everything somewhat smoother.

However, it'd have to be MUCH smaller than bpc.BufferSize, otherwise things might get too unpredictable.

Before committing the feature, test if it actually improves performance.

Also, test both with and without source stall detection enabled.

Currently graceful termination is only being done for screens.

Related to #3 -- for the sources/sinks, I think we really want to be sure all child processes are terminated.

Things to think about:

• Maybe we could then move nginx-rtmp / icecast support out of autotee too.
• I'd like to avoid constantly forking.
  • And I don't want to invent a custom protocol for long-running stream list providers.
• Should it really be pull based? Wouldn't it make sense to expose an API instead?
  • Hard to deploy / use? Not robust? Security problem? Complicated to write (need heartbeat or client could die and we'd never notice changes to streams anymore...)

To allow paths relative to the config file.

Cancellations and log context are currently implemented in a somewhat ad-hoc way (cancellation: channel closing + barriers, log context: by just passing the existing log context explicitly).

https://blog.golang.org/context may provide a better, or at least more standardized, way.

Maybe we could also do dependency injection that way, solving #1

Many options are not obvious. Especially the timers.

Currently only influxdb is supported.

Log why a sink was terminated. We had one source/sink combination that always failed to get started. (Maybe something buffer size related?)

Whats wrong

The current model (depending on reuse_screens config var that is either one screen per kind of process or one screen per process) doesn't work too well:

1. There are many screens. There is no quick way to switch between related screens.
2. Users could go into screens and ctrl-c the processes. autotee will just restart them, but...
3. Screens tend to look like gibberish because their size isn't explicitly set (and can't be without synthesizing screen config files).
4. When processes quit, the screen is thrown away (unless it's reused, anyway), so the screens don't help for troubleshooting exiting processes.

How to improve

Idea 1

We could remove screen support entirely and just rely on log files.

Problem: we wouldn't be able to see ffmpegs interactive progress display anymore.

Idea 2

We could use one screen per flow, with subwindows for processes.

This would not solve most of the problems mentioned above.

It would also mean we'd have to find another way to give the PTYs to the screens. We can't just generate a screen config file because processes (for individual screen windows) need to be restarted dynamically.

Zero-copy is the single most promising performance improvement. At the moment we clone:

1. once per source from a kernel buffer to a user space buffer,
2. and then once per sink back to a kernel buffer

See relevant todo in code:

A small source_timeout is desirable to minimize downtimes caused by source stalls.

However, sources may take some time to start.