juliadocs / iocapture.jl Goto Github PK

After #20 is merged, there's a second part for my ultimate motivation to capture only the first and last KB or so of some code that otherwise produces a couple of 100MB of output.

I can do this quite simply by replacing the output=IOBuffer() in line 107 with a custom IOBuffer-like object FirstLastBuffer defined as follows:

mutable struct FirstLastBuffer
    first::Vector{UInt8}
    last::Vector{UInt8}
    N::Int64
    written::Int64
end


function FirstLastBuffer(N=1024)
    first = Vector{UInt8}(undef, N)
    last = Vector{UInt8}(undef, N)
    FirstLastBuffer(first, last, N, 0)
end


function Base.write(buffer::FirstLastBuffer, data)
    for byte in data
        if buffer.written < buffer.N
            i = buffer.written + 1
            buffer.first[i] = byte
        else
            i = mod((buffer.written - buffer.N), buffer.N) + 1
            buffer.last[i] = byte
        end
        buffer.written += 1
    end
end


function Base.take!(buffer::FirstLastBuffer)
    N = buffer.N
    if buffer.written <= N
        result =  buffer.first[1:buffer.written]
    else
        written_last = buffer.written - N
        last = view(buffer.last, 1:min(written_last, N))
        if written_last <= N
            sep = Vector{UInt8}[]
            length_result = buffer.written
            shift = 0
        else
            sep = Vector{UInt8}("…\n…")
            length_result = 2 * N + length(sep)
            shift = - mod(buffer.written - buffer.N, buffer.N)
        end
        result = Vector{UInt8}(undef, length_result)
        result[1:N] .= buffer.first
        result[N+1:N+length(sep)] .= sep
        result[N+length(sep)+1:end] .= circshift(last, shift)
    end
    buffer.written = 0
    return result
end

(tested, but not very thoroughly, so may still be buggy). The methods write and take! are the only ones used by IOCapture, so that above code is all that's required.

In order to replace the internal output variable, at a minimum, I'd have to add a hidden/undocumented _output keyword argument to IOCapture.capture.

Of course, I could also just copy the entire 150 lines of the IOCapture code into my own package to make that small modification, but it seems like such a trivial change. So, would you mind doing me a favor by letting me make a PR for the _output keyword argument?

Alternatively, if you're actively interested in providing some capabilities within the package to only capture parts of the output, there's a number of options:

I could make that keyword argument an officially documented feature. We'd have to come up with a name for the keyword argument: probably not _output/output, maybe something like capture_buffer.

Or, I could go all the way, and contribute the entire above code for FirstLastBuffer to IOCapture. This would go with a new keyword argument, maybe limit=<number of bytes to capture>. If given, instead of using an IOBuffer(), IOCapture.capture would automatically use a FirstLastBuffer(limit ÷ 2). It's a bit opinionated in that it captures the beginning and end. Depending on their application, others might conceivably want limit to discard everything but the first part of the output, or keep only the last part of the output. Those use cases would be better-served by the capture_buffer option.

The two possibilities could be combined, too: capture_buffer could receive a type, not an object, which would be IOBuffer by default if limit=nothing and FirstLastBuffer if limit is not nothing. If the user supplies their own capture_buffer type, it will be instantiated with capture_buffer(limit) if limit is given or capture_buffer() otherwise.

Anyway, I'm open to all of the above possibilities, so please let me know what you think.

This currently fails:

using IOCapture, Test
function test_puts(str)
    cap = IOCapture.capture() do
        @ccall puts(str::Cstring)::Cint
    end
    @test cap.value == length(str) + 1
end
test_puts("Hello World\n"^100_000)

For Julia-only code using e.g. print this works fine.

Using Libc.flush_cstdio() doesn't seem to help:

function test_puts_flush(str)
    cap = IOCapture.capture() do
        ret = @ccall puts(str::Cstring)::Cint
        Libc.flush_cstdio()
        ret
    end
    @test cap.value == length(str) + 1
end
test_puts_flush("Hello World\n"^100_000)

Related: JuliaIO/Suppressor.jl#46 and JuliaIO/Suppressor.jl#47

Hello,

I currently use IOCapture as follows and it works well:

try
    captured = IOCapture.capture() do
        include_string(...)
    end
    # ... processing of captured ...
catch e
    # ...
end

I'd like to be able to do this in a multi-threaded context where these calls potentially happen on separate threads. I can add a ReentrantLock and do

lock(l)
try
   captured[k] = IOCapture.capture() do ... end
catch
   ...
finally
    unlock(l)
end

but this amounts to doing these evaluations sequentially as far as I understand. Is there a better way to capture output of separate threads with IOCapture?

Thanks

Note: I also asked this question in the Suppressor repo (JuliaIO/Suppressor.jl#45) not sure which one of the two would be more amenable to doing this.

I have run into an issue similar to that fixed in #15 and reported in JuliaDocs/Documenter.jl#1947

IOCapture hangs the first time it encounters a W_cmd macro as defined in MathLink.

Here is a minimal Project.toml. IOCapture 0.2.3 includes #15.

[deps]
IOCapture = "b5f81e59-6552-4d32-b1f0-c071b021bf89"
MathLink = "18c93696-a329-5786-9845-8443133fa0b4"

[compat]
IOCapture = "0.2.3"
MathLink = "0.5.1"
julia = "1.9.2"

Here is a demonstration in the REPL. I used ctrl-C to kill the second command after it got stuck. You can see the backtrace. Running the same command a second time works.

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.2 (2023-07-05)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using IOCapture, MathLink

julia> c = IOCapture.capture() do
           W`Sin`
       end
^CERROR: InterruptException:
Stacktrace:
  [1] poptask(W::Base.IntrusiveLinkedListSynchronized{Task})
    @ Base ./task.jl:974
  [2] wait()
    @ Base ./task.jl:983
  [3] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait(c::Base.GenericCondition{Base.Threads.SpinLock})
    @ Base ./condition.jl:125
  [5] _wait(t::Task)
    @ Base ./task.jl:308
  [6] wait
    @ ./task.jl:347 [inlined]
  [7] (::IOCapture.var"#3#5"{DataType, var"#3#4", Task, IOContext{Base.PipeEndpoint}, IOContext{Base.PipeEndpoint}, Base.TTY, Base.TTY})()
    @ IOCapture ~/.julia/packages/IOCapture/8Uj7o/src/IOCapture.jl:130
  [8] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:514
  [9] with_logger
    @ ./logging.jl:626 [inlined]
 [10] capture(f::var"#3#4"; rethrow::Type, color::Bool)
    @ IOCapture ~/.julia/packages/IOCapture/8Uj7o/src/IOCapture.jl:116
 [11] capture(f::Function)
    @ IOCapture ~/.julia/packages/IOCapture/8Uj7o/src/IOCapture.jl:72
 [12] top-level scope
    @ REPL[2]:1

julia> c = IOCapture.capture() do
           W`Sin`
       end
(value = W"Sin", output = "", error = false, backtrace = Ptr{Nothing}[])

julia>

Using the W command once before using within a IOCapture context works as expected.

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.2 (2023-07-05)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using IOCapture, MathLink
[ Info: Precompiling IOCapture [b5f81e59-6552-4d32-b1f0-c071b021bf89]

julia> W`Sin`
W"Sin"

julia> c = IOCapture.capture() do
           W`Sin`
       end
(value = W"Sin", output = "", error = false, backtrace = Ptr{Nothing}[])

julia>

This may be because MathLink needs to establish a connection with Mathematica during initialisation?

This issue may be tricky to reproduce without a Mathematica or Wolfram Engine installation.

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

I would like to have a tee option so that

c = IOCapture.capture(tee=true) do
     println("test")
     return 42
 end;

captures the output and value of the code block, but also still prints to the normal stdout and stderr (or combines stderr into stdout, I don't care about keeping them separate). What I do care about is that it should stream stdout as normal, while the function is running, as opposed to just dumping the captured output after the block finished running.

The option could be named something other than tee, of course, if that's too unix-jargony.

This package feels pretty "stable" to me, so it should probably be tagged as 1.0 to indicate that.

In my opinion, v1.0 releases shouldn't be breaking: A v1.0 release should be mostly identical to the last v0.x.y release and just serve as an indication that "the API is now stable".

So, if it were up to me, I'd merge/release the recent #20 and #23 as v0.2.4 and then release v1.0 some weeks/months after that release has proven not to break anything in the wider ecosystem.

Of course, there are different opinions on the correct interpretation of SemVer, so make your own decisions ;-)

I just wanted to put the idea out there that this package seems to be mature enough to have a v1.0 label.

Shorter, consistent with the rethrow keyword and feels more appropriate if we allow passing exception types directly to it (#2).

I think it would be better to call iocapture just capture and stop exporting it. That is, the recommended pattern for calling it calling it would be

import IOCapture
c = IOCapture.capture() do
    ...
end

I would also add to the docs that, if you do not want to qualify the module, you can import with an as:

using IOCapture: capture as iocapture
c = iocapture() do
    ...
end

is this correct? should it not be ;rethrow = Union{} as in the tests? setting rethrow=false leads to a TypeError.