rprichard / win32-console-docs Goto Github PK

The current docs leave this open:

I have not tested whether or how these flags affect non-console programs (i.e. programs whose PE header subsystem is WINDOWS rather than CONSOLE).

I did some exploration of this issue while digging into a winpty bug involving ConEmu.

I did very minimal testing, and only on WIndows 7 SP1 64-bit. The behavior I saw when spawning a process with a GUI executable was:

• Specifying both CREATE_NEW_CONSOLE and DETACHED_PROCESS is still an error.
• The flags are otherwise ignored, and CreationConsoleMode is Detach.
• I can still use STARTF_USESTDHANDLES to set an stdout handle.

ObjectSnap::ObjectSnap() {
    if (!hasBuiltinCompareObjectHandles() && !needsWow64HandleLookup()) {
        m_table = queryNtHandles();
        m_hasTable = true;
    }
}

uint64_t ObjectSnap::object(RemoteHandle h) {
    if (needsWow64HandleLookup()) {
        return wow64LookupKernelObject(h.worker().pid(), h.value());
    }
    if (!m_hasTable) {
        m_table = queryNtHandles();
    }
    return reinterpret_cast<uint64_t>(ntHandlePointer(
        m_table, h.worker().pid(), h.value()));
}

This code looks suspicious to me. I think ObjectSnap::object could in principle initialize m_table multiple times. I think it should also set m_hasTable to true, and then the constructor can be removed.

ObjectSnap::object is public, so it could be invoked from a test case without going through eq. I thought that actually mattered, but now I don't see any call sites, so maybe it doesn't.

The test suite passes on 14393, with the legacy and non-legacy consoles. With 15063, it passes with the legacy console, but fails with the non-legacy console.

Here's my test run (with successes elided):

rprichard@duck15063 ~/proj/win32-console-docs/src
$ ./build/HandleTests.exe
Using CREATE_NEW_CONSOLE as default creation mode
...
Test_Console_Without_Processes
HandleTests/Modern.cc:127: ERROR: check failed (windowText(hwnd) != title):  != Test_Console_Without_Processes-6276-131397375107502604-57
Test_Implicit_Buffer_Reference
HandleTests/Modern.cc:173: ERROR: check failed (activeFirstChar(p) != 'A'): 66 != A
...
Test_Active_ScreenBuffer_Order
HandleTests/MiscTests.cc:171: ERROR: check failed (firstChar(p) != 'a'): 98 != a
HandleTests/MiscTests.cc:183: ERROR: check failed (firstChar(p) != 'a'): 99 != a
...
Using CREATE_NO_WINDOW as default creation mode
...
Test_Console_Without_Processes
HandleTests/Modern.cc:127: ERROR: check failed (windowText(hwnd) != title):  != Test_Console_Without_Processes-6276-131397375107502604-304
Test_Implicit_Buffer_Reference
HandleTests/Modern.cc:173: ERROR: check failed (activeFirstChar(p) != 'A'): 66 != A
...
Test_Active_ScreenBuffer_Order
HandleTests/MiscTests.cc:171: ERROR: check failed (firstChar(p) != 'a'): 98 != a
HandleTests/MiscTests.cc:183: ERROR: check failed (firstChar(p) != 'a'): 99 != a
...

Failed tests:
  Test_Active_ScreenBuffer_Order
  Test_Console_Without_Processes
  Test_Implicit_Buffer_Reference

I'm thinking of writing utility functions that are intended to be like the POSIX popen(3), but with an argv, an environ (maybe), and I/O redirections. Your documentation sheds all the light I needed as to I/O redirections, but I don't really know how to properly encode an argv into an lpCommandLine, or if it is even possible (well, _spawn() must be doing it, no?).

In Win8.1 and Win10 (and presumably Win8 as well), it's impossible to free the initial screen buffer. It is possible to enter a state where the initial buffer can't be accessed via any means, but I can still see it taking up memory via the Task Manager.

Test 1:

// Tie the child to a new buffer, then kill the parent.  There is no way to
// reference the initial screen buffer, but it's still taking up memory.
int main() {
    Worker p;
    p.getStdout().setFirstChar('A');
    p.getStdout().trySetScreenBufferSize({ 30000, 1000 });
    p.newBuffer(false, 'B').activate();
    auto c = p.child();
    p.exit();
    Sleep(60000);
}

Test 2:

// This test is identical to Test 1, except that the parent reactivates the
// initial screen buffer just before it exits.  I would have expected buffer A
// to be freed after the parent died, activating the remaining buffer B,
// because SetConsoleActiveScreenBuffer does not normally prevent a buffer from
// being freed.  The initial buffer is never freed, though.
int main() {
    Worker p;
    p.getStdout().setFirstChar('A');
    p.getStdout().trySetScreenBufferSize({ 30000, 1000 });
    p.newBuffer(false, 'B').activate();
    auto c = p.child();
    p.getStdout().activate();
    p.exit();
    Sleep(60000);
}

This behavior does not apply to Windows 7. e.g.:

int main() {
    Worker w;
    w.getStdout().setFirstChar('A');
    w.getStdout().trySetScreenBufferSize({ 30000, 1000 });
    auto b = w.newBuffer(false, 'B');
    w.getStdout().close();                 // -- L1 --
    w.getStderr().close();                 // -- L2 --
    CHECK_EQ(w.openConout().firstChar(), 'B');
    Sleep(30000);
}

If I comment out lines L1 and L2 above, then the CHECK fails, and the RAM usage of conhost.exe increases to ~60MB on Win7 32-bit. Otherwise, it's only about 1.5MB.

(This is a discussion and not a real issue.)

I have experienced a similar situation to what you describe in your tests. CreateProcessW fails with ERROR_NO_SYSTEM_RESOURCES when used with EXTENDED_STARTUPINFO_PRESENT and an attempt is made to restrict the inherited handles. The problem appears only on Win7 OS as in your tests.

I am wondering if you have any further insights or if you ever found a good workaround for the problem. The obvious solution would be to call CreateProcessW and pass bInheritHandles==TRUE, but I was hoping to avoid this.

Many thanks for your attention.