matepek / vscode-catch2-test-adapter Goto Github PK

When I modify the default cwd settings and reload the tests, the cwd stays the same. According to the documentation the default cwd value is overwritten by an executable, but there is none set, the list is automatically generated. The project is using google test.

I don't tried to create a specific example to reproduce it, but I assume it is the same for any test. I guess it's because the automatic test discovery.

Describe the bug
A clear and concise description of what the bug is.
When clicking on the "play" button to run the test, it fail silently with the following error in the chrome console:

  ERR spawn c:\Users\Benoit\github\Catch2Test\build\tests\catch2_test.exe ENOENT: Error: spawn c:\Users\Benoit\github\Catch2Test\build\tests\catch2_test.exe ENOENT
	at _errnoException (util.js:1024:11)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:190:19)
	at onErrorNT (internal/child_process.js:372:16)
	at _combinedTickCallback (internal/process/next_tick.js:138:11)
	at process._tickCallback (internal/process/next_tick.js:180:9)

To Reproduce
Steps to reproduce the behavior:

I created a small test project here: https://github.com/benoxoft/Catch2Test

1. Clone the project
2. Compile it with cmake (I use CMakeTools in VSCode with the VS2017 compiler and a recent version of cmake. I also installed catch2 with cmake)
3. Open the Developer Tools to see the console
4. Click Play to run the test executable in the Test Explorer UI
5. Notice the error I posted above in the console
6. Copy and paste the full path of the executable in a cmd and notice that the executable exists.

Expected behavior
Executable should be found and tests are run

Screenshots
Here is a screenshot showing VS Code. I compiled the executable with cmake. We can see the error in the chrome console. If I copy and paste the executable in the terminal below, we can see that the executable exists.

Desktop (please complete the following information):

• Extension Version 2.0.2
• VS Code Version 1.29.1
• Catch2 Version latest from master
• OS Type and Version Windows 10

Additional context

Describe the bug
By default, Catch2 will split some lines after 80 characters (CATCH_CONFIG_CONSOLE_WIDTH). The problem is that this extension expect the Test Name, Description or Tags to be on one line only. Thus a test having those properties will cause an error like:
ERR Illegal argument: line must be non-negative : Error: Illegal argument: line must be non-negative

To Reproduce

1. Create a TEST_CASE with a list of tags longer than 80 characters.
2. Compile the executable.
3. Add the executable in the list of tests supported by the plugin.
4. The executable won't appear in the list (in fact, the list won't refresh).
5. Display developper console to see error.

Expected behavior
The executable and the test should appear in the list.

It should work in all the following case:

1. Test name longer than 80
2. File name longer than 80
3. Description longer than 80
4. Total length of tags longer than 80
5. A tag longer than 80

Desktop (please complete the following information):

• Extension Version: 2.0.1
• VS Code Version: 1.29.1
• Catch v2.0.1

Additional context
I've encountered the problem with a list of tag longer than 80 and I've found a solution for those in C2TestSuiteInfo.ts (by changing the "if" for the tags to a while and appending to the tags array).

But, I don't see how it would be possible to handle case 1 and 4 specifically. Those give the following output:
__A very long test name that should be longer than 80 characters to cause a
____line split
____PATH_TO_FILE.cpp:LINE
____(NO DESCRIPTION)
__Long tags
____PATH_TO_FILE.cpp:LINE
____(NO DESCRIPTION)
______[A_very_long_test_name_that_should_be_longer_than_80_characters_to_cause-
------_a_line_split]

I think it would be possible to support the long tags (it seems the tags have 6 spaces) but I don't see any way to differentiate between the title continuing and the path to the file.

Suggestion
Request a parameter to Catch2 allowing to disable the linebreaks. This would allow to keep the current parsing by simply giving that option.

Is your feature request related to a problem? Please describe.
If you load the test-explorer view, tests which are failing during test-case scan (e.g. Segfault), are not shown in the list. This behavior prevent the ability to debug this test.

Describe the solution you'd like
List also the failing test, but with the red cross...

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

I've noticed that when i run the test from the test adapter, the output wont be colored as opposed to running the executable itself. Would like a colourized output.

Is your feature request related to a problem? Please describe.
doctest tests are not recognized by the extension although doctest is based on catch and is very similar. Its main advantage is that it compiles tests much faster that catch2.

Describe the solution you'd like
Is it possible to make this extension recognize doctest tests just as well as catch2 tests? Maybe it is just a matter of configuring the extension or doctest.

Describe alternatives you've considered
If it is not just a configuration issue, then either a new test adapter extension would be needed or this one would need to be updated to work with doctest. Since doctest is based on catch, this extension would probably be a good starting point in both cases.

Additional context
Link to doctest:
https://github.com/onqtam/doctest

Is your feature request related to a problem? Please describe.
It is hard to get which part of the code are checked against which tests (and where do they fail). I believe Test Explorer has code lens and gutters for coverage for each test, but Catch2/GTest Adapter cannot generate data for it, because it does not give information about lines that run for given tests.

Describe the solution you'd like
Actually there is a FOSS tool (unfortunately Windows only) that can feed that information; OpenCppCoverage takes PDB and verify, for each executable run, which lines were actually executed and which were not. It generates cov.xml (format recognizable by Coverage Lens in VSCode).

It would be great if Catch2/GTest Adapter could use that data to populate lens and gutters at least on Windows.

Describe alternatives you've considered
There is no similiar tool for Linux / OS X I am aware of. There is no other plugin for GTest for VSCode as well.

Additional context
If effort would not be acceptable on Your side for any reason, I can implement this in fork, then propose PR. I would ask only for some basic directions. Still, it would be great, if You can add this feature directly - you know your code much better than myself.

Is your feature request related to a problem? Please describe.
It is hard to debug single catch2 SECTION

Describe the solution you'd like
Show list of TEST_CASE SECTIONs in Test Explorer panel. Allow to run and debug single section.

Describe alternatives you've considered
If it is hard to add next level to TestExplorer tree it would be great if TEST_CASE and its SECTIONs could be displayed next to each other on single list.

I have test executables inside build/. I added this folder to files.exclude in order to hide it from explorer tree, but my test executables are also become hidden. Executables are not shown even if I pass relative or absolute path instead of globbing pattern.

Is there a possibility to resolve executables pattern even if it intersects with 'files.exclude'?

I suggest an example of how to install and setup a project is provided in the readme.

When setting defaultEnv property with os_env that doesn't exist, the environment variable will still be created, with not resolved string value.

For example

"catch2TestExplorer.defaultEnv": {
	"VAR_DOESNT_EXIST": "${os_env:VAR_DOESNT_EXIST}"
}

The literal resulting value passed to the executable would be VAR_DOESNT_EXIST ==
"${os_env:VAR_DOESNT_EXIST}".

Gtest tests listing is not parsed correctly when gtest typed test are in use.

Output goes like this:
InstallationMasterServiceTest/0. # TypeParam =
RevokeInvokesRevokeEvent
InstallationMasterServiceTest/1. # TypeParam =
RevokeInvokesRevokeEvent

Regex in GoogleTestSuiteInfo.js about line 115 is not prepared to handle the #...

This issue causes any test after the typed ones to be ignored by extension.

If the google test or any function inside google test contains printf or std::cout functions,

printf("Is True");

it ends as unexpected error and all next test will stop running.
But if the printf command will have new line symbol at the end of text

printf("Is True \n");

next tests are continue running as expected
this fail

TEST_F(Temp, setTemp)
{
        printf("Is True");
}

this is OK

TEST_F(Temp, setTemp)
{
        printf("Is True \n");
}

Describe the solution you'd like
I would like to have an option where I can set a command which is run before any test is run (before scanning, running, debugging etc.) Background is, that we set up our build environment with conan and I cannot run the tests if the build-environment is not set up properly. Therefore I currently have to start vscode via console and run the command before i run vscode.

Before you file an issue, please check the README.md's known issues section.

Describe the bug (A clear and concise description of what the bug is.)
When running test (sometimes even after scanning) the tests are marked as failing with the following error profiling: /external/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.gcda: Merge mismatch for function 1

To Reproduce (Steps to reproduce the behavior:)

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior (A clear and concise description of what you expected to happen.)

Screenshots (If applicable, add screenshots to help explain your problem.)

Desktop (please complete the following information):

• Extension Version [e.g. 22] 2.6.4
• VS Code Version [e.g. 22] 1.36.1
• Catch2/Google Test Version 1.8
• OS Type and Version Win 10

Log (Attach log. (optional, Set: catch2TestExplorer.logfile and reproduce the bug.) Warning: Log probably contains file and test names too.)

Additional context Add any other context about the problem here.

• The issue is about this extension and NOT about a fork.
• Check the known issues list.
• The latest version of the extension was used.

In the test explorer click on a test that resides in a file with a name such as TestClass.cpp. Then the editor opened with name testcase.cpp. This cause some problems for example if the file was opened, there we have two tabs for one file, and intellisense will be disabled in second file, and...

• Extension Version [e.g. 2.7.6]
• VS Code Version [e.g. 1.40.1]
• Google Test Version [e.g. 1.8]
• OS Type and Version: Windows 10 x64 1909

It would be nice to see the execution time of each test next to the test name inside the test explorer. I'm currently using google test, maybe it is there for other test environments?

Describe the bug (A clear and concise description of what the bug is.)

Clicking on "Debug" starts the python debugger rather than calling only the cpp executable, resulting in ValueError: source code string cannot contain null bytes errors. While the "Run" click works fine with cpp executable.

To Reproduce (Steps to reproduce the behavior.)

1. Go to a cpp file contains Google test
2. Click on the "Debug" above the TEST
3. Scroll down to Output
4. See error

/home/sgao/anaconda3/bin/python /home/sgao/.vscode/extension
s/ms-python.python-2019.9.34474/pythonFiles/ptvsd_launcher.py --default --client --host localhost --port 45231 /home/sgao/Documents/sim/build-Debug/sim-slicer/test/sim_slicer_test --gtest_
color=no --gtest_filter=slicer.loading --gtest_break_on_failure 
Traceback (most recent call last):
  File "/home/sgao/.vscode/extensions/ms-python.python-2019.9.34474/pythonFiles/ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "/home/sgao/.vscode/extensions/ms-python.python-2019.9.34474/pythonFiles/lib/python/ptvsd/__main__.py", line 432, in main
    run()
  File "/home/sgao/.vscode/extensions/ms-python.python-2019.9.34474/pythonFiles/lib/python/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/home/sgao/anaconda3/lib/python3.7/runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "/home/sgao/anaconda3/lib/python3.7/runpy.py", line 236, in _get_code_from_file
    code = compile(f.read(), fname, 'exec')
ValueError: source code string cannot contain null byte

Screenshots (If applicable, add screenshots to help explain your problem.)

Desktop

• Extension Version 2.7.1
• VS Code Version 1.38.0
• Catch2/Google Test Version 1.8.1
• OS Type and Version ubuntu 16.04

Log (Attach log. (optional, Set: catch2TestExplorer.logfile and reproduce the bug.) Warning: Log probably contains file and test names too.)

[2019-09-05 06:28:36.140] [INFO] [at TestAdapter.debug (/home/sgao/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.7.1/out/src/TestAdapter.js:222:19)] Debugging
[2019-09-05 06:28:36.141] [INFO] [at TestAdapter.debug (/home/sgao/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.7.1/out/src/TestAdapter.js:246:19)] testInfo:  GoogleTestInfo {
  _shared:
   SharedVariables {
     log:
      Log {
        configSection: 'catch2TestExplorer',
        workspaceFolder: { uri: [g], name: 'sim', index: 0 },
        outputChannelName: 'Test Explorer: sim',
        inspectOptions: { showProxy: true, depth: 3 },
        includeLocation: true,
        targets: [ [FileTarget] ],
        nextInspectOptions: undefined,
        configChangeSubscription: { dispose: [Function: dispose] } },
     workspaceFolder:
      { uri:
         g {
           scheme: 'file',
           authority: '',
           path: '/home/sgao/Documents/sim',
           query: '',
           fragment: '',
           _formatted: 'file:///home/sgao/Documents/sim',
           _fsPath: '/home/sgao/Documents/sim' },
        name: 'sim',
        index: 0 },
     testStatesEmitter:
      u {
        _disposed: false,
        _options: undefined,
        _leakageMon: undefined,
        _event: [Function],
        _listeners: r { _first: [n], _last: [n], _size: 1 },
        _deliveryQueue: r { _first: [n], _last: [n], _size: 0 } },
     loadWithTaskEmitter:
      u {
        _disposed: false,
        _options: undefined,
        _leakageMon: undefined,
        _event: [Function],
        _listeners: r { _first: [n], _last: [n], _size: 1 } },
     sendTestEventEmitter:
      u {
        _disposed: false,
        _options: undefined,
        _leakageMon: undefined,
        _event: [Function],
        _listeners: r { _first: [n], _last: [n], _size: 1 } },
     retire:
      u {
        _disposed: false,
        _options: undefined,
        _leakageMon: undefined,
        _event: [Function],
        _listeners: r { _first: [n], _last: [n], _size: 1 } },
     rngSeed: null,
     execWatchTimeout: 10000,
     retireDebounceTime: 1000,
     _execRunningTimeout: 600000,
     execParsingTimeout: 5000,
     isNoThrow: false,
     enabledTestListCaching: false,
     googleTestTreatGMockWarningAs: 'nothing',
     googleTestGMockVerbose: 'default',
     _execRunningTimeoutChangeEmitter:
      u {
        _disposed: false,
        _options: undefined,
        _leakageMon: undefined,
        _event: [Function],
        _listeners: r { _first: [n], _last: [n], _size: 0 } },
     onDidChangeExecRunningTimeout: [Function],
     taskPool:
      TaskPool { _maxTaskCount: 1, _runningTaskCount: 0, _waitingTasks: [] } },
  testNameAsId: 'slicer.loading',
  label: 'loading',
  skipped: false,
  line: 5,
  type: 'test',
  lastRunState: 'passed',
  lastRunMilisec: 4270,
  id: '35',
  origLabel: 'loading',
  description: '',
  file:
   '/home/sgao/Documents/sim/sim-slicer/test/sim_slicer_test.cpp',
  tooltip: 'Name: slicer.loading' } [ '35' ];
[2019-09-05 06:28:36.144] [INFO] [at TestAdapter._getDebugConfigurationTemplate (/home/sgao/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.7.1/out/src/TestAdapter.js:346:35)] using debug config from launch.json
[2019-09-05 06:28:36.145] [INFO] [at TestAdapter.debug (/home/sgao/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.7.1/out/src/TestAdapter.js:300:19)] Debug: resolved debugConfig: { name: 'loading (sim_slicer_test ➡️ slicer)',
  type: 'python',
  request: 'launch',
  program:
   '/home/sgao/Documents/sim/build-Debug/sim-slicer/test/sim_slicer_test',
  args:
   [ '--gtest_color=no',
     '--gtest_filter=slicer.loading',
     '--gtest_break_on_failure' ],
  console: 'integratedTerminal',
  cwd: '/home/sgao/Documents/sim/sim-slicer',
  target:
   '/home/sgao/Documents/sim/build-Debug/sim-slicer/test/sim_slicer_test',
  arguments:
   '"--gtest_color=no" "--gtest_filter=slicer.loading" "--gtest_break_on_failure"',
  env:
   { KDE_FULL_SESSION: 'true',
     GS_LIB: '/home/sgao/.fonts',
     PAM_KWALLET5_LOGIN: '/tmp/kwallet5_sgao.socket',
     USER: 'sgao',
     LC_TIME: 'zh_CN.UTF-8',
     XDG_SEAT: 'seat0',
     XDG_SESSION_TYPE: 'x11',
     SSH_AGENT_PID: '2918',
     SHLVL: '1',
     HOME: '/home/sgao',
     QT4_IM_MODULE: 'fcitx',
     DESKTOP_SESSION: '/usr/share/xsessions/plasma',
     XIM: 'fcitx',
     QT_LINUX_ACCESSIBILITY_ALWAYS_ON: '1',
     GTK_MODULES: 'gail:atk-bridge',
     XDG_SEAT_PATH: '/org/freedesktop/DisplayManager/Seat0',
     KDE_SESSION_VERSION: '5',
     LC_MONETARY: 'zh_CN.UTF-8',
     DBUS_SESSION_BUS_ADDRESS:
      'unix:abstract=/tmp/dbus-77J7wCKzZp,guid=4f5920e469eed4f3cc7f0fb55d707c06',
     SSH_ASKPASS: '/usr/bin/ksshaskpass',
     QT_QPA_PLATFORMTHEME: 'appmenu-qt5',
     MANDATORY_PATH:
      '/usr/share/gconf//usr/share/xsessions/plasma.mandatory.path',
     XIM_PROGRAM: 'fcitx',
     LOGNAME: 'sgao',
     GTK_IM_MODULE: 'fcitx',
     XDG_SESSION_CLASS: 'user',
     DEFAULTS_PATH: '/usr/share/gconf//usr/share/xsessions/plasma.default.path',
     XDG_SESSION_ID: '2',
     GTK2_MODULES: 'overlay-scrollbar',
     PATH:
      '/home/sgao/bin:/home/sgao/.local/bin:/home/sgao/node/bin:/home/sgao/Documents/Go//bin/:/home/sgao/Documents/Apps/julia/bin/:/home/sgao/bin:/home/sgao/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin',
     LC_ADDRESS: 'zh_CN.UTF-8',
     XDG_SESSION_PATH: '/org/freedesktop/DisplayManager/Session1',
     XDG_RUNTIME_DIR: '/run/user/1001',
     XCURSOR_THEME: 'breeze_cursors',
     DISPLAY: ':0',
     LC_TELEPHONE: 'zh_CN.UTF-8',
     LANG: 'en_US.UTF-8',
     XDG_CURRENT_DESKTOP: 'KDE',
     XMODIFIERS: '@im=fcitx',
     XDG_SESSION_DESKTOP: 'KDE',
     XAUTHORITY: '/tmp/xauth-1001-_0',
     SSH_AUTH_SOCK: '/tmp/ssh-TRjgmwDZlh6X/agent.2877',
     SHELL: '/bin/bash',
     PAM_KWALLET_LOGIN: '/tmp/kwallet_sgao.socket',
     LC_NAME: 'zh_CN.UTF-8',
     QT_ACCESSIBILITY: '1',
     LC_MEASUREMENT: 'zh_CN.UTF-8',
     LC_IDENTIFICATION: 'zh_CN.UTF-8',
     XDG_VTNR: '7',
     QT_IM_MODULE: 'fcitx',
     PWD: '/home/sgao',
     XDG_CONFIG_DIRS:
      '/etc/xdg/xdg-/usr/share/xsessions/plasma:/etc/xdg:/usr/share/kubuntu-default-settings/kf5-settings',
     XDG_DATA_DIRS:
      '/usr/share//usr/share/xsessions/plasma:/usr/local/share:/usr/share:/var/lib/snapd/desktop',
     LC_NUMERIC: 'zh_CN.UTF-8',
     LC_PAPER: 'zh_CN.UTF-8',
     KDE_SESSION_UID: '1001',
     KDE_MULTIHEAD: 'false',
     GTK_RC_FILES: '/etc/gtk/gtkrc:/home/sgao/.gtkrc:/home/sgao/.config/gtkrc',
     GTK2_RC_FILES:
      '/etc/gtk-2.0/gtkrc:/home/sgao/.gtkrc-2.0:/home/sgao/.config/gtkrc-2.0',
     SESSION_MANAGER:
      'local/hy-System-Product-Name:@/tmp/.ICE-unix/3114,unix/hy-System-Product-Name:/tmp/.ICE-unix/3114',
     DESKTOP_STARTUP_ID: 'hy-System-Product-Name;1567652884;801628;3191_TIME39603',
     VSCODE_IPC_HOOK: '/run/user/1001/vscode-2c483866-1.38.0-main.sock',
     VSCODE_NLS_CONFIG:
      '{"locale":"en-us","availableLanguages":{},"_languagePackSupport":true}',
     VSCODE_LOGS: '/home/sgao/.config/Code/logs/20190905T110805',
     CHROME_DESKTOP: 'code-url-handler.desktop',
     GOPATH: '/home/sgao/Documents/Go/',
     GOROOT: '/home/sgao/Documents/Apps/go',
     LESSCLOSE: '/usr/bin/lesspipe %s %s',
     LESSOPEN: '| /usr/bin/lesspipe %s',
     LS_COLORS: '',
     NODE_PATH: '/home/sgao/node/lib/node_modules:',
     NO_AT_BRIDGE: '1',
     NPM_PACKAGES: '/home/sgao/node',
     VSCODE_NODE_CACHED_DATA_DIR:
      '/home/sgao/.config/Code/CachedData/3db7e09f3b61f915d03bbfa58e258d6eee843f35',
     VSCODE_PID: '4091',
     _: '/usr/share/code/code',
     AMD_ENTRYPOINT: 'vs/workbench/services/extensions/node/extensionHostProcess',
     PIPE_LOGGING: 'true',
     VERBOSE_LOGGING: 'true',
     VSCODE_IPC_HOOK_EXTHOST: '/tmp/vscode-ipc-91450c99-fe34-4ecf-8528-41c13c61f001.sock',
     VSCODE_HANDLES_UNCAUGHT_ERRORS: 'true',
     VSCODE_LOG_STACK: 'false',
     VSCODE_PREVENT_FOREIGN_INSPECT: 'true',
     ELECTRON_RUN_AS_NODE: '1',
     APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL: 'true' },
  'magic variable  🤦🏼‍': '39' }
[2019-09-05 06:28:37.161] [INFO] [at debug._mainTaskQueue.then (/home/sgao/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.7.1/out/src/TestAdapter.js:321:23)] debugSessionStarted
[2019-09-05 06:28:37.964] [INFO] [at debug._mainTaskQueue.then.Promise.finally (/home/sgao/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.7.1/out/src/TestAdapter.js:313:27)] debugSessionTerminated

Describe the bug (A clear and concise description of what the bug is.)
In the test panel no executable and no test is shown. When the "Reload tests" button is clicked, the arrow spins a few times but no executable and test is loaded. In the logfile there is the following (PATHS AND EXECUTABLES ARE HIDED):

[2019-03-01 07:23:41.807] [WARN] Not a test executable: ABSOLUTE_PATH_AND_NAME_OF_TESTEXECUTABLE_HERE reason: [Error: FsWrapper.spawnAsync timeout: 5000
at Timeout.setTimeout [as _onTimeout] (ABSOLUTE_PATH_TO_EXTENSION_FOLDER_HERE.vscode\extensions\matepek.vscode-catch2-test-adapter-2.3.21\out\FsWrapper.js:47:28)
at ontimeout (timers.js:427:11)
at tryOnTimeout (timers.js:289:5)
at listOnTimeout (timers.js:252:5)
at Timer.processTimers (timers.js:212:10)]

A timeout is happening when checking the executable for tests.

With Version 2.3.15 of the Catch2 extension it is still working. When the "Reload tests" button is clicked the arrow spins for about 30 secs and then the tests are shown.

Expected behavior (A clear and concise description of what you expected to happen.)
The tests are loaded and shown in the test panel.

Desktop (please complete the following information):

• Extension Version 2.3.21 (With version 2.3.15 it was working. Since version 2.3.16 it fails)
• VS Code Version 1.31.1
• Catch2 v2.6.1
• Windows 7

The test adapter never stops loading (refresh icon continue spinning) and no tests appear when it discovers gtest tests with default parameters. I tried to wait about 20 minutes, but still no success. If I set catch2TestExplorer.executables to discover tests within a specific executable, e.g.

    "catch2TestExplorer.executables": [
        {
            "name": "${filename} (${relDirpath})",
            "pattern": "build/training_test",
            "cwd": "${workspaceFolder}/test"
        }
    ]

tests are discovered just fine.

This is happening for the C++ part of my repo https://github.com/kondratyev-nv/training/tree/master/cpp.

Also, I'm using both Windows and Linux, and this affects only Linux (Ubuntu 18.04). With a little bit of debugging, it looks like discovery hangs on executing some python script from gtest, but I'm not sure about it. Maybe it works on Windows because python scripts are not executable by itself in there.

Log: cpp-test-adapter-linux.log

• Extension Version is 2.3.15
• VS Code Version is 1.30.2

• The issue is about this extension and NOT about a fork.
• Check the known issues list.
• The latest version of the extension was used.

Description:
Catch2 tests not populating the Test side panel.

To Reproduce (Steps to reproduce the behavior.)

1. Go to 'settings.json'
2. use default pattern:

"catch2TestExplorer.executables":[
        {
            "name": "${filename}",
            "description": "${relDirpath}/",
            "pattern": "{build,Build,BUILD,out,Out,OUT}/**/*{test,Test,TEST}*"
        }
    ],

3. Click the Test Explorer: Reload Tests
4. See error

Screenshots (If applicable, add screenshots to help explain your problem.)

Desktop

• Extension Version 2.7
• VS Code Version 1.37.1
• Catch2/Google Test Version 2.7
• OS Linux Fedora 30

Log Is a separate issue as I can't get it to write. Error is:

Couldn't write log file ~/logs/Catch.log: Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed

I will write another bug if I cant find a solution soon

Is your feature request related to a problem? Please describe.
You can't jump from the test case in the test explorer to the source file where the test is defined.

Describe the solution you'd like
For test discovery, run the executable with --verbosity high --list-test-names-only.
Catch will then produce output like:

map     @parsecxx/test/combinators.cpp(52)
times   @parsecxx/test/combinators.cpp(57)
operator<<      @parsecxx/test/combinators.cpp(96)

which can be split into test case name, file and line number.

Additional context
I tried to implement this myself, but I couldn't even get the extension to compile without any changes because of this error:

src/adapter.ts:302:70 - error TS2345: Argument of type '[string[], string[], [string, string[]], [string, string], [string, ProcessEnv]]' is not assignable to parameter of type '[string, string] | ConcatArray<[string, string]>'.
  Type '[string[], string[], [string, string[]], [string, string], [string, ProcessEnv]]' is not assignable to type 'ConcatArray<[string, string]>'.
    Types of property 'slice' are incompatible.
      Type '(start?: number, end?: number) => (string[] | [string, string] | [string, string[]] | [string, Pr...' is not assignable to type '(start?: number, end?: number) => [string, string][]'.
        Type '(string[] | [string, string] | [string, string[]] | [string, ProcessEnv])[]' is not assignable to type '[string, string][]'.
          Type 'string[] | [string, string] | [string, string[]] | [string, ProcessEnv]' is not assignable to type '[string, string]'.
            Type 'string[]' is not assignable to type '[string, string]'.
              Property '0' is missing in type 'string[]'.

302       const resolveDebugVariables = this.variableResolvedPair.concat([
                                                                         ~
303         ["${label}", testInfo.label],
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
307         ["${envObj}", testInfo.execOptions.env!]
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
308       ]);
    ~~~~~~~

Is your feature request related to a problem? Please describe.
While trying populate the list of tests using the refresh button, my project continually fails to populate. Upon enabling logging of the extension, I found that the test file was found, but was timing out after 5 seconds when trying to load tests.

Describe the solution you'd like
It would be absolutely awesome if the 2 x places where there a timeout specified on the async call with a timeout specified of "5000", in Catch2TestSuiteInfo.js and RunningTestExecutableInfo.js was a configurable value. Especially useful for people with slower machines or large projects.

Describe alternatives you've considered
Alternatively, increasing those timeouts to 15000 manually in the code worked for me, so I (but perhaps not others) would be happy for this default to just be increased in the code to a value that "works for me". :)

Additional context
I'm running on Linux, using the code-insiders build.
My same project running on Windows did not hit the timeout and worked fine.

When I refresh the tests toolbar, it doesn't show all tests I have in the specified folder.

My settings are:

"catch2TestExplorer.executables": [
    {
        "name": "${filename}",
        "pattern": "/absoluteTestFolderPath/*{test,Test,TEST}*"
    }
]

The logpanel output for the not found tests is the following:

[2019-06-26 18:47:42.655] [WARN] [at load.suiteCreationAndLoadingTasks.push.c2fs.isNativeExecutableAsync.then._createSuiteByUri.then (/home/user/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.5.0/out/src/TestExecutableInfo.js:73:38)] Not a test executable: /absoluteTestFolderPath/HistogramTest reason: [Error: FsWrapper.spawnAsync signal: SIGABRT
	at ChildProcess.<anonymous> (/home/user/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.5.0/out/src/FSWrapper.js:36:24)
	at ChildProcess.emit (events.js:182:13)
	at maybeClose (internal/child_process.js:961:16)
	at Socket.ChildProcess.spawn.stream.socket.on (internal/child_process.js:380:11)
	at Socket.emit (events.js:182:13)
	at Pipe.Socket._destroy._handle.close [as _onclose] (net.js:596:12)]

When skipping a test with GTEST_SKIP() the tree shows the nest test as skipped.
see the screenshot:

the skipped test is test_1 but the tree show test_2 as skipped.

Is your feature request related to a problem? Please describe.
Currently when I change code and want to test and click run. It runs the old binary.

Describe the solution you'd like
Call cmake build first before run.

Is your feature request related to a problem? Please describe.
I try to find my tests related to the environment variable SOFTTEMP.

{
    "catch2TestExplorer.executables": [
        {
            "name": "${filename}",
            "description": "${absDirpath}/",
            "pattern": "${SOFTTEMP}/Debug/SomeProject/*{test,Test,TEST}*"
        }
    ]
}

Unfortunately these settings do not detect any tests.

Describe the solution you'd like
The pattern should be able to detect all tests in this

${SOFTTEMP}/Debug/SomeProject
folder.

Describe alternatives you've considered
If I use hardcoded strings it works but I want to use the settings on different machines with different options.
I also tried to set the cwd to the folder:

"cwd": "${os_env:SOFTTEMP}/Debug/SomeProject"
"pattern": "*{test,Test,TEST}*",

Describe the bug
The extension fails to find several of my test suites. For the missing test suites, the log includes error messages like the following:

[2018-12-09 23:24:59.165] [WARN] Couldn't load suite: [ TypeError: Cannot read property 'length' of undefined
[...snip...]
[2018-12-09 23:24:59.165] [ERROR] but it was a Catch2 executable

To Reproduce
Steps to reproduce the behavior:

1. My project is available on GitHub. Check out this repo: https://github.com/foxostro/FlapjackOS
2. Run ./scripts/test.sh to configure and build unit tests.
3. Open VSCode.
4. Click the "test" button in the sidebar.
5. Click the "reload tests" button in the test explorer.

Expected behavior
Expected: All executables matching the pattern are listed in the test explorer.
Actual: Only some executables are listed in the test explorer. The results are consistent.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• Extension Version: 2.2.5
• VS Code Version: 1.29.1
• Catch2 Version: Catch v2.0.1
• OS Type and Version: macOS 10.13.6

Log
Attach log. (optional, Set: catch2TestExplorer.logfile and reproduce the bug.)
Warning: Log probably contains file and test names too. Consider security conserns before attach!
catch2TestExplorer.logfile.txt

Additional context
I hope that it is possible to diagnose this problem without actually going through the process of building my project. Please let me know if you need any additional information.

• The issue is about this extension and NOT about a fork.
• Check the known issues list.
• The latest version of the extension was used.

Describe the bug (A clear and concise description of what the bug is.)
The extension recognizes the test executable and is able to run tests from the sidebar.
There are 2 problems though:

• In the sidebar, the little icon that links to the source files is missing.
  

• There is no highlighting of passed/failed tests in the test sources.

To Reproduce (Steps to reproduce the behavior.)
I have a project with the following hierarchy:

src/
include/
test/
-- BiMap/
    -- BiMapTests.cpp
build/
-- test/
    -- gtest

I haven't changed the default configuration at all.

Desktop

• Extension Version 2.7.5
• VS Code Version 1.38.1
• Catch2/Google Test Version [Not sure]
• OS Type and Version Ubuntu 18.04

Hello,

It looks like since version 2.4.10 catch2TestExplorer.executables from folder or workspace settings are not used. I had to move my configuration to user settings to be effective.
Version 2.4.9 works fine.

There is no errors in the logs. Only default configuration is being used to search for tests executables.

Is your feature request related to a problem? Please describe.
I have multiple buildkit inside cmake to switch between compilers like GCC and Clang, there are build folders for both and Debug/Release. Currently there' s no way I can know which executable to run when I click the Run|Debug inside code.

Describe the solution you'd like
I want to set the executable of kit easier, and then click on the button inside code.

Describe alternatives you've considered
Now I have to click the Test Explorer and run Debug from there.

After running tests in (both gtest and catch2), tests do not re-run when clicking the run button, either the overall run button of the button for individual tests.

When looking at the log panel, this message is printed;

[2019-07-26 13:31:26.844] [INFO] [at TestAdapter.run (/home/danny/.vscode-server/extensions/matepek.vscode-catch2-test-adapter-2.6.4/out/src/TestAdapter.js:211:23)] Run is busy

• Extension Version 2.6.4
• VS Code Version 1.36.1
• Windows 10 and Ubuntu 18.04

Log Logfile is not generated

If the code throws the exception outside the google test for example:

int globalFunction() {
    throw(std::runtime_error("")); // throwing exception that is outside gtest
}

int globalVar = globalFunction();

TEST() {
    // Google test code here
}

Then the test adaptor extension will not list this executable.

The logpanel output for the missing test is the following:

[2019-06-26 18:47:42.655] [WARN] [at load.suiteCreationAndLoadingTasks.push.c2fs.isNativeExecutableAsync.then._createSuiteByUri.then (/home/user/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.5.0/out/src/TestExecutableInfo.js:73:38)] Not a test executable: /absoluteTestFolderPath/HistogramTest reason: [Error: FsWrapper.spawnAsync signal: SIGABRT
	at ChildProcess.<anonymous> (/home/user/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.5.0/out/src/FSWrapper.js:36:24)
	at ChildProcess.emit (events.js:182:13)
	at maybeClose (internal/child_process.js:961:16)
	at Socket.ChildProcess.spawn.stream.socket.on (internal/child_process.js:380:11)
	at Socket.emit (events.js:182:13)
	at Pipe.Socket._destroy._handle.close [as _onclose] (net.js:596:12)]

Hi,

Kenneth here from the VS Code team.

Your extension is affected by https://code.visualstudio.com/blogs/2018/11/26/event-stream, and we have blocked your extension.

In order to enable your extension again the workflow is:

1. Fix the extension, submit an update to the marketplace
2. Send mail to [email protected] and [email protected] and notify us.
3. We will verify that the extension is okay
4. Our marketplace will publish the extension again and we'll remove the extension from the blocklist

/k

In case one of the tests runs into an unexpected error, (e.g. segfault) the test explorer view resets, and it is hard to find where was the error. While the tests runs the icons are correct the reset is only occurs after the failure. When I run the test one-by-one the icons are fine, and the test with the unexpected error shows a fail icon.

To reproduce: I expected this on google test, and after reading a non-existing file.

Hello,

Executing tests wrapped by script are not working anymore since last update.
I know it is not standard way, but my tests must be executed on remote target not locally.

Log:

[2019-03-15 09:53:34.870] [INFO] [at load.suiteCreationAndLoadingTasks.push.c2fs.isNativeExecutableAsync.then (.vscode/extensions/matepek.vscode-catch2-test-adapter-2.3.24/out/TestExecutableInfo.js:81:34)] Not an executable: .vscode/target_ut.sh [Error: Not a native executable (filtered because of its extension): .vscode/target_ut.sh
	at Object.isNativeExecutableAsync (.vscode/extensions/matepek.vscode-catch2-test-adapter-2.3.24/out/FsWrapper.js:97:35)
	at TestExecutableInfo.load (.vscode/extensions/matepek.vscode-catch2-test-adapter-2.3.24/out/TestExecutableInfo.js:68:52)]

Before you file an issue, please check the README.md's known issues section.

Describe the bug (A clear and concise description of what the bug is.)
Nothing happens after running a parameterized test.

To Reproduce (Steps to reproduce the behavior:)

1. Try to run any parameterized test.
2. Nothing happens

Expected behavior (A clear and concise description of what you expected to happen.)
The test should run and display its result.

Screenshots (If applicable, add screenshots to help explain your problem.)

Desktop (please complete the following information):

• Extension Version: 2.3.7
• VS Code Version 1.30.1
• Catch2 Version: N/A
• OS Type and Version: Windows 10 Pro x64

Log (Attach log. (optional, Set: catch2TestExplorer.logfile and reproduce the bug.) Warning: Log probably contains file and test names too.)

[2019-01-03 09:38:25.442] [INFO] proc started: [ 'redacted.exe',
[ '--gtest_color=no',
'--gtest_filter=Test2Instance/Test2.DispatchResponse/0 # GetParam() = (false, false)',
'--gtest_also_run_disabled_tests' ] ]
[2019-01-03 09:38:25.624] [INFO] proc finished:

Additional context Add any other context about the problem here.

I think the issue here is the --gtest_filter that is too specific ( AFAIK you shouldn't specify the 'GetParam()' part there )

Enable autorun for all testes.
ninja -t clean; ninja

It hangs there.

Hi,
The size of the current version (2.7.3) vsix file size is 1.87MB, the size of version 2.7.1 was around 800K.
There was increase of 1MB between 2.7.1 to 2.7.3, Is it OK ?

(Experimental) In case your executable took too much time to list the tests, one can set this. It will preserve the output of --gtest_list_tests --gtest_output=xml:.... (Beware: Older Google Test doesn't support xml test list format.)

Please subscribe to this issue at Notifications 👉👉👉
if you would like to be updated about the changes.
Also I will have questions too.

👍if you like it.
👀if you tried it but didn't feel the difference.

Comment if you don't feel the difference or has some feedback. I'm interested about the situations when it has performance benefits (with details: file count; avg. test count/file ...)

When doing bdd style test with catch2, if the test name is too long, the test adapter wont list the test at all. Tried to #define CATCH_CONFIG_CONSOLE_WIDTH 300 but it didnt fix the issue. The limit seems to be 68 character, more than that, the test adapter will think it's too long.

Describe the bug
The glob syntax to find tests executables ("catch2TestExplorer.executables.pattern" setting) not work when the path point to a folder outside the workspace.

To Reproduce

1. Clone my test repository: https://github.com/DPD85/Catch2VSCodeExtensionIssue;
2. Open the repository with VSCode and build everything;
3. Open Tests Explorer tab;
4. See that there is no tests, see first screenshot.

Expected behavior
See one test from Tests.exe called "Factorials are computed", see second screenshot.

Screenshots

Desktop:

• Extension Version: 2.3.8
• VS Code Version: 1.30.1
• Catch2/Google Test Version: Catch2 v2.5.0
• OS Type and Version: Windows 10, version 1809 (build OS 17763.195), 64 Bits

Log
Log file when the error occur: catch2TestExplorer.log

Checklist (You can delete this section, don't have to be the part of your issue.)

• The issue is about this extension and NOT about a fork.
• Check the known issues list.
• The latest version of the extension was used.

Describe the bug (A clear and concise description of what the bug is.)
When writing a catch2 test case that contains a backtick (other symbol on tilde key) in its name, for example,

To Reproduce (Steps to reproduce the behavior.)

1. Make a c++ project that uses catch2, and create a new catch2 test file.
2. Put the following code in the test file:

#define CATCH_CONFIG_MAIN
#include <catch2/catch.hpp>

SCENARIO("`Config`s can be serialized") {
    REQUIRE(true==true);
}

3. Build the file
4. Try debugging the file via the Debug button directly above the SCENARIO statement
5. Results in the following unexpected behavior:

Filters: Scenario: s can be serialized
No test cases matched 'Scenario: s can be serialized'
===============================================================================
No tests ran

Note that clicking the Run button instead of Debug works fine. I suspect this is likely due to the shell parsing the back-ticks and evaluating them before actually running the command.

I would have included the logs, but I could not find them.

Desktop

• Extension Version 2.7.6
• VS Code Version 1.40.1
• Catch2 Version 2.11.0
• MacOS 10.14.6, but actual code and extension running in an Ubuntu 18.04 docker container

Describe the bug (A clear and concise description of what the bug is.)
Using windows. I have a system environment variable called "Path" that has several entries. Example from my settings.json:
"catch2TestExplorer.executables": [ { "name": "${filename}", "pattern": "C:/Projects/CMakeBuild/vscode/NSW-51770/Visual Studio Professional 2019 - x86/Debug/**/*{test,Test,TEST}*/*.exe", "dependsOn": ["C:/Projects/CMakeBuild/vscode/NSW-51770/Visual Studio Professional 2019 - x86/Debug/"], "env": {"Path": "C:/SDKs/Qt/Qt5.6.0-msvc2015/5.6/msvc2015/bin"} } ],
Test executables are not recognized because they fail to run as they are missing a bunch of libraries that are listed in my Path environment variable.

Expected behavior (A clear and concise description of what you expected to happen.)
System environment variables are not overwritten but only expanded by appending.

Additional context Add any other context about the problem here.
Or is there some other way to expand the environment for the executables? Also is it expected that vscode variables like ${env:Path} cannot be used inside the "env" in "catch2TestExplorer.executables" ?
Also is there a way to add multiple entries to one "env" variable like "Path" from the example. Something like
"env": {"Path": "C:/path1; C:/path2"}
?

Before you file an issue, please check the README.md's known issues section.

Describe the bug (A clear and concise description of what the bug is.)

In the test list of my project I find a strange test:
Running main() from gmock_main.c

Running this test does nothing.

To Reproduce (Steps to reproduce the behavior:)
It's always there.

Expected behavior (A clear and concise description of what you expected to happen.)
That test shouldn't be there.

Screenshots (If applicable, add screenshots to help explain your problem.)

Desktop (please complete the following information):

• Extension Version: 2.3.13
• VS Code Version:

Version: 1.30.2 (user setup)
Commit: 61122f88f0bf01e2ac16bdb9e1bc4571755f5bd8
Date: 2019-01-07T22:54:13.295Z
Electron: 2.0.12
Chrome: 61.0.3163.100
Node.js: 8.9.3
V8: 6.1.534.41
OS: Windows_NT x64 10.0.15063

• Catch2/Google Test Version: GoogleTest: 1.7
• OS Type and Version: Windows 10 Pro

Log (Attach log. (optional, Set: catch2TestExplorer.logfile and reproduce the bug.) Warning: Log probably contains file and test names too.)

Additional context Add any other context about the problem here.

Situation

We test the libraries we develop using googletest. One of our library depend on an external (closed source, shipped as .so) library. This library is dynamically linked to the test executable. It requires some environment variables to be set, which are directories relative to the executable. This works when setting these in a shell script, or in terminal. When these environment variables are not set, the execution will end with an return code of 255.

Issue

We want to start using this plugin for automation in vscode, so we tried adding this to our settings.json:

"catch2TestExplorer.executables": [
        {
            "name": "${filename}",
            "description": "${relDirpath}/",
            "pattern": "folder/to/test/executables/*",
            "cwd":"${absDirpath}",
            "env": {
                "VARIABLE_1": "${absDirPath}/../relative/directory",
                "VARIABLE_2": "${absDirPath}/../other/relative/directory"
            }
        }
    ]

But this test executable does not appear in the test explorer, while others appear just fine. Also with logging output as seen below in mind I think that during the test discovery the environment variables do not get set correctly.

I cannot find the console output that happens during the test discovery. Maybe this can give some insight to what is happening.

Logging output:

[2019-08-08 13:49:38.754] [WARN] [at load.suiteCreationAndLoadingTasks.push.c2fs.isNativeExecutableAsync.then._createSuiteByUri.then (/home/me/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.6.6/out/src/TestExecutableInfo.js:73:38)] Not a test executable: /directory/to/test/executable reason: [Error: Not a supported test executable: /directory/to/test/executable
 output: [object Object]
	at determineTestTypeOfExecutable.c2fs.isNativeExecutableAsync.then.c2fs.spawnAsync.then (/home/me/.vscode/extensions/matepek.vscode-catch2-test-adapter-2.6.6/out/src/TestSuiteInfoFactory.js:42:23)]

Version numbers

• Extension Version 2.6.6
• VS Code 1.36.1
• Google Test Version 1.8.1
• Manjaro 18.0.4

I can use the catch2TestExplorer.workerMaxNumber setting to run executables in parallel. This is useful, but I have some executables that take much longer than others.

As motivation, imagine a set of test executables that need to be run. In this set, there is a single executable that will take many hours longer than all other executables combined when run in serial, while the other executables run in minutes/seconds. It can be batched and run in parallel in minutes. Any user who wanted to use vscode-catch2-test-adapter extension would never be able to run the test executables in an timely fashion.

I would like to run mutually exclusive subsets of tests from a single executable in parallel. I would like to add a entry to catch2TestExplorer.executables with a field that indicates how to split tests contained in matching executables.

I would prefer if it was something along the lines of "testExecutableInParallel: { "defaultBatchSize": 10, "egregiousFixture": 1 }" to indicate that a matching executable should have no more than 10 tests assigned to a worker thread and a fixture matching the substring/prefix "egregiousFixture" should have no more than 1 test running per worker thread.

While some might say that the user group needs to emit more executables or reduce test runtimes, the reality of the situation is that the user group will not use this extension because the costs of changing build systems and test infrastructure are high and they already have a solution that runs on the order of minutes.

This is a pretty terrible bug. When clicking the test extension icon(flask) on one of my projects that have no test cases the spinner continuously spins while spewing browser pop-ups of karma and multiple npm and browser processes. It's like massive DOS attack and it's difficult to even type anything anywhere. It uses all CPU cores to hog at 100% and the only way to stop it is to go to external console and killall vscode/npm/firefox.

Checklist (You can delete this section, don't have to be the part of your issue.)

• The issue is about this extension and NOT about a fork.
• Check the known issues list.
• The latest version of the extension was used.

Describe the bug (A clear and concise description of what the bug is.)
The tests executables can't be found if I add a cin funtion in the main function.
To Reproduce (Steps to reproduce the behavior.)

1. write a single googleTest sample with VSCode;
2. add a "cin" fuction in the main funtion and build everything;
3. Open Tests Explorer tab;
4. See that there is no tests, see first screenshot.
   5.if you delete the "cin" fuction and build everything again, you will find the tests in the Tests Explorer tab;

Screenshots (If applicable, add screenshots to help explain your problem.)

Desktop

• Extension Version [2.14.1]
• VS Code Version [1.37. 0]
• Catch2/Google Test Version [2.6.5]
• OS Type and Version:Windows 10, 64 Bit

Log (Attach log. (optional, Set: catch2TestExplorer.logfile and reproduce the bug.) Warning: Log probably contains file and test names too.)