I created a module forbidder some time ago and it began to "spread in the wild" already. If it's incorporated to the framework, it will be more easily maintainable. That might be worth of a shot... (but that will need some work/rewrite).

Currently both JS and Python versions have test constructs to test if some pieces of code runs under a certain time limit, but they do not allow the use of custom error message like every other test constructs do.

So, like @timeout(sec) should become @timeout(sec, msg) with a default value?

When solution contains an assert statement (simple form, without a message) and that assert fails, tests are aborted without any error messages at all. It's hard to understand what's going on.

To reproduce: train on any Python kata, submit a function with assert False in its body. Compare with assert False, "User assertion failed".

I'm not sure what can/should be done here, but it just felt very confusing, I had to debug my solution locally. I often add asserts to find bugs early. Without messages, of course. Initially I expected a full traceback, like from raise RuntimeError(). That would be very useful for debugging without a local IDE. But it looks like AssertionError is used by the test framework, so it's displayed differently.

ok, there are actually several problems about it.

Context

So far, it handles properly:

• error raised in the function of the user while running
• wrong output from the user
• execution time too long
• max buffer limit error

But it has two problems in its current state:

1. tests are passing if the process just crashes because it runs out of memory
2. The "nesting status" (describe/it) of the assertions done inside the timeout utility is ambiguous for one, and even worse, may be different depending on what the function passed by the user to the decorator is doing.

Developping point 1:

Used that way:

# differnet behaviours for the user function

def bufferLimitError():
    while 1: print('qkjdhgkjqhfgdg')
def slowAF():
    while 1: pass
def raiseError():  raise KeyError()
def testFails():   test.fail("")
def testPass():    test.pass_()
def floodRAM():    elemental_forms(random_evil_string(15000))    # this explodes the heap

BEHAVIOURS = (slowAF, raiseError, testFails, testPass, floodRAM, bufferLimitError)

@test.describe(f'timeout inside it blocks')
def tests():
    for i,behave in enumerate(BEHAVIOURS):
        @test.it(behave.__name__)
        def _():
            test.timeout(.1 if i!=4 else 8)(behave)

leads to floodRAM displaying no assertion result at all, meaning this is a "win" for the user if all other tests are passing.

Developping point 2:

Current state of the decorator:

def timeout(sec):
    def wrapper(func):
        from multiprocessing import Process
        msg = 'Should not throw any exceptions inside timeout'

        def wrapped():
            expect_no_error(msg, func)
            """ >>> func is supposed to do some assertions on the result of the user's function <<< """

        process = Process(target=wrapped)
        process.start()
        process.join(sec)
        if process.is_alive():
            fail('Exceeded time limit of {:.3f} seconds'.format(sec))
            """ >>> This is the problematic assertion <<< """
            process.terminate()
            process.join()
    return wrapper

So, imagining the following possible uses:

@test.describe(f'developping point 2')
def tests():
    @test.it('inside it block')
    def _():
        @test.timeout(1)
        def _():
            test.expect(True)
            
    @test.describe('wrapping describe/it blocks')
    def _():
        
        @test.timeout(1)
        def _():
            @test.it('lvl 1')
            def _():
                test.expect(True)
                
        @test.timeout(1)
        def _():
            @test.describe('meh')
            def _():
                @test.it('lvl 2')
                def _():
                    test.expect(True)

This leads to assertions that can be done outside of a @test.it block, which is what we are currently fighting against...
For now, the documentation tells nothing about that (about "how to use or not the timeout decorator", I mean)

Suggestions

Resolving point 1

I already found a way to handle this. I still have to test the behaviour in different situations to check that it works as expected.
The idea is to pass a Value to the Process, so that this value is updated only if the decorated function doesn't raise/crash. This value is examined afterward at the end of the wrapper to check that the process actually updated it

Resolving point 2

Here I'd need some guidance... I see mostly 3 ways out of this:

1. State in the documentation that the timeout utility has to be used only inside a @test.it block. This way, nothing to change in the code about that.
2. State in the documentation that the timeout utility has to be used wrapping the test.it block(s). Then we can add a new it block at the end of the wrapper to get the correct level.
3. the flexible way... but might lead to more troubles than anything: add a new argument to the decorator stating if the final assertions of the wrapper must be made insde a it block or not. I sort of like the idea, and dislike it at the same time... :/

What would you prefere?

Corollary:

I blieve that the tests checking the framework (python-test-framework/tests/fixtures/) need an update too, to test for all those scenarii (except maybe the buffer limit error?)

final question?

That may be the occasion to add the option asked by Voile (see #4). Even if it' effectively doesn't seem that useful (see FArekkusu's answer), I like the idea of giiving more flexibility to the user.
Go or no go?

We can't use assertions from libraries yet because of this.
See #5 (comment)

the expected value's == implementation should be prefered over whatever the solver returned

AssertionError doesn't show up in the output when it's raised inside a function decorated by test.describe, test.it, or inside any function called by those. It correctly stops the execution, emits a non-zero exit code, and results in a test failure, but there's no error message. The issue is present both in Python 3.6 and Python 3.8.

This works:

import codewars_test as test

assert 0

@test.describe("T")
def _():
    @test.it("t")
    def _():
        test.pass_()

While any of these doesn't:

import codewars_test as test

@test.describe("T")
def _():
    
    assert 0
    
    @test.it("t")
    def _():
        test.pass_()

import codewars_test as test

@test.it("t")
def _():
    assert 0
    test.pass_()

import codewars_test as test

# whether this is defined in the test file or imported from the solution doesn't matter
def f():
    assert 0

@test.it("t")
def _():
    f()
    test.pass_()

When a test suite times out, unless the kata author or solver has manually included a print line with flush=True, then no outputs are visible at in the results panel.

the current wording, while correct, seems to clash a bit with beginner expectations, it can easily be misread as "given input x, expected y"

the (especially beginner) solver is likely not even aware of what the assertion looks like, and therefore doesn't have the context for the message