Running all tests within a class on the same thread? about xunit.stafact HOT 12 OPEN

Thanks for sharing the repro project. I think I need to re-familiarize myself both with this project and how xunit does thread management since this issue and #53 may have overlap so I'd want to look at them together. I don't have time now to investigate though, but I'll keep it active.

from xunit.stafact.

No problem. My merge request for changing from static to dynamic resources was merged into AvalonDock today so the immediate problem is solved. If nothing else this thread might help others facing the same problem.

from xunit.stafact.

I think, this issue is related to #53.

from xunit.stafact.

I think, this issue is related to #53.

I agree. It seems like this issue would be solved if #53 was solved.

from xunit.stafact.

@eriove Why is it important to reuse a thread for all tests in a class? Are your tests sharing objects with each other? That tends to be discouraged so that one test doesn't discretely lead to another test's failure.

from xunit.stafact.

I think the source of my problem is that WPF applications usually have one UI thread, hence there might be assumptions in the WPF code or WPF component code that relies on that fact.

In this merge request you can see an example of that. I create a new DockingManager in each test (and there is no shared state between the tests). Despite this the ContextMenu (which isn't explicitly used in the test) gets reused. Since it was created on another thread it will throw when accessed on the current test's thread.

This StackOverflow question seems to suggest that there is a caching mechanism for ContextMenus.

I'm not quite sure where to solve this. When I'm back at work on Tuesday I will see if I can do some experiments with styles or just set the context menu explicitly to get around my particular problem. But this seems like a problem that could occur for other people and solving it here would solve it for everyone without having to track down what UI element that gets cached.

But I'm not sure, maybe the libraries should be left alone and I should just dig further in my own tests...

from xunit.stafact.

Aside: If you're testing WPF code, are you using [StaFact] or [WpfFact]? (this library defines both)
I don't think that switching would fix this problem, but in general [WpfFact] does a better job at emulating what WPF expects.

"Fixing" this would mean that all tests have to run sequentially instead of in parallel, which could be a huge takeback for some folks with many tests. The test title qualifies it with "within a class" but if ContextMenu really is retaining some state behind the scenes, wouldn't fixing it require that all tests run on the same thread, whether they belong to the same test class?

from xunit.stafact.

Sorry for taking so long to reply. It has taken a while to process this. I've been using [WpfFact] and as you thought it does not help.

You are right about needing to run all tests sequential to resolve the problem and that that would be a bad idea for projects with many UI dependent tests. I did not think about that. I'm lucky enough to only have 10 to 20 tests that needs to run on a STA-thread.

I've created a minimal project that reproduces the problem and the problem is that a style contains a ContextMenu. Styles are cached in a static cache so the context menu will be re-used between tests. By changing from StaticResource to DynamicResource the problem disappears.

I imagine that this wouldn't be an unusual problem. Do you have any other suggests on how to solve the problem?

from xunit.stafact.

Ok, so now that #53 is fixed and didn't impact this, I'm ready to revisit. Since so many other WpfFact tests don't hit this, and you have a workaround, we might entertain not "fixing" this here. If we did, I'd want to look for a way for a test project, class, or method to "opt in" so that tests that do not require sharing a single thread are not unnecessarily slowed down.

One obvious idea is to add an optional named argument to the WpfFactAttribute to specify that a single thread must be shared with all other tests that also set this boolean property. But maybe that's too tedious to maintain on every single appearance of the attribute. Would an assembly or class-level attribute be more appropriate?

from xunit.stafact.

I like the idea of an optional argument to WpfFactAttribute. It seems like this is an uncommon problem, so there shouldn't be many tests that need it. Optional arguments are much easier to discover than assembly level attributes and configuration files. Since it might be a couple of years between the times a single developer needs this, it is better to see the optional argument than to google for the error message. Especially since searching for dispatcher exceptions will give you pages of explanations for beginners on how to use a dispatcher.

from xunit.stafact.

I would prefer to use an assembly or class-level attribute as an alternative to a optional parameter. ALL my tests require that they run on the same thread, so it would have been a pain in the ass to have to add a parameter to each test.

from xunit.stafact.

I think at least class-level makes sense. If only a couple tests need it, people can put those in another class. I'm not against a test-level attribute as well, but maybe that can be based on demand.
Is anyone interested in trying out a change and sending a PR for review? I'm not promising to take iteration 1, of course.

from xunit.stafact.