gasparnagy / specflow.xunitadapter Goto Github PK

Before using SpecFlow, our team used MSTest via VSTest which has a feature where you could inject the TestContext into you TestClass allowing you to add files to your results. Behind the scenes, this would add the file as an attachment to the TRX file that was generated. When we switched to SpecFlow, I had to create a PowerShell script that runs after our tests execute that reads the logs for the tests, searches for a line that contains file:///C:/Path/To/Attachment.png and adds the relevant XML for making that an attachment in the TRX.

I know that the VSTest framework supports attachments via Microsoft.VisualStudio.TestPlatform.ObjectModel.TestResult.Attachments. It would be nice to see this added as a feature in SpecFlow so that I don't have to run a post testrun script.

I haven't done much research on how this could be implemented in SpecFlow, but I think the limitation right now is that SpecFlow doesn't use custom adapters for executing the tests (besides SpecRun), so it doesn't have access to the TestResults object. This means the only way to make this work would have been via the generators somehow.

Any thoughts on this? Should this issue be in the SpecFlow issue log? I put it here, because I think its related to the work being done in this repo more than SpecFlow itself.

Thanks!

Chris

When I clone this repository and run the sample test, it works great!

But when I copied the source of the sample project to an existing solution and adjusted bits like assembly and namespaces used to fit in the overall existing solution, I get the test exception result below. But the suggested part of code in the bottom of the details below, is actually available. Any suggestions on how to fix this?

[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9590224]       Stack Trace:
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9590600]            at TechTalk.SpecFlow.UnitTestProvider.XUnitRuntimeProvider.TestPending(String message)
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9590965]            at TechTalk.SpecFlow.ErrorHandling.ErrorProvider.ThrowPendingError(TestStatus testStatus, String message)
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9591260]            at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9591479]            at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9591738]            at SpecFlow.xUnitAdapter.SpecFlowPlugin.Runners.ScenarioTestCaseRunner.ScenarioCleanup()
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9592026]            at SpecFlow.xUnitAdapter.SpecFlowPlugin.Runners.ScenarioTestCaseRunner.RunScenario(SpecFlowDocument gherkinDocument, Scenario scenario)
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9592299]            at SpecFlow.xUnitAdapter.SpecFlowPlugin.Runners.ScenarioTestCaseRunner.<>c__DisplayClass9_0.<RunTestAsync>b__2()
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9592817]       Output:
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9593222]         Given I have entered the following numbers
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9593459]           --- table step argument ---
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9593670]           | number |
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9593886]           | 29     |
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9594068]           | 13     |
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9594228]         -> No matching step definition found for the step. Use the following code to create one:
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9594400]                 [Given(@"I have entered the following numbers")]
public void GivenIHaveEnteredTheFollowingNumbers(Table table)
{
    ScenarioContext.Current.Pending();
}

Projects using this package do not build when using SpecFlow 2.4.0.

Is there any plan to progress this?

@instecjason reported this issue as a follow up of #6

Whenever the feature file associated to source control which automaticaally marks file attribute as readOnly. Now during SourceMap when the build task try to write sourceMap content to file it fails with below exception -

_at System.IO._Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
at System.IO.File.AppendAllLines(String path, IEnumerable`1 contents)
at SpecFlow.xUnitAdapter.Build.SpecFlowSourceMapAppender.AppendSourceMap(ITaskItem item)
at SpecFlow.xUnitAdapter.Build.SpecFlowSourceMapAppender.Execute()

It's good idea to remove Readonly attribute before we write clone copy of feature file .

I'm following the steps outlined in the readme file, but I cant complete step 2, since neither of SpecFlowFeature nor SpecFlowEmbeddedFeature are valid values for the Build Action property of my .feature file.

Anyone else ran in to this?

In order to log messages to output when running xUnit tests, I'm using the method that is demonstrated in this sample: https://github.com/aptester/SpecflowSamples

When I add xUnitAdapter and remove the code behind files to this solution it's causing a problem with the context injection step:

    public CalcContext(ScenarioContext context)
    {
        ..
        Output = context.ScenarioContainer.Resolve<ITestOutputHelper>();
    }

Which throws

Message: BoDi.ObjectContainerException : Interface cannot be resolved: Xunit.Abstractions.ITestOutputHelper

Any idea why this is breaking or if there is a workaround?

I have created branches for a few features that rely on MSBuild tasks. I wanted to open a discussion to discuss if this is the approach we should take to make sure I am not going off in the complete wrong direction.

Some of the features I was able to get working using MSBuild tasks:

• Adding source maps to the feature file build artifacts (embedded and non-embedded)
• Feature file build validation that returns errors to the error log with the correct line/column numbers so IDE integration works.
• Integration with the new csproj system without having to modify the csproj file

Should we be sticking to MSBuild integration as much as possible? I think this will give us the most flexibility. The more MSBuild integrated the process is, the more cross platform and tool agnostic it will be.

We're using xUnitAdapter with new csproj targeting net461 framework. Everything was working fine until recent update of xUnit NuGet package to version 2.4.0. Since then no tests are being discovered.

I'm attaching MyCalculator sample project converted to new csproj format with SpecFlow.xUnitAdapter.Config.cs file added and using SpecFlowEmbeddedFeature to add feature files.
Specflow.xUnitAdapter.xUnit24.Repro.zip

If I switch the package version back to 2.3.1 all test are discovered and pass as expected.

Add support for having a standard xUnit test in the same project as a SpecFlow test. Currently, since we are using a custom TestFramework, standard xUnit tests get filtered out. We will need to modify SpecFlowTestDiscoverer to support this, probably by inheriting XunitTestFrameworkDiscoverer or by creating a wrapper that calls into both types.

Just noticed that in feature files I've converted to remove code behind, Visual Studio ignores break points set in the feature file when debugging.

Still works on breakpoints set in the code in the step definitions

In VS 2017, I Installed the necessary SpecFlow packages, made some feature files and emptied the "Custom Tool" property.
However, I can not enter SpecFlowFeature or SpecFlowEmbeddedFeature as "Property value is invalid".
I've double-checked all steps and seen your video (where you don't set it that way btw).
Did I miss something or is it a bug?

The test project builds fine, but on building it adds the .cs code-behind files again.
I should mention that I'm using .NET Core. The example project works fine.

HI there,

First of all I just wanted to say that I love the work you are doing in this project!! Not having the code behind files is amazing, especially for separating the people writing the test cases from the ones developing the framework. Well done and keep up the great work!

Now this is probably something stupid that I am doing, but when generating a Specflow HTML report, I am not getting any details in the report. I get the header level summary data (how many tests passed/failed etc) but no list of features or details of scenarios. Looking in the XML report that comes out of XUnit, I can see that there is no detail other than the overall outcome.

          <test-suite type="TestCollection" executed="True" name="Default Collection" result="Success" success="True" time="35.944">
            <results>
              <test-case name="Create Full Case with All Fields" executed="True" result="Success" success="True" time="35.9417168">
                <properties>
                  <property name="Category" value="functional" />
                </properties>
              </test-case>

Any ideas what I am missing?

Some info about my project:

• I am using VS2017 and .NEt 4.7.1
• I have upgraded to the latest versions (nuget packages) of Specflow, xUnit, xUnit.Runner and this project (v 1.2.0)
• I have set my feature files to BuildAction SpecFlowFeature and copy always.
• I am building and running from commandline using Debug configuration (in case this is needed for full details)
• everything seems to run correctly in VisualStudio and I get meaningful output in the test window including full given/when/then details and outcomes when I open the test outcome window by clicking the 'output' link in the Test Explorer View/Results window.

Let me know what else I can try or provide as info, as I say probably something stupid I am doing at this end, but wondered if others have seen this.

Thanks, Rob

Hi,

I followed your instructions in your readme file and got it working under mstest explorer but when i try to see my scenarios in resharper they don't appear. Is there anything I need to do to get it to work with resharper?

Thanks!

What about this attempt to run feature files w/o code behind in Specflow v3? I don't see any news anywhere...
Thank you

Hello,

Does this work with VS for Mac?

What changes would be needed to make it work with VS for Mac?

Thank you,
Cosmin

Hi

When running the tests in the xunit console runner, it outputs "Could not find a source mapping". This sounds like something is wrong. But the tests actually pass. No showstopper for us, just thought you should know.

Is there a deserialization mechanism available to convert a Table to a strong typed collection?