gasparnagy / specflow.xunitadapter Goto Github PK

View Code? Open in Web Editor NEW

24.0 12.0 7.0 295 KB

xUnit adapter for SpecFlow that allows running scenarios without code generation

License: Apache License 2.0

Smalltalk 0.97% Batchfile 1.66% C# 93.61% Gherkin 3.75%

specflow plugin xunit

specflow.xunitadapter's Introduction

SpecFlow.xUnitAdapter

SpecFlow.xUnitAdapter is an xUnit adapter for SpecFlow that allows running scenarios without code generation.

Supports:

SpecFlow v2.3.1
xUnit v2.3 or above

Requires:

.NET Framework 4.5 or above (new project format has limited support, see below)
Visual Studio for Windows

License: Apache (https://github.com/gasparnagy/SpecFlow.xUnitAdapter/blob/master/LICENSE)

NuGet: https://www.nuget.org/packages/SpecFlow.xUnitAdapter

See my blog post (http://gasparnagy.com/2017/04/specflow-without-code-behind-files/) for more information and background, you can also look at the complete example at https://github.com/gasparnagy/SpecFlow.xUnitAdapter/tree/master/sample/MyCalculator or watch the video demo on YouTube.

Donate

If you like this plugin and would like to support developing this or similar plugins, please consider donating the project. (For receiving an invoice for your donation, please contact me upfront.)

Usage

Install plugin from NuGet into your SpecFlow project.

PM> Install-Package SpecFlow.xUnitAdapter

Disable code-behind code generation

To disable the code-behind code generation, you have to REMOVE (set it to empty) the "SpecFlowSingleFileGenerator" "Custom Tool" setting on the file properties of all feature files in Visual Studio.

If you do this, Visual Studio will automatically delete the code behind files and removes them from the project.

Make sure the feature files have a `SpecFlowFeature` or `SpecFlowEmbeddedFeature` build action.

The SpecFlow.xUnitAdapter currently processes feature files in the folder (or subfolders) of the SpecFlow project assembly (typically in bin\Debug) and when they are embedded into the assembly.

To achieve that, you should change the Build Action setting of the feature files to SpecFlowFeature if you want them to not be embedded, or SpecFlowEmbeddedFeature if you want them to be embedded in the assembly.

Setup for new project file format

When using the new project file format for .NET 4.5+ projects, the build actions SpecFlowFeature and SpecFlowEmbeddedFeature are not available. Instead of that, you need to set the Copy to Output Directory setting to Copy if newer to ensure that the feature files are copied to the output directory. (Source information will not be available in this case.)

Also to be able to use the extension, you have to manually add the file SpecFlow.xUnitAdapter.Config.cs to the project.

Install xUnit Visual Studio adapter

In order to run the tests from the Visual Studio Test Explorer Window, you can install the xUnit Visual Studio adapter.

PM> Install-Package xunit.runner.visualstudio

Limitations

The adapter is currently in BETA and there are some limitations.

If you use SpecFlowFeature: As Visual Studio Test Explorer window only triggers the re-discovery of tests when the output assembly changes, if you only change the feature files, but nothing else in the code, you need to Rebuild the project in order to see the changes. (This is important only if the change is related to the discovered tests, so for example if you change the scenario name or the examples section of a scenario outline.)
The adapter currently does not load the project-level feature file language setting (but assumes en-US).
For feature files with #language setting, the setting will only be used for data conversions if it is a specific culture (e.g. de-AT). For neutral languages (e.g. de), currently en-US is used for conversions.
The new project file format is not fully supported, see details in section Setup for new project file format above.

For diagnosing test discovery errors, you can add the following setting to the App.config file of the SpecFlow project. With this setting, the test discovery errors are displayed in the Tests pane of the output window.

<appSettings>
  <add key="xunit.diagnosticMessages" value="true"/>
</appSettings>

Release History

See https://github.com/gasparnagy/SpecFlow.xUnitAdapter/releases

specflow.xunitadapter's People

Contributors

Stargazers

Watchers

Forkers

christopherhaws richard-keast-bluefruit khan007 boriphuth mjalav teonivalois xlegalles

specflow.xunitadapter's Issues

Causes problem with ITestOutputHelper

In order to log messages to output when running xUnit tests, I'm using the method that is demonstrated in this sample: https://github.com/aptester/SpecflowSamples

When I add xUnitAdapter and remove the code behind files to this solution it's causing a problem with the context injection step:

    public CalcContext(ScenarioContext context)
    {
        ..
        Output = context.ScenarioContainer.Resolve<ITestOutputHelper>();
    }

Which throws

Message: BoDi.ObjectContainerException : Interface cannot be resolved: Xunit.Abstractions.ITestOutputHelper

Any idea why this is breaking or if there is a workaround?

"Could not find a source mapping" despite passing tests

When running the tests in the xunit console runner, it outputs "Could not find a source mapping". This sounds like something is wrong. But the tests actually pass. No showstopper for us, just thought you should know.

What about Specflow v3?

What about this attempt to run feature files w/o code behind in Specflow v3? I don't see any news anywhere...
Thank you

Source controller marked class files throw System.UnauthorizedAccessException during SourceMap

Whenever the feature file associated to source control which automaticaally marks file attribute as readOnly. Now during SourceMap when the build task try to write sourceMap content to file it fails with below exception -

_at System.IO._Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
at System.IO.File.AppendAllLines(String path, IEnumerable`1 contents)
at SpecFlow.xUnitAdapter.Build.SpecFlowSourceMapAppender.AppendSourceMap(ITaskItem item)
at SpecFlow.xUnitAdapter.Build.SpecFlowSourceMapAppender.Execute()

It's good idea to remove Readonly attribute before we write clone copy of feature file .

"Property value is not valid" when setting Build Action to "SpecFlowFeature"

I'm following the steps outlined in the readme file, but I cant complete step 2, since neither of SpecFlowFeature nor SpecFlowEmbeddedFeature are valid values for the Build Action property of my .feature file.

Anyone else ran in to this?

[Discussion] MSBuild Integration

I have created branches for a few features that rely on MSBuild tasks. I wanted to open a discussion to discuss if this is the approach we should take to make sure I am not going off in the complete wrong direction.

Some of the features I was able to get working using MSBuild tasks:

Adding source maps to the feature file build artifacts (embedded and non-embedded)
Feature file build validation that returns errors to the error log with the correct line/column numbers so IDE integration works.
Integration with the new csproj system without having to modify the csproj file

Should we be sticking to MSBuild integration as much as possible? I think this will give us the most flexibility. The more MSBuild integrated the process is, the more cross platform and tool agnostic it will be.

SpecFlow mutant architecture (noSpecFlow) based on targets (Web POM, APIs, Blockchain, and AI)

Unable to select Build Action as per Readme

In VS 2017, I Installed the necessary SpecFlow packages, made some feature files and emptied the "Custom Tool" property.
However, I can not enter SpecFlowFeature or SpecFlowEmbeddedFeature as "Property value is invalid".
I've double-checked all steps and seen your video (where you don't set it that way btw).
Did I miss something or is it a bug?

The test project builds fine, but on building it adds the .cs code-behind files again.
I should mention that I'm using .NET Core. The example project works fine.

Does not work with SpecFlow 2.4.0

Projects using this package do not build when using SpecFlow 2.4.0.

Is there any plan to progress this?

[Feature] Test Attachments

Before using SpecFlow, our team used MSTest via VSTest which has a feature where you could inject the TestContext into you TestClass allowing you to add files to your results. Behind the scenes, this would add the file as an attachment to the TRX file that was generated. When we switched to SpecFlow, I had to create a PowerShell script that runs after our tests execute that reads the logs for the tests, searches for a line that contains file:///C:/Path/To/Attachment.png and adds the relevant XML for making that an attachment in the TRX.

I know that the VSTest framework supports attachments via Microsoft.VisualStudio.TestPlatform.ObjectModel.TestResult.Attachments. It would be nice to see this added as a feature in SpecFlow so that I don't have to run a post testrun script.

I haven't done much research on how this could be implemented in SpecFlow, but I think the limitation right now is that SpecFlow doesn't use custom adapters for executing the tests (besides SpecRun), so it doesn't have access to the TestResults object. This means the only way to make this work would have been via the generators somehow.

Any thoughts on this? Should this issue be in the SpecFlow issue log? I put it here, because I think its related to the work being done in this repo more than SpecFlow itself.

Thanks!

Chris

error MSB4062: The "SpecFlow.xUnitAdapter.Build.SpecFlowSourceMapAppender" task could not be loaded from the assembly

@instecjason reported this issue as a follow up of #6

Exception: No matching step definition found for the step.

When I clone this repository and run the sample test, it works great!

But when I copied the source of the sample project to an existing solution and adjusted bits like assembly and namespaces used to fit in the overall existing solution, I get the test exception result below. But the suggested part of code in the bottom of the details below, is actually available. Any suggestions on how to fix this?

[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9590224]       Stack Trace:
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9590600]            at TechTalk.SpecFlow.UnitTestProvider.XUnitRuntimeProvider.TestPending(String message)
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9590965]            at TechTalk.SpecFlow.ErrorHandling.ErrorProvider.ThrowPendingError(TestStatus testStatus, String message)
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9591260]            at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9591479]            at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9591738]            at SpecFlow.xUnitAdapter.SpecFlowPlugin.Runners.ScenarioTestCaseRunner.ScenarioCleanup()
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9592026]            at SpecFlow.xUnitAdapter.SpecFlowPlugin.Runners.ScenarioTestCaseRunner.RunScenario(SpecFlowDocument gherkinDocument, Scenario scenario)
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9592299]            at SpecFlow.xUnitAdapter.SpecFlowPlugin.Runners.ScenarioTestCaseRunner.<>c__DisplayClass9_0.<RunTestAsync>b__2()
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9592817]       Output:
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9593222]         Given I have entered the following numbers
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9593459]           --- table step argument ---
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9593670]           | number |
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9593886]           | 29     |
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9594068]           | 13     |
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9594228]         -> No matching step definition found for the step. Use the following code to create one:
[1/18/2018 12:18:37 PM Informational] [xUnit.net 00:00:01.9594400]                 [Given(@"I have entered the following numbers")]
public void GivenIHaveEnteredTheFollowingNumbers(Table table)
{
    ScenarioContext.Current.Pending();
}

Missing "Feature Execution Details" from test execution report

HI there,

First of all I just wanted to say that I love the work you are doing in this project!! Not having the code behind files is amazing, especially for separating the people writing the test cases from the ones developing the framework. Well done and keep up the great work!

Now this is probably something stupid that I am doing, but when generating a Specflow HTML report, I am not getting any details in the report. I get the header level summary data (how many tests passed/failed etc) but no list of features or details of scenarios. Looking in the XML report that comes out of XUnit, I can see that there is no detail other than the overall outcome.

          <test-suite type="TestCollection" executed="True" name="Default Collection" result="Success" success="True" time="35.944">
            <results>
              <test-case name="Create Full Case with All Fields" executed="True" result="Success" success="True" time="35.9417168">
                <properties>
                  <property name="Category" value="functional" />
                </properties>
              </test-case>

Any ideas what I am missing?

Some info about my project:

I am using VS2017 and .NEt 4.7.1
I have upgraded to the latest versions (nuget packages) of Specflow, xUnit, xUnit.Runner and this project (v 1.2.0)
I have set my feature files to BuildAction SpecFlowFeature and copy always.
I am building and running from commandline using Debug configuration (in case this is needed for full details)
everything seems to run correctly in VisualStudio and I get meaningful output in the test window including full given/when/then details and outcomes when I open the test outcome window by clicking the 'output' link in the Test Explorer View/Results window.

Let me know what else I can try or provide as info, as I say probably something stupid I am doing at this end, but wondered if others have seen this.

Thanks, Rob

[Feature] Support standard xunit tests

Add support for having a standard xUnit test in the same project as a SpecFlow test. Currently, since we are using a custom TestFramework, standard xUnit tests get filtered out. We will need to modify SpecFlowTestDiscoverer to support this, probably by inheriting XunitTestFrameworkDiscoverer or by creating a wrapper that calls into both types.

Deserialize Table to List<POCO>

Is there a deserialization mechanism available to convert a Table to a strong typed collection?

Breakpoints in feature files ignored

Just noticed that in feature files I've converted to remove code behind, Visual Studio ignores break points set in the feature file when debugging.

Still works on breakpoints set in the code in the step definitions

Tests not discovered after updating xunit package from 2.3.1 to 2.4.0 (new csproj)

We're using xUnitAdapter with new csproj targeting net461 framework. Everything was working fine until recent update of xUnit NuGet package to version 2.4.0. Since then no tests are being discovered.

I'm attaching MyCalculator sample project converted to new csproj format with SpecFlow.xUnitAdapter.Config.cs file added and using SpecFlowEmbeddedFeature to add feature files.
Specflow.xUnitAdapter.xUnit24.Repro.zip

If I switch the package version back to 2.3.1 all test are discovered and pass as expected.

Scenarios don't show in resharper

Hi,

I followed your instructions in your readme file and got it working under mstest explorer but when i try to see my scenarios in resharper they don't appear. Is there anything I need to do to get it to work with resharper?

Thanks!

VS for Mac support

Hello,

Does this work with VS for Mac?

What changes would be needed to make it work with VS for Mac?

Thank you,
Cosmin