Context

Since upgrading to Xcode 8.3 and Swift 3.1, we encounter the following warning:

Method 'initialize()' defines Objective-C class method 'initialize', which is not guaranteed to be invoked by Swift and will be disallowed in future versions

Currently the NativeTestCase relies on behaviour provided by + initialize to work.

Since we merged #50, NativeTestCase and NativeRunner have similar functionality. NativeRunner doesn't have the limitations of NativeTestCase, but NativeTestCase has the benefit of more meaningful test names. NativeRunner also has an API that feels more natural.

I think for now there's value in keeping both NativeTestCase and NativeRunner.

Possibilities

1. We re-write NativeTestCase in Objective-C. There are some downsides to this like compile times would increase although perhaps only negligibly.
2. Deprecate NativeTestCase and find a way to incorporate its advantages to NativeRunner. Then remove NativeTestCase for 1.0 to remove duplication and reduce API surface. I think we also need to decide if there's any value in having two ways of doing native feature files, especially if we can clean up the implementations and are able to implement #55.
3. Implement something similar to this: http://stackoverflow.com/a/42824542/1027644
4. ???

I'd love to hear some feedback from @smaljaar and any other users of the library to hear use cases.

🌮 🚀

I recently received an email with the (very reasonable) question as to why this library isn't at version 1. All the reasons I had to not do this have gone away, so should I release a version 1?

Thoughts? @kerrmarin @ilyapuchka

As far as I can see, the only real downside is that we would have more pressure to keep the api backwards compatible - but that's something we are doing anyway!

Version 0.10.2 doesn't include commit 17ec271 with the changes to ClassHelperMethods.swift. Our project tests fail due to that crash, and even if we have been able to work around this by pulling the latest master, it'll be nice to have a pod version to point to.

I can create a PR with a version bump myself to 0.10.3 if everyone is ok with it. Just wanted to check if that fits with your roadmap.

Currently all step definers, as in subclasses of StepDefiner, are loaded by default before test case starts, which may require to make sure that steps expressions are not ambiguous and results in issues like #17
Probably the simplest solution to that would be to allow each test to specify what step definers it uses to load only their steps.
This behaviour is also inconsistent in native and core tests, which is fixed with #131

Thoughts @kerrmarin @deanWombourne ?

All's been fine until this morning's update to Xcode 8.3.

Now the UI tests are failing with a "exc_bad_access" at line 35 of ClassHelperMethods.swift:
"if class_getSuperclass(currentClass) is T"

Enabling Zombie objects didn't give any additional output.

Anyone else using Xcode 8.3?

Feature files should allow the Example block to come after the Outline i.e.

Outline {
  Given("Some step with a <field_name>")
}

Examples(
   [ "field_name" ], 
   [ "value 1"],
   [ "value 2"])

should be a valid scenario.

I'm sure you've already looked into this but is there any reason as to why the scenarios written using XCTest-Gherkin are separate XCTestCases within a XCTestSuite?

Just wondered if it might make controlling the full set of tests a bit easier? Giving you access to individual teardowns and startups for each case etc

It's be nice if the suite would tell you if a step is too similar to the next one. In this case, it reads the third step and executes the second step again. Screenshots attached.

Include how to update the changelog

Steps definers are loaded only once, when first test is started, so they will keep the reference to it. When next step starts this reference is not updated, so if steps definers try to use it to access any property of currently running test case they will actually get values from previous tests. We are using launchArguments property in our XCTestCase subclasses to configure the app and some steps are designed to change these arguments. So when several tests are running with the same steps, they will all change first test property instead of the currently running test.

Hey guys,

I am experiencing a crash when trying to run tests with the GoogleTagManager cocoapod added to my project.

I've attached a sample project that demonstrates the issue.

Steps to reproduce: (the easy way)

1. Open the attached project's workspace - GherkinCrash.xcworkspace
2. Run the tests: ⌘+U
3. Observe the crash

Steps to reproduce: (the harder way)

1. Create an Xcode project.
2. Add pod 'GoogleTagManager', '~> 6.0' to the Podfile of the project.
3. pod install and open the workspace.
4. Add the XCTest-Gherkin files manually into the project.
5. Write a test using NativeRunner.
6. Try to run the test.
7. EXC_BAD_ACCESS in ClassHelperMethods

GherkinCrash.zip

When compiling XCTest-Gherking v. 0.14.0 with XCode Version 9.3.1 (9E501) I got:

./Pods/XCTest-Gherkin/Pod/Core/ClassHelperMethods.swift:53:46: 
Cannot convert value of type 'AutoreleasingUnsafeMutablePointer<AnyClass>' 
(aka 'AutoreleasingUnsafeMutablePointer<AnyObject.Type>') 
to expected argument type 'AutoreleasingUnsafeMutablePointer<AnyClass?>!' 
(aka 'ImplicitlyUnwrappedOptional<AutoreleasingUnsafeMutablePointer<Optional<AnyObject.Type>>>')

I am experiencing a strange issue.

If I have a step named as "I tap email button" it is simply not executed. It is not matter if it is Given/And/When/Then ... step("I tap email button") {...} is simply not executing.

The problem is "email button". If you put these two words in any existing step names, the step is simply not executed. Which is quite disturbing if you need to test inputs for... email button :)

Can I hope this will be fixed in the near future?

Given the following users exist:
| name | email | twitter |
| Aslak | [email protected] | @aslak_hellesoy |
| Julien | [email protected] | @jbpros |
| Matt | [email protected] | @mattwynne |

Can I use data tables with XCTest-Gherkin? If yes, then, what is the correct way to define steps?

ld: '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/Frameworks/XCTest.framework/XCTest' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I tried "ENABLE_BITCODE = No" it dint work, any help would be great!

How can I install this library manually without using CocoaPods?

I added Example/Pods/Pod.xcodeproj to my project, and XCTest-Gherkin (Pod) "Target Dependency" and "XCTest_Gherkin.framework" in "Link Binary With Libraries". Still doesn't seem to recognize the library.

I've tried to change the pod project to Swift 4 but obviously I don't have the rights to push up a branch - is there any reason as to why this pod shouldn't be bumped?

This shouldn't need any code changes, just adding a package.swift pointing at the right places for the source code.

Our primary method of distribution will still be cocoapods, so we should leave everything in the default place for that and create a custom swift package manager config.

I noticed that there is currently different behavior when it comes to validating steps in NativeScenario and native tests. In native tests, there is no error because the first match is picked up, but in NativeScenario it results in ambiguity. Here is the example:

step("I'm logged in as (?!known)(.+)")
step("I'm logged in as (?<aKnownUser>Alice|Bob)")

// in feature file causes error
Given I'm logged in as Bob

// in native test - fine
Given("I'm logged in as Bob")

It is definitely possible to rewrite those two steps to avoid ambiguity but I think framework should behave the same way in both scenarios.

I have for example these steps defined:

step("This is a cool test step") {
   ...        
}
        
step("This is a cool test step with extras") {
   ...
}

In the feature file I use this step: "This is a cool test step with extras".
This leads to an error "Multiple step definitions found for: This is a cool test step with extras".

In the source I saw that this is caused by de regex to find the step definition.

There was a warning in Xcode 8.3 in NativeTestCase.swift
Method 'initialize()' defines Objective-C class method 'initialize', which is not guaranteed to be invoked by Swift and will be disallowed in future versions

@kerrmarin implemented an alternative in f7d5906 which removes the warning, but this does not work. The tests are not executed anymore.

The tests driven by the NativeRunner are working. My suggestion would be to delete the NativeTestCase.swift class and only use NativeRunner to drive test cases that are driven by native Gherkin feature files. This also cleans up some duplicate code between these classes.

Hi, I have found an issue during writing tests using Gherkin. I need to call multiple test methods from one single method but it crashes here with unexpected nil in self.invocation

let rawName = String(describing: self.invocation!.selector)

I have test func with this body

    func testUsingsFunctions() {
        let login = GherkinLogin()
        login.testLoginSuccess()
        
        let payments = GherkinPayments()
        payments.testPaymentDetail()
    }

and calling method in this class

import XCTest
import XCTest_Gherkin

class GherkinLogin: XCTestCase {
    
    //MARK: Test cases
    func testLoginSuccess() {
        Given("Launched app")
        When("I fill password qawsed")
        Then("I tap on Login button")
        And("I see toolbar with title After login")
    }
}

is here any solution?

Thanks a lot! 🙌

Hi there! When I try to run the UI Tests I get the following error message:

Test target XcuitUITests encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted)

I'm using Xcode 7.3.1 and Cocoapods 1.0.1. You can check the project here: https://github.com/jgonfer/xcode-ui-testing-cheat-sheet

I've also created a blank new project with just the XCTest-Gherkin pod and I get the same error.

Hi,
We are looking for installation of XCTest-Gherkin using carthage.

Hi mates! I want to use the latest version 0.12 in our project but I'm not able because the project crashes. The commit 4d2a390 includes all the necessary changes to fix the crash.

Do you have plans to release the next version?

Hey Sam, since @ilyapuchka has been really pushing this forward the past few weeks I think it would be nice if he had merge access - if we can't do this because we're still under the NAP umbrella perhaps this is a good time to fork into our own org?

Hi everyone,
Are you going to add Swift 2.3 or 3.0 support anytime soon? When this can be?

Hello, its me again :)

One of main advantages I can see for using the Core is an option to run each scenario (implemented as a function) separately. But using Core needs manual writing which kinda sucks.

Native allows automation. Tests runs automatically but debugging optionsare limited here (as written in the README).

Is it possible to enhnace Native implementation with an option to run scenarios separately? Eg with help of NativeFeature Parser generate scenarios as functions into a separate .swift file as XCtest functions so that a tester could run scenarios the same way as with Core.

Currently, when using the wrong variable in scenario outline, it is left unchanged in the step expression, resulting in the step being invoked with a placeholder as a value which then may lead to test failure. Instead if should raise precondition as this is a programmer error to use wrong variable name in steps. I.e.

        Examples(
            [ "flow"],
            [ "Nutrition"],
            [ "Activity"],
            [ "Mood"],
            [ "Full Assessment"]
        )
        
        Outline(shouldTerminate: true) {
            And("log in as a new user using \"United Kingdom\" credentials and \"sentwithlove\" registration code")
            And("I see the home screen")
            And("I tap on HEALTHCHECK in the application tab bar")
            And("the user navigates to the select main focus screen")
            And("I select \"<flow>\" assessment flow") // ----> correct name
            Then("the user starts the \"<focus>\" onboarding questionnaire") // ----> wrong name due to programmer error, should cause exception instead of running step normally
            And("I close the chatbot screen")
            And("I tap on HOME in the application tab bar")
        }

We're marking lots of things as open when I don't think they need/should be i.e. the step(_:file:line:function:) methods in StepDefiner. They should be public, but I don't see why they need to be overridable.

I think the Xcode auto migrate tool has been a bit liberal, and I should probably have checked more closely!

Issue

Native Gherkin lines are not clearly traceable in Xcode results output

Solution

New XCTContext.runActivity API in XCTest introduced in Xcode 9 makes it possible to wrap everything together that is executed within a native Gherkin step 🎉. See screenshot:

I am ready to make a pull request with this, but I decided to go via the Issue section first, especially because this is based on the Xcode 9 beta. See for the diff: https://github.com/net-a-porter-mobile/XCTest-Gherkin/compare/master...smaljaar:feature/xcode9-use-xcactivity-to-improve-native-logging?expand=1

Please let me know if any comments or suggestions.

Last time we implemented a lot of tests with XCTest-Gherkin. They are running smooth on a local machine. Now we want to run the tests in AWS Device Farm, they support XCTest UI, so our tests should run there.

But when I upload our UITests-Runner ipa to AWS and start the app, the tests are no recognised and no tests did run. At first I thought is has something to do with inheritance; if the class does not inherit from XCTestCase but some other class, that should probably not work. But I tried to implement my own inheritance chain based on NativeTestCase, but then the testcase was recognised in AWS and did run smoothly.

This does not work in AWS (it does on a local machine):

import XCTest
import XCTest_Gherkin

class MyTestCase  : NativeTestCase {
}

This does work in AWS:

import XCTest
import XCTest_Gherkin

class MyXCGNativeInitializer : XCTestCase {
}

class MyNativeTestCase : MyXCGNativeInitializer {
}

class MyTestCase  : MyNativeTestCase {
}

Although more an issue in XCode, the way Native tests are dynamically built up does not play nicely with XCode's test runner, for one it does not show up until after running.

I've created a native runner, with you create normal XCUnit tests - and you simply pass in the feature file and scenario name. It works pretty well for my use case (in which we share the feature files between Android and iOS).

I will create a PR for this once my unit tests are done - just want to know if you see a value for this additional Native runner?

When it comes to writing new test it might be challenging to find the right step to use, especially being unfamiliar with already existing tests. Using page objects and corresponding step definers helps but still can be not enough in large projects.

Another issue with steps being defined as strings is a way to discover them - it is only possible from Find Navigator, you can't simply navigate to the declaration of step as if it was a regular method.
All that makes it a bit cumbersome to discover steps and write new tests with already existing steps. Which can, for example, result in steps duplication.

One way of solving that that I was trying recently is to use code snippets for that. There might be a script that will automatically extract steps from step definers files, using their doc comments as a summary and copy into a destination where Xcode reads them from.

Will you be interested in including such a script with a library?

When using native feature file parsing the current implementation treats a "Scenario Outline" and all the permutations as a single Test this means the test isn't torn down between runs and the app isn't therefore restarted.

This is causing problems for us as each test in our scenario outline expects to start from the same place. If the app doesn't restart we would have to unwind all the steps.

There doesn't appear to be a hook where we can restart the app once a scenario from an outline as finished.

I am pretty new to XCtest and I am from Testing background, still trying to grab hold of swift and XCtest.
Issue: I have updated the Pod yesterday, since then I am seeing my test passing even though it is a fail(Only when multiple tests are run from a class), it is all good when that specific test is run(ie. the test fails).

I am using :

extension XCTestCase {

    func waitForHittable(element: XCUIElement, waitSeconds: Double, file: String = #file, line: UInt = #line) {
        let existsPredicate = NSPredicate(format: "hittable == true")
        expectation(for: existsPredicate, evaluatedWith: element, handler: nil)   

        waitForExpectations(timeout: waitSeconds) { (error) -> Void in
            if (error != nil) {
                let message = "Failed to find \(element) after \(waitSeconds) seconds."
                self.recordFailure(withDescription: message,
                    inFile: file, atLine: line, expected: true)
            }
        }
}

What I found was that when multiple tests are run and method is called , if it is actual failure in second testcase then the test just proceeds as if nothing happened and it passes (self.recordFailure seems not working as expected for second one).But on the other hand if the actual failure is in first test the testcase fails.

I have another machine with a bit older version of XCTest-Gherkin(around a month older version) and all is working as expected.Seems like the recent latest commit has broken something.

Not sure If this issue has something to do with #48 (May be similar)

We are currently using your library and ran into some trouble.
We have the same step definition in 3 different XCTestCase-classes and a StepDefiner-subclass per testcase.
The library only executes the last definition it found, thus not associating the right automation code with a test.

Request:
Make XCTestCase & StepDefiner mappable to avoid super long steps.
The current implementation as default.

We currently have around + 20 feature files to test, so this would be handy.

Coming from Cucumber and Ruby I noticed that step matching is done only using regular expressions.

In Ruby you can step match using string literals or regex literals (something Swift doesn't have).

We had a bug recently when porting out Ruby Cucumber tests to XCTest-Gherkin where we were unaware that Step("Blaaa") was actually using regex behind the scenes. This meant that multiple step definitions where matched. For example step:

When I do something

Would match both definitions:

Step("I do something")
Step("I do something else")

As the Step function expects a regular expression represented as a String literal, without the ^ and $ tokens a sub-match is possible as shown above.

In Ruby cucumber, both step definitions would be performed using equality on the string literal and not using regular expressions at all.

Ruby:

Step("I do something")
Step("I do something else")

Using regex is opted-in using regex literals:

Ruby:

Step(/^I do something$/)
Step(/^I do something else$/)

So there is no confusion here that one us using regular expressions and the other is using String matching.

Could we add support for String matching by default and regular expressions by opt-in so that it's clear how steps are matched? String matching should be faster than regex and for most cases, I expect string matching is what you need most of the time.

Ideally an API like:

Step(_ string: String)
Step(_ regex: NSRegularExpression)

Would be ideal, I was thinking of how we could reduce the boilerplate of having to create NSRegularExpression objects and we could introduce a regex literal type using a custom operator, to simplify how you would use the second method above, something like this:

Step(~="My regex [0-9]*")

What do you think, I'm happy to have a go and create a PR?

This is simplistic example but proves my point.

Given I have a feature file like:

Scenario Outline: My test outline
    Given I enter the name <name>

    Examples:
    | name       |
    | Jill Smith |

and the step definition:

step("I enter the name (.+)") { (match: Person) in 
    ...
}

and the model:

struct Person: CodableMatchedStringRepresentable {
    let name: String
    ...
    func doSomethingToPerson() {
        ...
    }
}

In the above example, the expression will be I enter the name (Jill Smith) and I would like to turn the String "Jill Smith" from the capture group into the type Person in my step definition.

I note that you can do this when using native Swift based feature steps as shown in your README by resolving the Person instance into JSON via the description property.

Given("User is loggeed in as \(Person(name: "Nick"))")

I assume what happens here is Person(name: "Nick") is encoded into JSON so the expression becomes: "User is logged in as ({"name": "Nick"}))" and then when the test is run it encodes that back into the Person type?

I assume in my situation I would have to write the feature file like:

Scenario Outline: My test outline
    Given I enter the name {"name": "<name>"}

    Examples:
    | name       |
    | Jill Smith |

But this is less than ideal.

Is there another way I achieve this? Thanks.

When multiple scenario's are defined in one native feature file they are executed in reversed alphabetical order. This is caused by the line of code in NativeTestCase.swift:

testCaseClass.testInvocations().sort { (a,b) in NSStringFromSelector(a.selector) > NSStringFromSelector(b.selector) }

Without this sort operation the test invocations (aka 'scenarios') are executed in alphabetical order, which is natural for tests in an XCTestCase class. However, I found this unexpected behaviour when executing multiple scenario's from one native feature file. I would expect the scenario's to be executed in the order that they are specified.

My proposal is to remove the sort operation, and implement/refactor 'something' which makes sure that the scenario's are executed in the order that they are specified in the native feature file.

Any ideas, suggestions?

A script which would run over a feature file and create a stubbed XCTestCase implementation in swift.

* or similar

After updating to 0.13.2, all test runs started to have the following compile errors. Nothing else in the code has changed.
I got it working for now by going back to 0.12.0 in the podfile.

Swift Compile Error:
Pods/XCTest-Gherkin/Pod/Core/ClassHelperMethods.swift:53:46: Cannot convert value of type 'AutoreleasingUnsafeMutablePointer' (aka 'AutoreleasingUnsafeMutablePointer<AnyObject.Type>') to expected argument type 'AutoreleasingUnsafeMutablePointer<AnyClass?>!' (aka 'ImplicitlyUnwrappedOptional<AutoreleasingUnsafeMutablePointer<Optional<AnyObject.Type>>>')

Pods/XCTest-Gherkin/Pod/Core/XCTestCase+Gherkin.swift:286:35: 'range(at:)' has been renamed to 'rangeAt(_:)'

/Pods/Foundation.NSTextCheckingResult:28:15: 'range(at:)' was introduced in Swift 4

Hello,

I noticed an issue with getting false positive when using the XCTest method recordFailure.

extension XCTestCase {

    func waitForElementToAppear(element: XCUIElement, file: String = #file, line: UInt = #line) {
        let existsPredicate = NSPredicate(format: "exists == true")
        expectation(for: existsPredicate, evaluatedWith: element, handler: nil)

        waitForExpectations(timeout: 2) { (error) -> Void in
            if (error != nil) {
                let message = "Failed to find \(element) after 2 seconds."

                //Issue: self.recordFailure does not result in a failed test case the second time it is called. 
                //This issue does not occur while using regular subclasses of XCTestCase, so something in XCTest-Gherkin causes this not to work properly.
                self.recordFailure(withDescription: message, inFile: file, atLine: UInt(arc4random()), expected: true)

            }
        }
    }
}

The issue occurs if you call the method waitForElementToAppear for a second time from a StepDefiner subclass.

Both Feature tags and Scenario tags

Currently, unimplemented steps result in a fatal error, this is the opposite to Cucumber which allows unimplemented steps and entire scenarios.

It would really aid the development of tests if undefined steps were allowed, may be raised as warnings instead. In my opinion, this should be the default behaviour, but at a minimum a setting that we can toggle.

Background

When porting our .feature files directly from Cucumber to XCTest-Gherkin and using the XCTest-Gherkin/Native to parse our .feature files directly, often we won't have implemented all the steps definitions yet and won't always do so in the order they are defined in the feature file.

This is especially true when working in teams where another team member has already created a step definition that is also used in your .feature but has not merged into master yet. I would like to be able to test the other step definitions in the meantime without having to comment-out lines in the .feature files.

Also, the Gherkin language is designed to be human readable by stakeholders not just developers and it's not unreasonable to have .feature files that don't even have tests yet as the .feature files still represent documentation for the wider business.

This should just be a trivial s/xcode8.2/xcode8.3 in the .travis.yml config file

Hello! Today I ran in an issue using XCTest-Gherkin with Xcode 8. I can successfully use it and run the tests, until at the end of the run, Xcode crashes and it says:

DVTAssertions: ASSERTION FAILURE in /Library/Caches/com.apple.xbs/Sources/IDEFrameworks/IDEFrameworks-11246/IDEFoundation/Testing/IDETestRunSession.m:399
Details: Unexpected test identifier SomethingSomething/testRunNativeTests() (expected (null)).

So XCTest does not seem to like this testRunNativeTests method which dynamically creates the other tests. I have tried removing the subclassing of NativeTestCase and setting the path directly within testRunNativeTests but it gave the same crash. Any ideas what to do or how to fix this?

Update: I just checked out the latest source from github XCTest-Gherkin and you can reproduce the issue there yourself. Steps:

1. open the file XCTest-Gherkin.xcworkspace
2. hit Command + U
3. See Xcode 8 crash

However, if you only run the tests that have Gherkin specified in Swift syntax, then Xcode does not crash. So the issue seem purely to be with the NativeTestCase class

Hi I am wondering if there is a plan to create and share Carthage files with you library. We do not use pods in our team.