Comments (8)
Hey @DogeWowzie - this is the first I've heard of it. Unfortunately we don't have a lot of teams within Pivotal that are using Cedar much these days - possibly none at all. In short, we don't find out that there are problems and that Xcode and Cedar are no longer best friends until someone files a helpful issue, like yourself :)
Have you investigated at all yet ? Are you sure that a small test case illustrates the same problem as your codebase does ? If you create a bare-bones minimal test suite with a single test case, do you see the same behavior ?
During the next few weeks I'll try to dedicate some of my own time to look into this, but I'm a parent, have several days I'll be out of the office on vacation here in France, running interviews and managing a team in my office, so I can't promise a fix on any given timeline. That said, I love helping first-time contributors to open source projects and I'd love to help you out if you're interested in digging in. Let me know how I can best help you, if you'd like :)
from cedar.
@DogeWowzie have you tried to create a new, standalone project and see if you can reproduce the same behavior ? I've been running Cedar's specs today in Xcode 9.2 and I haven't yet been able to reproduce this yet.
The thing that would be the most beneficial to me is having a standalone project (ideally something I could put on my machine and debug) so that we have a realistic scenario to explore. I'd say the chances are pretty slim of me being able to fix the bug alone, unless I happen to come across some tests in Cedar's test suite that happen to reproduce the issue.
from cedar.
@DogeWowzie I've tried a few times to replicate this with Xcode 9.3, comparing my results with Xcode 8.3.3 and haven't seen any cases where specs have been running repeatedly - even for some of our larger specs where we have something like 200 examples inside a single spec file.
Have you found out anything new ? Ideally, in order to resolve this, I would want to be able to dig into a codebase that consistently reproduces this problem that I can run on my machine. If you can upload a stand-alone example of this as a repo on github that would allow me to make a lot more progress.
Barring that I'd love to know some other details that might help identify the problem:
- What exact version of Xcode are you running ?
- What simulator do you use ? (device and OS version)
- How do you run your tests ?
- Do you know if you are using a test bundle, or a "spec suite" (the answer to the question above might identify this)
- Is there anything special about the configuration for your test target ?
- Do your tests link against any interesting frameworks ?
- Can you tell me the threshold for the number of tests when the "Looping problem" occurs ?
- What kinds of things are your tests doing ? Are they testing
UIViewController
subclasses ? Are they testing simple value objects ? Are they testing the network ? You can test any number of things with Cedar, and it would be curious to know if there's any sort of pattern here...
from cedar.
Hey @tjarratt, sorry for the late reply, we took a closer look into our "looping" test target. We have three targets with specs, only one loops. This problem appeared since we updated to Xcode 9, we are currently running Xcode 9.31 with iOS 11.3 simulators.
Turns out specs are looping due to our excessive usage of spy_on
in that test target. Now our tests don't loop anymore when we perform a "clean build folder" with Xcode in advance. Everything seems fine now, but we were kind of confused by the fact that everything works just fine with Xcode 8.
from cedar.
Oh wow, that's really interesting. spy_on
and friends are some dark magic, for sure - I've seen a number of interesting bugs related to Cedar spies throughout the years, especially when you throw Swift into the mix.
I wish I had thought of cleaning the build folder. When strange things start to happen, that's usually my initial instinct. My best guess to explain this behavior is that perhaps there were some changes in XCTest between Xcode 8 and 9 and Xcode somehow managed to load multiple versions of the same spec that were compiled separately, but somehow loadable into the same process after you upgraded, if it was using the same build directory. Normally this changes between releases, but who knows in this case.
Are there any more issues to investigate, or do you feel we could close this issue for now ?
from cedar.
Hey @tjarratt, thank you for your reply.
Cedar tests in our project just work fine with Xcode 8, we tried to isolate the problem by deactivating "suspicious" spec classes, but the "looping problem" always occurs when the test target contains a certain number of specs, that's why we think it might be a general problem due to the recent XCTest changes of Xcode 9.
Furthermore, we feel like Cedar tends to be behave inconstantly in Xcode 9, some specs classes are only succeeding while being focussed. I appreciate your effort in having a look into this.
Thank you very much in advance :)
from cedar.
Regarding "cold folding" - I've opened up a separate issue to discuss that. Apparently apple broke it when they introduced a new editor in Xcode 9.
from cedar.
Hey @tjarratt, I feel like closing this issue for now. Please note that we refactored all the specs (removed the spy_on
s) and then managed to prevent the specs from looping by performing "clean build folder" afterwards. If we don't do that, looping does still occur, but the spec target itself feels more stable. Thank you very much for your patience :) Cheers
from cedar.
Related Issues (20)
- Console output is logged out of order HOT 1
- new project can't find cedar HOT 1
- Xcode 7.3 beta 5 errors on __weak in manual reference counting HOT 6
- Use of @import HOT 2
- Custom matchers fails to compile using Xcode 7.3 HOT 4
- Test observation issues HOT 2
- Weak References in XCode 7.3 HOT 4
- Playground support
- Installing template fails on AppCode 2016.1 HOT 1
- Cedar silently accepts misuse of the test structure DSL
- Running Individual Specs HOT 9
- Cedar, Swift and Protocol Declarations - Compiler error HOT 7
- testing watchOS with Cedar and Swift
- Cedar Standalone with XCode 8/ios-sim 5.0.8 HOT 4
- Cedar needs a sticker. HOT 4
- should equal throws exception when comparing const pointer to a non-const pointer HOT 5
- Cedar status HOT 1
- [Known Issue] Cedar's test specs cannot be folded in Xcode 9
- spy_on NSDate crashes on Xcode 10 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cedar.