mobilenativefoundation / bluepill Goto Github PK

We have both BPUtils::printInfo and BPUtils::printError, which is confusing.

Let’s nix printError and make printInfo:ERROR go to stderr.
Quite mode shouldn't suppress error messages.

Need help with the error.

[ INFO ] Using 781 of 1064 processes.
[ ERROR ] (2) Completed: [1] Launch Application
[ ERROR ] (2) Could not launch app and tests: The request to open "com.xyz.internal.debug" failed.
[ INFO ] (2) [1] Delete Simulator
[ ERROR ] (3) Completed: [1] Launch Application
[ ERROR ] (3) Could not launch app and tests: The request to open "com.xyz.internal.debug" failed.

My English is not good, So you may not understand what I mean.when i use bluepill, and I have some problems.These are my input codes:

./bluepill -a /Users/Wil/Library/Developer/Xcode/DerivedData/someApp-dtewmgszpypiziaywgdbcnictchx/Build/Products/Debug-iphonesimulator/someApp.app  -s /Users/Wil/Desktop/bluepill_build/someApp.xcodeproj/xcuserdata/Wil.xcuserdatad/xcschemes/someApp_Tests.xcscheme -o build/output/ -r com.apple.CoreSimulator.SimRuntime.iOS-10-0

and I got this information

[  INFO  ] This is Bluepill v0.1.0-33-g0d56983
[ WARNING] Trying to pack too few tests (3) into too many bundles (4).
[ WARNING] Lowering number of simulators from 4 to 3 because there aren't enough tests.
[  INFO  ] Running with 3 simulators.
[  INFO  ] Packed tests into 3 bundles
[  INFO  ] Started Simulator 1 (PID 11174).
bp: invalid configuration
	com.apple.CoreSimulator.SimRuntime.iOS-10-0 is not a valid runtime.
Use `xcrun simctl list runtimes` for a list of valid runtimes.
Usage:
-a | --app
		The path to the host application to execute (your .app)

-o | --output-dir
		Directory where to put output log files (bluepill only).

-s | --scheme-path
		The scheme to run tests.

-d | --device
		On which device to run the app. (Default 'iPhone 6').

-c | --config
		Read options from the specified configuration file instead of the command line

-C | --repeat-count
		Number of times we'll run the entire test suite (used for stability testing). (Default '1').

-N | --no-split
		A list of NO split test bundles

-P | --print-config
		Print a configuration file suitable for passing back using the `-c` option. (Default 'stdout').

-R | --error-retries
		Number of times we'll recover from crashes to continue running the current test suite. (Default '4').

-S | --stuck-timeout
		Timeout in seconds for a test that seems stuck (no output). (Default '300').

-T | --test-timeout
		Timeout in seconds for a test that is producing output. (Default '300').

-f | --failure-tolerance
		The number of retries on any failures (app crash/test failure).

-i | --include
		Include a testcase in the set of tests to run (unless specified in `exclude`).

-n | --num-sims
		Number of simulators to run in parallel. (bluepill only) (Default '4').

-r | --runtime
		What runtime to use. (Default 'iOS 10.2').

-t | --test
		The path to the test bundle to execute (your .xctest).

-x | --exclude
		Exclude a testcase in the set of tests to run (takes priority over `include`).

-X | --xcode-path
		Path to xcode.

-H | --headless
		Run in headless mode (no GUI). (Default 'Off').

-J | --json-output
		Print test timing information in JSON format. (Default 'Off').

-h | --help
		This help.

-p | --plain-output
		Print results in plain text. (Default 'Off').

-q | --quiet
		Turn off all output except fatal errors. (Default 'Off').

-j | --junit-output
		Print results in JUnit format. (Default 'Off').

-l | --list-tests
		Only list tests in bundle

--additional-xctests
		Additional XCTest bundles to test.

[  INFO  ] Simulator 1 (PID 11174) has finished with exit code 1.
[  INFO  ] PID 11174 exited 1.
[  INFO  ] 0 Simulators still running. []
[  INFO  ] Using 387 of 1064 processes.
[  INFO  ] Started Simulator 2 (PID 11177).
bp: invalid configuration
	com.apple.CoreSimulator.SimRuntime.iOS-10-0 is not a valid runtime.
Use `xcrun simctl list runtimes` for a list of valid runtimes.
Usage:
-a | --app
		The path to the host application to execute (your .app)

-o | --output-dir
		Directory where to put output log files (bluepill only).

-s | --scheme-path
		The scheme to run tests.

-d | --device
		On which device to run the app. (Default 'iPhone 6').

-c | --config
		Read options from the specified configuration file instead of the command line

-C | --repeat-count
		Number of times we'll run the entire test suite (used for stability testing). (Default '1').

-N | --no-split
		A list of NO split test bundles

-P | --print-config
		Print a configuration file suitable for passing back using the `-c` option. (Default 'stdout').

-R | --error-retries
		Number of times we'll recover from crashes to continue running the current test suite. (Default '4').

-S | --stuck-timeout
		Timeout in seconds for a test that seems stuck (no output). (Default '300').

-T | --test-timeout
		Timeout in seconds for a test that is producing output. (Default '300').

-f | --failure-tolerance
		The number of retries on any failures (app crash/test failure).

-i | --include
		Include a testcase in the set of tests to run (unless specified in `exclude`).

-n | --num-sims
		Number of simulators to run in parallel. (bluepill only) (Default '4').

-r | --runtime
		What runtime to use. (Default 'iOS 10.2').

-t | --test
		The path to the test bundle to execute (your .xctest).

-x | --exclude
		Exclude a testcase in the set of tests to run (takes priority over `include`).

-X | --xcode-path
		Path to xcode.

-H | --headless
		Run in headless mode (no GUI). (Default 'Off').

-J | --json-output
		Print test timing information in JSON format. (Default 'Off').

-h | --help
		This help.

-p | --plain-output
		Print results in plain text. (Default 'Off').

-q | --quiet
		Turn off all output except fatal errors. (Default 'Off').

-j | --junit-output
		Print results in JUnit format. (Default 'Off').

-l | --list-tests
		Only list tests in bundle

--additional-xctests
		Additional XCTest bundles to test.

[  INFO  ] Simulator 2 (PID 11177) has finished with exit code 1.
[  INFO  ] PID 11177 exited 1.
[  INFO  ] Started Simulator 3 (PID 11181).
bp: invalid configuration
	com.apple.CoreSimulator.SimRuntime.iOS-10-0 is not a valid runtime.
Use `xcrun simctl list runtimes` for a list of valid runtimes.
Usage:
-a | --app
		The path to the host application to execute (your .app)

-o | --output-dir
		Directory where to put output log files (bluepill only).

-s | --scheme-path
		The scheme to run tests.

-d | --device
		On which device to run the app. (Default 'iPhone 6').

-c | --config
		Read options from the specified configuration file instead of the command line

-C | --repeat-count
		Number of times we'll run the entire test suite (used for stability testing). (Default '1').

-N | --no-split
		A list of NO split test bundles

-P | --print-config
		Print a configuration file suitable for passing back using the `-c` option. (Default 'stdout').

-R | --error-retries
		Number of times we'll recover from crashes to continue running the current test suite. (Default '4').

-S | --stuck-timeout
		Timeout in seconds for a test that seems stuck (no output). (Default '300').

-T | --test-timeout
		Timeout in seconds for a test that is producing output. (Default '300').

-f | --failure-tolerance
		The number of retries on any failures (app crash/test failure).

-i | --include
		Include a testcase in the set of tests to run (unless specified in `exclude`).

-n | --num-sims
		Number of simulators to run in parallel. (bluepill only) (Default '4').

-r | --runtime
		What runtime to use. (Default 'iOS 10.2').

-t | --test
		The path to the test bundle to execute (your .xctest).

-x | --exclude
		Exclude a testcase in the set of tests to run (takes priority over `include`).

-X | --xcode-path
		Path to xcode.

-H | --headless
		Run in headless mode (no GUI). (Default 'Off').

-J | --json-output
		Print test timing information in JSON format. (Default 'Off').

-h | --help
		This help.

-p | --plain-output
		Print results in plain text. (Default 'Off').

-q | --quiet
		Turn off all output except fatal errors. (Default 'Off').

-j | --junit-output
		Print results in JUnit format. (Default 'Off').

-l | --list-tests
		Only list tests in bundle

--additional-xctests
		Additional XCTest bundles to test.

[  INFO  ] Simulator 3 (PID 11181) has finished with exit code 1.
[  INFO  ] PID 11181 exited 1.
[  INFO  ] All simulators have finished.

Use xcrun simctl list runtimes for a list of valid runtimes.When I use this operation, I get these valid runtimes:

== Runtimes ==
iOS 8.1 (8.1 - 12B411) (com.apple.CoreSimulator.SimRuntime.iOS-8-1)
iOS 9.0 (9.0 - 13A344) (com.apple.CoreSimulator.SimRuntime.iOS-9-0)
iOS 10.0 (10.0 - 14A345) (com.apple.CoreSimulator.SimRuntime.iOS-10-0)
tvOS 10.0 (10.0 - 14T328) (com.apple.CoreSimulator.SimRuntime.tvOS-10-0)
watchOS 3.0 (3.0 - 14S326) (com.apple.CoreSimulator.SimRuntime.watchOS-3-0)

i have these schemes in someApp.xcodeproj/xcuserdata/Wil.xcuserdatad/xcschemes:

someApp_Tests.xcscheme
someApp_UITests.xcscheme
someApp.xcscheme

and i have these .appin 'Build/Products/Debug-iphonesimulator'

somApp_UITests-Runnera.pp///This folder contains the `PlugIns` folder
someApp.app///This folder contains the `PlugIns` folder

I do not know what the problem, I can not run successfully, please help me solve it.thank you

In Bluepill-runner/Bluepill-runner/BPRunner.m, the onInterrupt interrupt handler is doing too much. It should just increment the variable, and the logic below the increment should be moved to the interrupt method.

We have some duplicated functions in SimulatorRunner, such as:

• (BOOL)installApplicationAndReturnError:(NSError *__autoreleasing *)error

• (BOOL)installAppWithBundleID:(NSString *)hostBundleID
  bundlePath:(NSString *)hostBundlePath
  device:(SimDevice *)device
  error:(NSError **)error

@ob We should definitely reduce the function count there. remove them and uplevel the call.

@vargon So rather than merging these duplicate methods as we previously discussed, it seems we should just locate them in different classes. The mixing of the static methods that can act on anything and the config-specific methods that know the current context is what's confusing. Confining them to separate files would be the way forward, I believe.

The various outputs of bluepill have become quite a mess. I'm opening this issue as a place for discussion about what the different output modes should be. My proposal is as follows:

• Have two first class options:
  • --quiet only prints ERROR messages in stderr
  • --debug prints DEBUGINFO level messages in stderr
    The default is to print ERROR, WARNING, and INFO messages.
• Add two environment variables: BP_QUIET_MODE and BP_DEBUG_MODE that can be used to set the two modes. The command line options override the environment.
• Remove the BPBuildScript environment variable and have bluepill.sh use the previously mentioned environment variables instead.

Thoughts?

I create a new project and run ./bluepill -c config.json,There are some errors,what should I do?

mactekiiMac:bluepill-v0.2.2 mac$ ./bluepill -c config.json
[  INFO  ] This is Bluepill v0.1.0-46-gd35f56c
[ WARNING] Lowering number of simulators from 4 to 2 because there aren't enough tests.
[  INFO  ] Running with 4 simulators.
[  INFO  ] Packed tests into 2 bundles
[  INFO  ] Started Simulator 1 (PID 2498).
2017-02-25 09:01:18.995 bp[2498:47348] +[SimServiceContext sharedServiceContextForDeveloperDir:error:]: unrecognized selector sent to class 0x10774e8c8
2017-02-25 09:01:18.998 bp[2498:47348] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '+[SimServiceContext sharedServiceContextForDeveloperDir:error:]: unrecognized selector sent to class 0x10774e8c8'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fffd36957bb __exceptionPreprocess + 171
	1   libobjc.A.dylib                     0x00007fffe7e02a2a objc_exception_throw + 48
	2   CoreFoundation                      0x00007fffd3715cc4 +[NSObject(NSObject) doesNotRecognizeSelector:] + 132
	3   CoreFoundation                      0x00007fffd3608fc3 ___forwarding___ + 1059
	4   CoreFoundation                      0x00007fffd3608b18 _CF_forwarding_prep_0 + 120
	5   bp                                  0x0000000106f9e780 -[BPConfiguration validateConfigWithError:] + 2895
	6   bp                                  0x0000000106fb8573 main + 632
	7   libdyld.dylib                       0x00007fffe86de255 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
[  INFO  ] Simulator 1 (PID 2498) has finished with exit code 6.
[  INFO  ] PID 2498 exited 6.
[  INFO  ] 0 Simulators still running. []
[  INFO  ] Using 264 of 1064 processes.
...

this is my config.json

{
   "app": "/Users/mac/Library/Developer/Xcode/DerivedData/bluepill-ezyxkypxgoiujngjzpwyqpshbatu/Build/Products/Debug-iphonesimulator/bluepill.app",
   "scheme-path": "/Users/mac/Documents/bluepill/bluepill.xcodeproj/xcuserdata/mac.xcuserdatad/xcschemes/bluepill.xcscheme",
   "output-dir": "./build/a/",
   "xcode-path": "/Applications/Xcode8.2.app",
   "runtime": "10.2"
}

Right now, bluepill will exit with different exit codes depending on the overall success/failure of the tests. However it doesn't distinguish success from failure in the printed messages. It would be nice if after the last line:

[  INFO  ] All simulators have finished.

it printed something along the lines of:

[ FAIL ] Tests failed

or

[ FAIL ] Failed to run tests: Could not create simulator

Or something like that.

I managed to run bluepill against one of my apps with a bunch of UI tests based on Xcode UI testing. However I'm getting the following error. A bit googling suggests that this might be due to UI testing having different test template/config than "regular" XCTests. Is Xcode UI testing supported by bluepill? :)

Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'No target application path specified via test configuration: <XCTestConfiguration: 0x60800011a9d0>

Thanks!

[self.device terminateApplicationWithID:self.hostBundleId error:&error]; in SimulatorMonitor.m doesn't work in iOS 10.2 runtime.

Hi, I'm not sure if I do it correctly, but here are my steps to run the demo

1. Run Bluepill.xcworkspace, scheme blueprint, I get bp executable inside build/Build/Products/Debug
2. Run BPSampleApp.xcodeproj, scheme BPSampleApp, and copy the BPSampleApp.app to the same folder as bp
3. Copy BPAppNegativeTests.xcscheme inside BPSampleApp.xcodeproj/xcshareddata/xcschemes to the same folder as bp
4. Run ./bp -a ./BPSampleApp.app -s ./BPAppNegativeTests.xcscheme -o ./output/ and get

bp: invalid configuration. No test bundle provided.

If I need to do these steps to get the bluepill to run, then I must be doing something wrong. Please help

It'd be great if the console reported an overall progress indicator for each running simulator.

Examples

[  INFO ] (1) [######          ] 40% Complete
[  INFO ] (2) [########        ] 55% Complete

[  INFO ] (1) [######          ] 40% Complete (21 passed, 4 failed, 2 crashed)
[  INFO ] (2) [########        ] 55% Complete (19 passed, 7 failed, 0 crashed)

Placeholder for Xcode 8.3 support.

We need two new fields in BPConfiguration.m:BPOptions, one to know whether this option applies to bp or bluepill or both, and another to know whether the actual option is required or not. Right now we have the no_argument, required, and optional part but that is for optarg from getopt.

Also, we need to stop printing the full usage on error since it's grown too big.

Maybe also create a man page... troff anyone?

Hi,

When I try to run my UI test then I get the below issue. Can you please help me to solve it?

ERROR: The folder “Plugins” doesn’t exist.

When running tests, it appears each simulator is running the same test...

[  INFO  ] 4 Simulators still running. [1, 2, 3, 4]
[  INFO  ] Using 794 of 1064 processes.
[ FAILED ] (3)  67.031106s HelpAndFeedback/testSubmittingFeedback
[ FAILED ] (3)  67.032047s HelpAndFeedback/testSubmittingFeedback
[ FAILED ] (3)  67.033130s HelpAndFeedback/testSubmittingFeedback

I'd anticipated all tests within the target to be distributed equally among the simulators.

Command

bluepill -a ./build/Unit\ Test-iphonesimulator/GuestCenter.app -s ./GuestCenter.xcodeproj/xcshareddata/xcschemes/Acceptance\ Tests1.xcscheme -o ./output/ -d "iPad Pro (9.7-inch)"

Of note, we have our UI tests split between 4 targets currently to minimize their run times. In the command above, I'm trying to only run the Acceptance Tests1 tests, however the command is loading all tests in the product.

Am I correct in understanding what I'm seeing? If so, suggestions on what I'm doing wrong?

I'm trying to debug the cause of this but I'm unable to figure it out.

There should be a verbose logging mode which prints all the NSError descriptions so we can figure out such issues.

After we merged the change "Fix hangs (#49)", we started to observed the errors like
20170208.154417 [ ERROR ] (12) Application crashed before tests started!
20170208.154417 [ INFO ] (12) [11] Delete Simulator,
but the test cases were actually running well without crash. It happens in the CI environment and also fail the bluepill integration test cases.

To reproduce with the test case, check out the reuse-sim branch, run test case BluepillRunnerTests/BPIntegrationTests/testOneBPInstance
To see the error messages, you need to disable quite mode first in the line 47 of BPIntegrationTests.m:
//self.config.quiet = YES;

It is taking about 2.5 minutes for data to be saved in Core Data while running a test using Bluepill.
Subsequent saves are quick until the simulator is restarted and the initial Core Data save is slow again.

We import JSON data over the web, convert it to objects and save the objects in core data at intervals as we loop over the JSON. The save is done on a background thread.

Using Xcode to run the same test all the Core Data saves are about a second or sub second to complete as expected.

This change: 93b7f31
broke command line parsing. It needs to be fixed.

In current setup, if test hangs, we don't know what is the cause.

It would be much nicer if we provide a process sample of the hanging app before killing it and record it in the report. Like the output of Sample command.

When the test-bundle-path argument is passed, the program immediately exits. When debugging, in BPConfiguration.m, test-bundle-path program is only 'BP_SLAVE' instead of 'BP_MASTER | BP_SLAVE' so in handleOpt() it exits because !(BPOptions[i].program & self.program) is true for test-bundle-path. I can make a PR for adding BP_MASTER but don't know what those are for so not sure if its intentional.

When debugging failures, it'd be useful if bluepill printed the UUID of the simulator when it creates a new one. E.g.

[ FAILED ] (1) Completed: [1] Create Simulator

Doesn't help debugging failures. If it printed the UUID we could go dig for crash logs.

We were trying to address issue #12 - [self.device terminateApplicationWithID:self.hostBundleId error:&error]; in SimulatorMonitor.m doesn't work in iOS 10.2 runtime. in PR #19 , however, it is still flaky, seeing hangings sometimes.

Initial inspection is that - the SimulatorRunner may be nil. Assigned to @vargon to take a look

I had a test that failed in a set where all others succeeded, and when it retried it crashed the app, but then Bluepill marked it as success. Here's the relevant portion of the output:

20170213.134303 [  INFO  ] (2) Completed: [2] Install Application
20170213.134303 [  INFO  ] (2) [2] Launch Application
20170213.134311 [  INFO  ] (2) Completed: [2] Launch Application
20170213.134312 [  INFO  ] (2) All Tests started.
20170213.134320 [ CRASH  ] (2) AnnotationTests/testCreateNote crashed app.
20170213.134320 [  ERROR ] (2) Will kill the process with appPID: 7676
20170213.134320 [  INFO  ] (2) [2] Delete Simulator
20170213.134332 [  INFO  ] (2) Completed: [2] Delete Simulator
20170213.134332 [  INFO  ] (2) Exit Status: BPExitStatusTestsAllPassed

Nothing else was retried after and Bluepill finished with success.

After providing a valid path to my app, an output path, and a path to a scheme containing XCTest UI tests, the output looks like this:

XXX@XXXX-mbp15-01 ~/develop/XXXX (develop)*$ ./bluepill -a TestBuild/Build/Products/UITest-iphonesimulator/XXXX.app -o ~/Desktop/ -s XXXX.xcodeproj/xcshareddata/xcschemes/UITests.xcscheme
[  INFO  ] This is Bluepill
[  ERROR ] Packing failed

After some light digging I can see where the error comes from:
https://github.com/linkedin/bluepill/blob/9d704360cdb87ecd9f4fc59fa8c1b1690c8a6d10/Bluepill-runner/Bluepill-runner/BPRunner.m#L140

but it's not obvious why it's failing. There's a few switches listed in ./bluepill -help that are "required" but have no default value, and it's not obvious what they mean. For example:

-N<required_arg> | --no-split<required_arg>
		A list of NO split test bundles

Hey guys, great project! I'm testing bluepill with a Titanium-based project, created a config.json like the following:

{ 
   "app": "./mensa-app/build/Products/Debug-iphonesimulator/Mensa.app",
   "scheme-path": "./mensa-app/Mensa.xcodeproj/xcshareddata/xcschemes/Mensa.xcscheme",
   "output-dir": "./mensa-app/build"
}

but getting the error The folder “Plugins” doesn’t exist.. Any idea what that could mean? Thx!

[reportXML nodesForXPath:@".//testsuites" error:nil] returns nil for the output, so compareElement early returns.

I found this because I was wondering how this actual resulting XML file:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Selected tests" tests="0" failures="0" errors="0" time="0.002000">
  <testsuite tests="0" failures="0" errors="0" time="0.000000" timestamp="2017-02-13T16:05:39GMT-07:00" name="BPSampleAppCrashingTests.xctest">
  </testsuite>
</testsuites>

was being considered equal to the expected one:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="All tests" tests="3" failures="1" errors="1" time="0.002000">
    <testsuite tests="2" failures="1" errors="1" time="0.041532" timestamp="2016-11-07T16:54:44GMT-08:00" name="BPSampleAppCrashingTests.xctest">
        <testsuite tests="2" failures="1" errors="1" time="0.020770" timestamp="2016-11-07T16:54:44GMT-08:00" name="BPSampleAppCrashingTests">
            <testcase classname="BPSampleAppCrashingTests" name="testAppCrash0" time="0.000000">
            </testcase>
            <testcase classname="BPSampleAppCrashingTests" name="testAppCrash1" time="0.000000">
                <failure type="Failure" message="App Crashed">
                    Unknown File:0
                </failure>
                <error type="Error"/>
            </testcase>
        </testsuite>
    </testsuite>
    <testsuite tests="1" failures="0" errors="0" time="0.001000" timestamp="2016-11-07T16:55:13GMT-08:00" name="BPSampleAppCrashingTests.xctest">
        <testsuite tests="1" failures="0" errors="0" time="0.000000" timestamp="2016-11-07T16:55:13GMT-08:00" name="BPSampleAppCrashingTests">
            <testcase classname="BPSampleAppCrashingTests" name="testAppCrash2" time="0.000000">
            </testcase>
        </testsuite>
    </testsuite>
</testsuites>

Hi
let say I have test targets targetA, TargetB and want to run like below. Is that possible if yes how?
targetA ----iPhone 6 version 10.1
targetB ----iPhone 6 version 10.1

Title says it all. If you send SIGINT to bluepill it'll abort the tests, but it'll report All Tests Passed which is not true.

If you do a standard Cmd+B build and point bluepill to the .app, you get this error:

ERROR: The folder “Plugins” doesn’t exist.

Would be good to print a helpful message along the lines of "Please build the test target of your app (Cmd + Shift + U)"

The code in the main loop and the completion block of task termination in the run() method of BPRunner.m are not not synchronized. Because the completion block will be called in other threads than the main thread, that causes race condition for variables such as launchedTasks, taskList.

I will fix it together will implementing the feature of reusing simulators. just create an issue to track it.

Upload a recipe fo bluepill to brew so that people don't have to build bluepill through source code.

I Run single test using xcodebuild like this

xcodebuild test -workspace xxxWorkspace.xcworkspace/ -scheme "xxx.com Debug-UITests" -destination "platform=iOS Simulator,name=iPhone 6s,OS=latest" -only-testing:zzzUITests/SmokeTest

Can I run the similar test using bluepill?

Thanks

Running this:

bluepill -a XXXX.app/ -s XXXX.xcscheme -o output/

crashes and returns this:

bluepill[78419:3224648] *** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSSingleObjectArrayI objectAtIndex:]: index 1 beyond bounds [0 .. 0]'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff7d8390db __exceptionPreprocess + 171
	1   libobjc.A.dylib                     0x00007fff924cba2a objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff7d8a3b4f -[__NSSingleObjectArrayI objectAtIndex:] + 111
	3   bluepill                            0x000000010bb80448 +[BPXCTestFile BPXCTestFileFromExecutable:withError:] + 872
	4   bluepill                            0x000000010bb7f746 +[BPApp BPAppWithAppBundlePath:withExtraTestBundles:withError:] + 907
	5   bluepill                            0x000000010bb78b3c main + 739
	6   libdyld.dylib                       0x00007fff92dad255 start + 1
	7   ???                                 0x0000000000000007 0x0 + 7
)
libc++abi.dylib: terminating with uncaught exception of type NSException
Abort trap: 6

when i run ./bluepill -a ./Sample.app -s ./SampleAppTestScheme.xcscheme -o ./output/
it replay me -bash: ./bluepill: is a directory

Hello,

I'm trying to run bluepill for iPhone SE on 9.3. I've done xcodebuild with the proper destination:

-destination 'platform=iOS Simulator,name=iPhone SE,OS=9.3'

Here's my config.json:

{
   "app": "../build/to/App.app",
   "scheme-path": "../build/to/test.xcscheme",
   "output-dir": "../output",
   "num-sims": 3,
   "plain-output": true,
   "failure-tolerance": 1,
   "device": "iPhone SE",
   "runtime": 9.3,
   "headless": false,
   "only-retry-failed": true
}

But I'm getting the following error:

2017-02-22 15:59:02.301 bp[31126:28986974] -[__NSCFNumber length]: unrecognized selector sent to instance 0x7f94b1c23c70
2017-02-22 15:59:02.301 bp[31126:28986974] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFNumber length]: unrecognized selector sent to instance 0x7f94b1c23c70'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff92c514f2 __exceptionPreprocess + 178
	1   libobjc.A.dylib                     0x00007fff99190f7e objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff92cbb1ad -[NSObject(NSObject) doesNotRecognizeSelector:] + 205
	3   CoreFoundation                      0x00007fff92bc1571 ___forwarding___ + 1009
	4   CoreFoundation                      0x00007fff92bc10f8 _CF_forwarding_prep_0 + 120
	5   CoreFoundation                      0x00007fff92b5a241 isEqualToString + 65
	6   CoreFoundation                      0x00007fff92ba7228 -[NSTaggedPointerString isEqualToString:] + 24
	7   bp                                  0x0000000108312ca1 -[BPConfiguration validateConfigWithError:] + 4208
	8   bp                                  0x000000010832c573 main + 632
	9   libdyld.dylib                       0x00007fff935065ad start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException

However, when I run tests manually via xcode or xcodebuild command, I do not get those errors. Am I doing something wrong?

Listing tests by default just lists the test bundles, which I think is kind of useless. Had I not looked at the code and noticed that values greater than 1 indicate verbose mode which lists all the tests in the bundle, I would have tried to find a different solution.

I think listing all of the tests in the bundle should be default, and values greater than 1 indicate "quiet" mode if the other functionality is desired.

when I run bluepill, there was an Exception,here is logs...
CCTest:bluepill-v0.2.1 chen$ ./bluepill -c config.json
[ INFO ] This is Bluepill v0.1.0-33-g0d56983
[ INFO ] Running with 4 simulators.
[ INFO ] Packed tests into 6 bundles
[ INFO ] Started Simulator 1 (PID 11521).
[ INFO ] 1 Simulator still running. [1]
[ INFO ] Using 366 of 1064 processes.
[ INFO ] Started Simulator 2 (PID 11526).
2017-02-07 11:07:21.087 bp[11526:120290] -[__NSCFNumber length]: unrecognized selector sent to instance 0x7fbfc870d6f0
2017-02-07 11:07:21.087 bp[11521:120251] -[__NSCFNumber length]: unrecognized selector sent to instance 0x7f8b79d03380
2017-02-07 11:07:21.160 bp[11521:120251] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFNumber length]: unrecognized selector sent to instance 0x7f8b79d03380'
*** First throw call stack:
(
0 CoreFoundation 0x00007fffaddfb48b __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fffc255dcad objc_exception_throw + 48
2 CoreFoundation 0x00007fffade7cc94 -[NSObject(NSObject) doesNotRecognizeSelector:] + 132
3 CoreFoundation 0x00007fffadd6db55 forwarding + 1061
4 CoreFoundation 0x00007fffadd6d6a8 _CF_forwarding_prep_0 + 120
5 CoreFoundation 0x00007fffadd06c8d isEqualToString + 77
6 CoreFoundation 0x00007fffadd52cd8 -[NSTaggedPointerString isEqualToString:] + 24
7 bp 0x00000001089af491 -[BPConfiguration validateConfigWithError:] + 4208
8 bp 0x00000001089c8897 main + 632
9 libdyld.dylib 0x00007fffc2e3c255 start + 1
10 ??? 0x0000000000000003 0x0 + 3
)
libc++abi.dylib: terminating with uncaught exception of type NSException
What should I do, thanks!

I was testing it locally, and the only place it's used is in testEnvironmentWithConfiguration which is only called from bluepill's unit tests.

Bluepill handles the case when a test crashes. However, if the app itself crashes on launch, the console gets a useless message like this:

[ CRASH  ] (1) (null)/(null) crashed app.

And nothing useful is reported in the output report. Bluepill should have a special way of logging and reporting an app crashing on launch.

When I run bluepill on my (objc) app, I get

2017-02-18 00:30:04.236 bluepill[58215:969741] *** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSArrayM objectAtIndex:]: index 2 beyond bounds [0 .. 1]'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff9c443e7b __exceptionPreprocess + 171
	1   libobjc.A.dylib                     0x00007fffb102dcad objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff9c35d1dc -[__NSArrayM objectAtIndex:] + 204
	3   bluepill                            0x000000010e4e0f9a +[BPXCTestFile BPXCTestFileFromExecutable:withError:] + 1138
	4   bluepill                            0x000000010e4e0285 +[BPApp BPAppWithAppBundlePath:onlyTestingBundlePath:withExtraTestBundles:withError:] + 1589
	5   bluepill                            0x000000010e4d92f5 main + 786
	6   libdyld.dylib                       0x00007fffb1911255 start + 1
	7   ???                                 0x0000000000000005 0x0 + 5
)

I confirmed this is problematic line.

I am using appium for iOS automation on real device, Now I want to execute my test simultaneously on real devices. How can I use bluepill in my project. If bluepill doesn't work then what can I use ?

I spent some time puzzled about this error:

$ bluepill -c config.json
[  INFO  ] This is Bluepill v0.1.0-21-g95fcfd8
[  INFO  ] Running with 4 simulators.
[  INFO  ] Packed tests into 6 bundles
[  INFO  ] Started Simulator 1 (PID 53404).
bp: invalid configuration
	iPhone 6 is not a valid device type.
Use `xcrun simctl list devicetypes` for a list of valid devices.

It turns out that I had the wrong path in my config.json file. Bluepill should at least check that the Xcode path give is valid and not fall through to a misleading error.

bluepill was the project that mentioned in below article?
https://engineering.linkedin.com/blog/2016/04/3x3--ios-build-speed-and-stability

Currently bluepill recreate the simulator for each test bundle. We could save a lot of time if reusing the same simulator for more than one test bundles.

Proposals:

1. Add CLI option "not-delete-simulator" to bp command, so it won't delete the simulator when test cases finish normally, and return the device id to the parent bluepill process. By default bp command Always delete simulator.

2. Add CLI option "use-simulator" to bp command. The bluepill process pass the device id to bp command so bp can reuse the specified simulator without recreating a new one.

How to change language during test? For instance, launch five tests simultaneously for five different languages (Japanese, French, German, Spanish and Chinese)

Thanks

set up travis for bluepill to get build information

Hey folks,

Really excited to give bluepill a whirl. I cloned the repository and ran build.sh and it's bailing with 11 test failures:

	-[BPCLITests testNoSchemeinCLI]
	-[BluepillTests testReportWithAppCrashingTestsSet]
	-[BluepillTests testReportWithAppHangingTestsSet]
	-[BluepillTests testReportWithAppHangingTestsShouldReturnFailure]
	-[BluepillTests testReportWithFailingTestsSet]
	-[BluepillTests testReportWithFatalErrorTestsSet]
	-[BluepillTests testRetryWithFailingTestsSet]
	-[BluepillTests testRunningOnlyCertainTestcases]
	-[BluepillTests testRunWithConfigurationFile]
	-[BluepillTests testRunWithPassingTestsSet]
	-[BluepillTests testRunWithRelativePathInConfigurationFile]

Wondering if this a bug, configuration problem, or something else? Is there a better forum for troubleshooting or is this an appropriate place? Adding an installation or getting started section to README would be great and I'd be happy to contribute.

Instead of traversing file system for the test targets, parse .xctestrun file to fetch them.