Interaction should be better about go-ios HOT 9 CLOSED

I've spent the last 3 years working on and creating essentially what is described. The end result is a product I've created called ControlFloor. It is mainly sold to companies in need of a device farm.

The technology I created is what drives the iOS interactivity of LambdaTest. If you wish to purchase an existing easily available service for a handful of devices, I would recommend them as they are a partner of my company.

I am considering doing as is proposed above and creating a simple web UI for go-ios and giving that away freely. I don't have an issue with that and I have my own reasons for doing so...

Creating something like srccpy for iOS is extremely difficult and requires a lot of moving parts, due to the many restrictions Apple has placed on devices.

For a brief overview of what it takes, you should watch my YouTube video on it from the Appium conference: https://www.youtube.com/watch?v=5aOCpBb4BNI

There are two main aspects to creating something like srccpy:

1. Getting a video feed of the screen
2. Simulating touch events via mouse

There are a handful of different ways to get a video feed:

1. Via AirPlay; This is Apple proprietary and encrypted. Generally no go here, although there are various open source things that do it ( illegally might I add )

2. Via quicktime_video_hack. This uses a mostly undocumented Apple USB tweak to get the h264 stream from the device similar to what goes out to USB HDMI dongles. It works, but you can't control the video resolution, framerate, or quality level. It also requires handling h264 well, which is complex in and of itself. There are MANY open source projects in existence now that grab frames this way, but the problem there is scalability. Encoding h264 into frames has a cost, and you can only hardware render so many h264 streams on a single machine via hardware decoding.

3. Via WebDriverAgent. This uses the Xctest screen capture function. It is costly CPU wise and low frame rate. Many commercial providers of iOS device farms are using this method, and it is poor quality.

4. Via ControlFloorAgent ( closed source; I wrote it ) This is similar to the WDA method, also using Xctest screen capture, but is at least done more optimally than WDA and suffers less issues. It works, but still has high CPU use. It is reliable though...

5. Via an "upload broadcast extension". This is the official Apple method. You get handed an essentially raw screen buffer for each frame and have to encode it yourself. There are various open source versions of this, and most of them are terrible. LambdaTest notably has done this relatively well encoding to h264, and this is what drives LambdaTest iOS video. Their app, is, of course, closed source and part of their service. I released one of these as open source myself; see https://github.com/nanoscopic/ios_video_app It is designed to work as part of my paid ControlFloor product. I haven't updated the open source version in a long time, as the updates have gone closed source. Anyone who likes is free to continue developing the open source version ( as long as they abide by the license of course )

6. Using a USB HDMI dongle, and HDMI capture/streaming equipment. This works, but suffers additional lag due to having to capture the HDMI stream. It also requires buying expensive dongles.

7. By building some reverse engineered custom designed board to simulate a USB HDMI dongle and capture that video stream. I know that various companies in China know how to do this, because that knowledge is required in order to build a knock-off HDMI dongle for iPhones. ( one that doesn't use Xctest screen capture at least... ) Unfortunately none of them are sharing the information. Also, all such things are likely illegal due to what is required to build them...

8. By jailbreaking your device and using Veency. ( supposing it even works on modern iOS; it hasn't been updated in forever )

9. By injecting hooks into your app to make the app itself capable of rendering itself out. This is how EarlGrey works. It is an interesting solution but I haven't seen anyone make an easy to use streaming ability that can be injected into any app and work painlessly. It is theoretically possible. This can be done on non-jailbroken devices through injection during app re-signing.

There are a handful of ways to simulate touchscreen interaction via mouse:

1. Via WebDriverAgent, which in turn uses the undocumented Apple API call SynthesizeEvent. This works pretty well. The WDA incarnation of this has excessive and pointless delay added due to the terrible way WDA is written... I wrote a cleaner and more direct version using SynthesizeEvent and it works pretty well. The last public version of ControlFloorAgent I released is here: https://github.com/nanoscopic/ControlFloorAgent The current version is a complete rewrite from the ground up, using no code from WebDriverAgent, and is closed source.

2. Via a synthetic bluetooth mouse device. This is relative motion only. Works well. A bunch of people have done this, but it suffers from not being absolute, so you have to have good video feedback to see where the mouse cursor is to understand where you are clicking, so it isn't great. Many commercial vendors use this as their solution.

3. Via a synthetic bluetooth touchpad. This supports absolute clicking, but is only possible with iPad, not iPhone. I haven't seen any open source code that does this.

4. Via a synthetic usb mouse. Also relative motion only. Essentially the same as bluetooth, but you need to plug it in. This has more problems.

5. Via unknown mechanism through Apple's newly released "drag your mouse cursor onto an iPad" feature. It is iPad only. No one has reverse engineered how this works yet to my knowledge.

6. Via app injection ( similar to the video injection I mention above )

Best of luck. You should buy my product once I begin selling individual licenses to users ( or giving away to individual users... I haven't decided yet... ) In the short term pay for LambdaTest, as they are a partner of my company and partly using the research I have done over the years on this.

There are many more complexities than what I've mentioned above. Video is not even as easy as what I'm making it sound to be, because Apple restricts display of video alerts on many of the video methods.

from go-ios.

I don't understand what you are trying to say.

from go-ios.

I don't understand what you are trying to say.

I mean, go IOS has achieved many functions, but we can only use the go IOS command in the CMD window of windows or the terminal window of MacOS to do some things, such as installing an IPA software... I think if you make all the functions of go IOS into an EXE or DMG, everyone can get a function interface through installation, and everyone can use all the functions of go IOS on this function interface. For many people, using the command line is not a convenient thing.

from go-ios.

Oh now I get what you mean. You are talking about an actual GUI or UI. I have thought about it but then never found the time to actually build it. What I would do is add a REST API to go-ios and then build a web application around it. So you could make it work as a PWA f.ex.

Or a Visual Studio Code plugin is also something I considered.

Do you have a use-case yourself for the tool? I am always really curious to learn if people are actually using this and for what :-D As you can see in the Readme, the main goal for the project was to enable people to use it.

from go-ios.

Oh now I get what you mean. You are talking about an actual GUI or UI. I have thought about it but then never found the time to actually build it. What I would do is add a REST API to go-ios and then build a web application around it. So you could make it work as a PWA f.ex.

Or a Visual Studio Code plugin is also something I considered.

Do you have a use-case yourself for the tool? I am always really curious to learn if people are actually using this and for what :-D As you can see in the Readme, the main goal for the project was to enable people to use it.

I found an instance of GitHub, which I have successfully built locally and can run. Unfortunately, it only has Windows version, not MacOS version, so I think many people's projects are only half done, and do not really cover the user needs of a variety of systems. But I think go IOS has a greater influence, and many similar tools are actually based on go IOS, so I think if you build this general tool for windows and MacOS, it should help many people in need in the world, because no one in the whole world makes the interface of this tool.

I think it would be better if we could use go IOS to make a tool similar to scrcpy, make it suitable for windows and MacOS, and use USB cable to display the iPhone screen on the screen suitable for windows and MacOS!

I've tried your QuickTime_ video_ Hack, but I think QuickTime_ video_ Hack would be better if it could be packaged into an installation package. Unfortunately, it can't run on Windows system, so I always think that only half of the good tools are not perfect. Because I use windows and MacOS at the same time every day, I can't find a complete tool.

Instance:

(1)https://github.com/yhforthefreedom/iOSToolBox
(2)https://github.com/Genymobile/scrcpy

from go-ios.

Can't say anything about streaming, I spent some time trying to create an mjpeg stream using the screenshot functionality but I couldn't wrap my head around the proper synchronization and I got mediocre results :D I am willing to spend some effort on a web UI (seems more practical) for go-ios though. I don't see a use case for some of the available functionality but it can be useful for others. It will need a REST API as Daniel mentioned so it might be best to do it in parallel. @danielpaulus let me know if you are interested in scaling go-ios in such a way, I would love to contribute

from go-ios.

What @nanoscopic says makes perfect sense. I have no intentions to compete with ControlFloor by building my own free version of it. The goal of go-iOS was to enable people like David to build device farm software and build businesses with a solid foundation either in code or in knowledge. So I wanted to allow people to use a solid, prod ready tool to build stuff for iOS devices.
I do have an interest though to build cool stuff on top of go-iOS myself. That is why I was curious what the use case would be. If it's building your own device farm, then you should buy a license from David :-) Because this stuff is hard and maintaining it is beyond what I can do in my free time.

I am still open to building a fun UI that enables people to use go-iOS more comfortably though.

from go-ios.

You are all elites! Thank you very much!

from go-ios.

I moved the discussion to here: #158
I need to think more about use cases and then we can move forward with it.

from go-ios.