iOS support about esp32_mouse_keyboard HOT 51 CLOSED

It works!
Fantastic fix, nicely done.

from esp32_mouse_keyboard.

@chegewara @mattcrump

I added the access permissions to all characteristics in the BLEHID ctor.
It works like before for Android.

commit: efed298

@mattcrump Could you please test this, maybe this helps :-)

from esp32_mouse_keyboard.

@JuanPotato
Hi, thanks for testing. I cannot see a good error description here, please send me a verbose log.

Greetings

from esp32_mouse_keyboard.

I'm afraid there isn't anything related to the error beyond that one line https://pastebin.com/HdqEAbip

from esp32_mouse_keyboard.

Hm, I cannot find anything as well :-(.
I can have a further look this week, when I have access to an iOS device.

Most of this project are originally from espressif:
espressif/esp-idf#782

Maybe we can reopen this issue if we can find the problem.

from esp32_mouse_keyboard.

Hi @benjaminaigner,

Seems like I got the same issue too. the project is great on android but it just failed in iOS
It connected the ESP32 without requesting for any code and interesting enough it can read the battery service (50%), however, it cannot read any other service (unlike the behavior in android after connected and verified in Android.

same console log I get as @JuanPotato , also I am quite new to esp-idf, may I know how can I provide the verbose log?

P.S. The service information read from the app nRF Connect in both android and iPhone.

Thanks in advance!

from esp32_mouse_keyboard.

@cheuklam

According to https://esp-idf.readthedocs.io/en/latest/api-reference/system/log.html
either you set logging to verbose by make menuconfig or at runtime with a esp_log_level_set command.

Unfortunately, I'm really inexperienced with iOS, so it would be nice if somebody could provide iOS logs :-).
I tried this demo also with an iPad, it seems to be connected, but iOS is not opening the HID service...

from esp32_mouse_keyboard.

Maybe i can help. Try to add code like this in onConnected() callback for each input report:

BLE2902* desc = (BLE2902*) input->getDescriptorByUUID(BLEUUID((uint16_t)0x2902));
      desc->setNotifications(true);

input is each of those:
https://github.com/asterics/esp32_mouse_keyboard/blob/newBLE/components/nkolban_BLE/HID_kbdmousejoystick.cpp#L64-L69

from esp32_mouse_keyboard.

@chegewara
Thank you very much, I'm sad that I don't have an ESP32 at hand right now to test this :-).
Done via ce8d7fe

from esp32_mouse_keyboard.

Im not saying this will work with iOS, but this code fixed my issue with reconnecting to windows 10.

from esp32_mouse_keyboard.

@benjaminaigner @chegewara I am new here. Just FYI when trying out branch newBLE I get the following error(I realize the branch is in progress, just reporting here what I saw):

make[1]: *** No rule to make target 'C:/msys32/home/USERNAME/esp/esp32_mouse_keyboard/main/ble_hidd_deeded by 'ble_hidd_demo_main.o'.  Stop.
make: *** [C:/msys32/home/USERNAME/esp/esp-idf/make/project.mk:467: component-main-build] Error 2

I'll be digging into this IOS issue over the next few days, but can't promise anything. Has anyone dug into this at all?

I have had "mild" success with another bluetooth chip by sending a security request. Anyone know if this case ever gets hit?:

case ESP_GAP_BLE_SEC_REQ_EVT:
        /* send the positive(true) security response to the peer device to accept the security request.
        If not accept the security request, should sent the security response with negative(false) accept value*/
        esp_ble_gap_security_rsp(param->ble_security.ble_req.bd_addr, true);
        ESP_LOGI(GATTS_TAG,"ESP_GAP_BLE_SEC_REQ_EVT");
        break;

I'll start poking around. Thanks!

from esp32_mouse_keyboard.

hi @dalinarkholin
Please try to delete all files in folder build/main and compile it again.

As far as i know this should have work ESP_GAP_BLE_SEC_REQ_EVT
BLESecurity class is also written by me. If there are any issue please report.

from esp32_mouse_keyboard.

@chegewara that resolved that issue. Now I am getting:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
ets_main.c 371
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5676
load:0x40078000,len:0
load:0x40078000,len:14324
entry 0x40078620
I (29) boot: ESP-IDF v3.1-dev-957-g139a8a27 2nd stage bootloader
I (29) boot: compile time 16:53:30
I (29) boot: Enabling RNG early entropy source...
I (35) boot: SPI Speed      : 40MHz
I (39) boot: SPI Mode       : DIO
I (43) boot: SPI Flash Size : 4MB
I (47) boot: Partition Table:
I (50) boot: ## Label            Usage          Type ST Offset   Length
I (58) boot:  0 nvs              WiFi data        01 02 00009000 00004000
I (65) boot:  1 otadata          OTA data         01 00 0000d000 00002000
I (73) boot:  2 phy_init         RF data          01 01 0000f000 00001000
I (80) boot:  3 factory          factory app      00 00 00010000 00200000
I (88) boot: End of partition table

Let me know if you need anything else. Is the plan to move forward with newBLE? If so I can work off of it.

from esp32_mouse_keyboard.

I am only guest here for "consultation".

From my point of view your code works fine now. Im suspecting you have log level set to info in menuconfig. Try to find esp32 with some ble scanner like nRF connect, LightBlue etc.

PS i think newBLE is working pretty well thanks to @benjaminaigner

from esp32_mouse_keyboard.

IOS doesn't see "FABI"(default device name I thing) anymore. Android sees it but gets a "Unable to communicate with FABI" error.

from esp32_mouse_keyboard.

Please change log level to debug or verbose and provide more logs.

from esp32_mouse_keyboard.

Just an update. Not sure Android ever actual was seeing the esp32 "FABI" on newBLE branch. I have a older android phone. Galaxy S3. Here are verbose logs:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
ets_main.c 371
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:7928
load:0x40078000,len:0
ho 12 tail 0 room 4
load:0x40078000,len:15684
entry 0x400786dc
I (30) boot: ESP-IDF v3.1-dev-957-g139a8a27 2nd stage bootloader
I (30) boot: compile time 17:23:59
I (30) boot: Enabling RNG early entropy source...
D (36) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (43) boot: magic e9
D (45) boot: segments 04
D (48) boot: spi_mode 02
D (51) boot: spi_speed 00
D (53) boot: spi_size 02
I (56) boot: SPI Speed      : 40MHz
I (60) boot: SPI Mode       : DIO
I (64) boot: SPI Flash Size : 4MB
D (68) bootloader_flash: mmu set paddr=00000000 count=1
D (73) boot: mapped partition table 0x8000 at 0x3f408000
D (79) flash_parts: partition table verified, 5 entries
I (84) boot: Partition Table:
I (88) boot: ## Label            Usage          Type ST Offset   Length
D (95) boot: load partition table entry 0x3f408000
D (100) boot: type=1 subtype=2
I (103) boot:  0 nvs              WiFi data        01 02 00009000 00004000
D (111) boot: load partition table entry 0x3f408020
D (115) boot: type=1 subtype=0
I (119) boot:  1 otadata          OTA data         01 00 0000d000 00002000
D (126) boot: load partition table entry 0x3f408040
D (131) boot: type=1 subtype=1
I (134) boot:  2 phy_init         RF data          01 01 0000f000 00001000
D (142) boot: load partition table entry 0x3f408060
D (147) boot: type=0 subtype=0
I (150) boot:  3 factory          factory app      00 00 00010000 00200000
I (157) boot: End of partition table
D (162) boot: OTA data offset 0xd000
D (165) bootloader_flash: mmu set paddr=00000000 count=1
D (171) boot: OTA sequence values A 0xffffffff B 0x00000001

from esp32_mouse_keyboard.

Ok, i think the issue is with partition table and otadata. Try make erase_flash and flash one more time.

from esp32_mouse_keyboard.

Yeah that fixed it thanks! Checking IOS now...will update:

UPDATE:

No errors on connection for IOS or android. But the example is different so not 100% sure things are totally working. I see the mouse events firing for both, but those don't register visibly on IOS. I see it on android though. I'll write something and get back, unless someone can quick tell me how to test. I have to run for now though. Thanks @chegewara

from esp32_mouse_keyboard.

Thanks @chegewara for supporting @dalinarkholin .

If you need help, just write here. I'm really not experienced with iOS/macOS and I have just one device for testing (and only for a short time), so it would be really helpful if we can figure out if this example is working well.

from esp32_mouse_keyboard.

Hi guys, just got a chance to try the branch, as I just got an iphone with the module both at the same time.
Seems now it has correct service scanned (0x180A) and there is no error for iOS when connecting. However, when I clicked Q, it is not sending out 'y'. The verbose output like raining but there is no error...... what should I try or test next?

from esp32_mouse_keyboard.

@cheuklam

THX for testing. I think you worked with the old version of the branch, where no keyboard input was used.
Please update to current commit & try again (would be very nice :-) ).

from esp32_mouse_keyboard.

@chegewara @benjaminaigner I just tested with most recent commit to branch newBLE. I tried sending "y" via q.

I (171569) HAL_BLE: Client connected !
I (171779) BLEDevice: ESP_GATTS_MTU_EVT, MTU 517
I (202239) CONSOLE_UART: received q: sending key y for test purposes
I (202239) HAL_BLE: Keyboard received: 2/28

I Tried both the messenger App and the Notes app but I did not see "y" show up on either screen. The cursor was down and blinking. What logs can I pull to help resolve this?

I did verify this is still working with Android.

Let me know! Thanks

from esp32_mouse_keyboard.

@benjaminaigner @chegewara Any ideas?

I just picked up a ble sniffer. I want to check what is being sent and if something is out of place, but documentation on this stuff is so limited. Not sure i'll have enough of a reference even be able to tell. If you have any thoughts as to what could be wrong, I can dive into troubleshooting them. Thanks!

from esp32_mouse_keyboard.

Its really hard to say whats its wrong. Thats why i dont like iPhones, ts so freaking hermetic in software development matters. With android you want to write something you just seat and write it on linux or windows or even on macOS i think, with iPhone you need special dev accounts, hardware and even simplest ble app does not want to work on iPhone because it requires some very strict settings.

You can try to contact https://github.com/S-March, hes doing very nice BLE iPhone project and maybe he can shed some light. His project

from esp32_mouse_keyboard.

@dalinarkholin

It's the same here, I'm mostly a noob regarding iOS.

Because Apple doesn't allow mice on iOS, maybe you try to compile this app without the mouse & joystick profile:

Is there any way to get access to log files on iOS?

from esp32_mouse_keyboard.

@chegewara Thanks I reached out.

@benjaminaigner Thanks! I'll give that a go tonight. Will update with result.

from esp32_mouse_keyboard.

I'm looking forward for somebody to have access to instructions for Apple debugging, even these are locked:
https://developer.apple.com/bug-reporting/profiles-and-logs/

Accessory guidelines don't provide information too...

@dalinarkholin Hopefully this one works :-)

from esp32_mouse_keyboard.

@benjaminaigner, I didn't even know about that 👍. I have an account and will pull logs.

from esp32_mouse_keyboard.

@dalinarkholin, just got your email. I wish I had better insights to this, but unfortunately I had to dig through tons of documentation for the ANCS stuff (wouldn't really call myself an expert on anything else). One place that may be worth checking is looking into example code offered by nRF or Cypress for their bluetooth stacks?

I like the idea of turning off the mouse since that may be causing issues.

from esp32_mouse_keyboard.

I'm having similar issues with an Apple TV. The demo pairs, but no events appear in the ESP32 logs which leads the app to think its not connected. Forcing a connection by setting sec_conn=true also doesn't work.

If I pair with Win10 I get the events and a request for a PIN.

``
E (859) FABI/FLIPMOUSE: hidd_event_callback(), ESP_BAT_EVENT_REG

E (859) FABI/FLIPMOUSE: hidd_event_callback(), ESP_HIDD_EVENT_REG_FINISH

I (869) HID_DEVICE: esp_hidd_prf_cb_hdl(), start added the hid service to the stack database. incl_handle = 40

E (879) FABI/FLIPMOUSE: gap_event_handler(), ESP_GAP_BLE_ADV_DATA_SET_COMPLETE_EVT

I (889) HID_DEVICE: hid svc handle = 2d

E (889) FABI/FLIPMOUSE: gap_event_handler(), ESP_GAP_BLE_ADV_START_COMPLETE_EVT

V (899) intr_alloc: esp_intr_alloc_intrstatus (cpu 1): checking args

V (899) intr_alloc: esp_intr_alloc_intrstatus (cpu 1): Args okay. Resulting flags 0xE

D (909) intr_alloc: Connected src 34 to int 3 (cpu 1)

Not connected, ignoring 'a'
``

from esp32_mouse_keyboard.

Further to my message above, attached is a diff on logs for pairing with WIN10 and ATV4K.
The left hand side is WIN10, the right ATV.
Everything is as you would expect until around line 486. At this point, WIN10 triggers the PIN request.
However, the ATV causes what looks to be some kind of failover.
I traced the debug message to gatt_sr.c: (sorry about the formatting, no idea why its removing line breaks)
if (p_list->p_last_primary == p_srv && p_list->p_last_primary == p_list->p_last) { GATT_TRACE_DEBUG("Use 0xFFFF for the last primary attribute"); UINT16_TO_STREAM(p, 0xFFFF); /* see GATT ERRATA 4065, 4063, ATT ERRATA 4062 */ } else { UINT16_TO_STREAM(p, p_rcb->e_hdl); }
Anyone have any idea what GATT ERRATA 4065, 4063, ATT ERRATA 4062 are?

win10_vs_atv.pdf

from esp32_mouse_keyboard.

@mattcrump
THX for the logs, maybe we can have some insights based on these logs.

Personally, I've no idea what that GATT errata might be. A quick search didn't bring up any errata document or something similar.

Interestingly, there are no sec_flags set on the ATV device.

Did you use the newBLE branch, just to be sure?

Maybe we get some more insights after digging through some Apple docs (unfortunately I don't have a Apple dev ID)...

from esp32_mouse_keyboard.

Yes I used the newBLE branch.

I have an Apple dev login, what are you after?

from esp32_mouse_keyboard.

I think (but I'm not sure due to lack of access), it is the BLE debugging information, maybe there are useful hints on this kind of problems:

https://developer.apple.com/services-account/download?path=/OS_X/OS_X_Logs/Bluetooth_Logging_Instructions.pdf

(macOS)

https://developer.apple.com/services-account/download?path=/iOS/iOS_Logs/Bluetooth_Logging_Instructions.pdf

(iOS)

from esp32_mouse_keyboard.

More likely its part of bludroid stack not esp-idf:
https://github.com/mozilla-b2g/platform_external_bluetooth_bluedroid/blob/master/stack/gatt/gatt_sr.c#L581

from esp32_mouse_keyboard.

@mattcrump From logs you provided i can see there can be some sort of issue with security during connection to atv. Could you get rid off all security from code and try again?

from esp32_mouse_keyboard.

Here's the logging instructions. I'll try without the security in the morning.

Bluetooth_Logging_Instructions.pdf
Bluetooth_Logging_Instructions(1).pdf
iOSBluetoothLogging.zip

from esp32_mouse_keyboard.

No go on removing mouse. :(

I (768) EXT_UART: external UART processing task started
W (978) BLEService: << Adding a new characteristic with the same UUID as a previ
ous one
I (978) HAL_BLE: Keyboard added @report ID 1, current report Map:
I (988) HAL_BLE: 0x3ffe58f8   05 01 09 06 a1 01 85 01  75 01 95 08 05 07 19 e0
|........u.......|
I (988) HAL_BLE: 0x3ffe5908   29 e7 15 00 25 01 81 02  95 01 75 08 81 03 95 05
|)...%.....u.....|
I (998) HAL_BLE: 0x3ffe5918   75 01 05 08 19 01 29 05  91 02 95 01 75 03 91 03
|u.....).....u...|
I (1008) HAL_BLE: 0x3ffe5928   95 06 75 08 15 00 25 68  05 07 19 00 29 68 81 00
 |..u...%h....)h..|
I (1018) HAL_BLE: 0x3ffe5938   c0
 |.|
I (1028) HAL_BLE: Final report map size: 65 B
I (1068) HAL_BLE: Advertising started!
E (1078) BT_HCI: hci write adv params error 0x30
I (19228) HAL_BLE: Client connected !
I (19428) BLEDevice: ESP_GATTS_MTU_EVT, MTU 517
I (45658) CONSOLE_UART: received q: sending key y for test purposes
I (45658) HAL_BLE: Keyboard received: 2/28

Pulling Logs from IOS device (iphone 7+) here is some information that might be relevant:

in WiFi/bluetooth_status.txt

    Power                : Off
    MAC Address          : None
    Discoverable         : No
    Connectable          : No
    Scanning             : No
    Devices              : 0 (paired=0 cloud=0 connected=0)

summaries/BluetoothAccessories.txt was empty

in diagnostics.txt

...
10:02:42.842 High BT Paired Count          0.118       No        Paired Bluetooth device count (0)
10:02:42.960 High BT Connected Count       0.104       No        Connected Bluetooth device count (0)
10:02:43.065 BT Coex Active                0.097       No        Bluetooth coexistence is not active
...

Other than this there are bunch of references to bluetoothd in the folder crashes_and_spins Here is one of the logs I found. This was in a file called jetsamevent-2018-06-16-blahblah.ips.sync

{
    "rpages" : 355,
    "states" : [
      "daemon"
    ],
    "name" : "bluetoothd",
    "pid" : 86,
    "age" : 1211767694837,
    "fds" : 50,
    "uuid" : "28479029-374a-3b2c-9e8e-51ac4cf04a2f",
    "purgeable" : 16,
    "coalition" : 134,
    "cpuTime" : 222.09581299999999,
    "lifetimeMax" : 811
  },

This doesn't seem right to me unless the IOS app interface is blatantly lying about being connected. Any ideas? Is there anything you guys want me to search for?

Let me know

from esp32_mouse_keyboard.

Maybe its nothing very complicated. Maybe chapter 9 from this pdf can shad some light:
https://developer.apple.com/accessories/Accessory-Design-Guidelines.pdf
In particular 9.13 HID and chapter 10

If i would work with iPhone i would try to change those:

• turn on/off security/bonding etc.
• set MTU to 23, 158,
• change device address from public to RPA,
• change connection parameters (intervalMIN, intervalMAX)
  Except last one, which i dont know where to change it, other settings are pretty easy to test.

from esp32_mouse_keyboard.

In that same document Apple states:

10.10 Pairing
The accessory should not request pairing until an ATT request is rejected using the Insufficient Authentication error code. See the Bluetooth 4.0 specification, Volume 3, Part F, Section 4 for details.

If, for security reasons, the accessory requires a bonded relationship with the Central, the Peripheral should reject the ATT request using the Insufficient Authentication error code, as appropriate. As a result, the Apple product may proceed with the necessary security procedures.

Sounds like an HID device should first reject the insecure connection, which will result in a further offer of a secure connection?

from esp32_mouse_keyboard.

It means that secure connection is not forced on connection, but characteristics and descriptors needs to have read and write access with those flags ESP_GATT_PERM_READ_ENCRYPTED | ESP_GATT_PERM_WRITE_ENCRYPTED.
This is example how to use it:
https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/tests/BLETests/security/SampleServer_authorization.cpp

Also, i think, line 74 is commented on purpose, so this example should exactly depict that quote.
You can connect with no pairing/bonding but when you try to read/write characteristics or descriptors you will be asked to pair/bond first.

EDIT: If its the case then HID class can be changed a bit to easy code usage.

from esp32_mouse_keyboard.

Code:
esp_ble_auth_req_t auth_req = ESP_LE_AUTH_REQ_SC_ONLY;
esp_ble_gap_set_security_param(ESP_BLE_SM_AUTHEN_REQ_MODE, &auth_req, sizeof(uint8_t));

esp_ble_io_cap_t iocap = ESP_IO_CAP_NONE;           //set the IO capability to No output No input
esp_ble_gap_set_security_param(ESP_BLE_SM_IOCAP_MODE, &iocap, sizeof(uint8_t));

uint8_t init_key = ESP_BLE_ENC_KEY_MASK | ESP_BLE_ID_KEY_MASK;
esp_ble_gap_set_security_param(ESP_BLE_SM_SET_INIT_KEY, &init_key, sizeof(uint8_t));

So this is how I've setup the connection, the only bit I'm not sure how to add is the BLE2902 descriptor/characteristic. Any idea how to do that in this code?

Edit: Does it have something to do with:
/// Full Hid device Database Description - Used to add attributes into the database static esp_gatts_attr_db_t hidd_le_gatt_db[HIDD_LE_IDX_NB] =

from esp32_mouse_keyboard.

BLEHID class has been write the way to add all required elements like descriptors in the background. For example inputReport characteristic:
https://github.com/asterics/esp32_mouse_keyboard/blob/newBLE/components/nkolban_BLE/BLEHIDDevice.cpp#L111

You have 2 descriptors for every inputReport characteristic. By design there is no need to access descriptors to write good hid app , but you can do it anyway with:
https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/BLECharacteristic.cpp#L144
BLEDescriptor* desc = inputReport->getDescriptorByUUID("0x2902");

If at the end it will be the case then i will have to update BLEHID class and all of this will be implemented inside this class. If you need more info or help just ask.

from esp32_mouse_keyboard.

@JuanPotato @cheuklam @dalinarkholin @S-March

This issue seemed to be related to following wrong settings:

• iOS needs an unencrypted connection at the beginning
• Instead of an encrypted connection, each descriptor/characteristics needs access permission settings (ESP_GATT_PERM_READ_ENCRYPTED and/or ESP_GATT_PERM_WRITE_ENCRYPTED)

If it is possible for you, it would be nice if you give it another try (don't forget to use the newBLE branch) and report back if we can close this issue.

Thank you very much for suggestions/hints/log files!

from esp32_mouse_keyboard.

I've been doing some more testing and have noticed 2 things:

1. If you disable mouse support in the HID_kbdmousejoystick_init() call, it still pairs, but the keycodes are ignored on iOS.
2. Unpairing works, however selecting disconnect doesn't appear to disconnect the HID device.
   Edit: I don't see any logs when selecting disconnect and can continue to send keycodes. Perhaps iOS isn't actually sending a disconnect?

from esp32_mouse_keyboard.

Maybe now i am wiser and i will be able to help/answer other iPhone maniacs/users with ble issues.

Obviously it still require some more testing.

from esp32_mouse_keyboard.

@benjaminaigner verified it works on IOS 11.3 iphone 7+!!!!!

Well done and thank you! Super pumped I can move forward with my project. Im excited about this module and will building off of it in the near future. Would be good to hear about where you want to go with it. Thanks again man.

from esp32_mouse_keyboard.

@dalinarkholin

THX for the verification.

As we will use this example within an assistive device ( Flipmouse & FLipMouse ESP32 ) , we need to have iOS support.
Most of AAC grids use iPads...

from esp32_mouse_keyboard.

I will close this issue now, if somebody does not agree I will re-open it again.

from esp32_mouse_keyboard.

I also confirm it is working in the latest iOS11.4 iPhone 6S! Thanks @benjaminaigner and everyone for the contribution

from esp32_mouse_keyboard.