The grid-based and element-based navigation modes work incorrectly if your machine does not have the "automatically hide and show the Dock" setting enabled and the Dock is positioned on the left or bottom edge of the screen.

Stated a different way:

• if you auto-hide your Dock, or
• if you have your Dock always visible on the right edge of the screen

...then everything works properly.

Otherwise:

• element-based nav will jump to the correct pixel coordinates, but the labels are drawn in an incorrect location on the screen (they're offset by the size of the Dock)
• grid-based nav will jump to incorrect pixel coordinates (offset by the size of the Dock), but the grid will draw correctly

This isn't a regression; I just never thought to test without an auto-hidden Dock 😓

Like #7, but for all of the keybinds that are not global-registered.

Heya – amazing app, thanks so much for creating this!

I was wondering if you’d consider supporting disabling Scoot shortcuts when some app is in foreground.

My use case: I don’t want to trigger Scoot shortcuts when I’m using VS Code. I’d never control VS Code with Scoot as I already actively rely on shortcuts there; and Scoot steals some of these shortcuts, which means I need to re-assign them in one of the apps.

is it possible to only show the grid on the active monitor? I have three monitors and it’s a little jarring for the grid to show on all 3 at the same time.

thanks!

First off, thank you for the useful tool! It's absolutely brilliant.

In some context, drag and drop works great. For instance, dragging files around in Finder. In other cases, the grid disappears as soon as i hit the = key. Opening the grid again with Cmd-shift-k freezes the application, requiring manually using the mouse to click away.

Contexts that break scoot:

• Dragging windows using their title bar (such as trying to re-position a terminal window)
• Trying to reorder chrome tabs

Steps to reproduce:

• Open scoot grid Cmd-shift-k
• Position your cursor over a window title bar
• Press =

Observe that the grid is hidden now. Attempting to reopen it yields an unresponsive scoot grid

Improve this project's CI, for example to enable the app to automatically be built/signed/notarized/released.

For example, see: https://betterprogramming.pub/indie-mac-app-devops-with-github-actions-b16764a3ebe7

Just want to say this is amazing. Much cheaper than hardware eye tracking (Talon, Precision Gauze) for not using the mouse.

Would be great if the grid layout could persist surviving the mouse click.
There are times where you want to narrow down a tree that requires multiple grid activation.

When I run Scoot it conflicts with my Keyboard Maestro macro's that handle window management for me. For example, I have a shortcut to move and size a window to 50% of the screen. It toggles between the left side and the right side. When scoot is running, this is suddenly broken.

This is the crucial part in Keyboard Maestro macro for resizing and moving the front most window.

This is the behavior with and without scoot running.

This project is almost perfect and it replaces 3 or 4 apps at once. But there is one thing that I would really like to see: have both element and grid based navigation work with the macOS menu bar. Is this feasible?

Now it just enters in to navigation mode, would be useful to toggle it by the same key press, without needing to move finger to ESCAPE key.

I know this is probably out of scope due to implementation complexity, but I’d thought I’d file for posterity. Perhaps there’s a statically linkable library that makes implementing this easy?

The feature request is to support vi bindings in a more complete way, so that I could e.g. hit 10j to move 10 steps down.

If the focused window in the frontmost app has a scroll view, and you pull up Scoot (in element-based nav mode), and then start scrolling, the regions that Scoot has drawn won't update.

There are (at least) two ways to scroll:

• using Scoot's keyboard shortcuts for scrolling, or
• using your physical mouse or trackpad)

For the latter case, are there any accessibility-related hooks in MacOS to detect if the user is scrolling?

Originally posted by @mjrusso in #1 (comment)

• Invoke Scoot freestyle with Cmd+Shift+L
• Open settings with Cmd+,
• Invoke Scoot freestyle again with Cmd+Shift+L
• Navigate to point to Scoot settings window close button and hit Enter
• The window seemingly goes away
• Invoke Scoot freestyle for the third time with Cmd+Shift+L
• The settings window pops up to the forefront, blocking the view

The current click simulation code is written as follows (source):

    func click(button: Mouse.Button) {
        // It should be possible to `pressDown()` and then `release()` (after a
        // short delay), however that approach doesn't work reliably: in some
        // applications (for example, when trying to click on a hyperlink in a
        // browser, or in an apps that render their user interface using web
        // technologies), mouse clicks don't actually register.
        //
        // For an unknown reason, posting two `leftMouseDown` events, followed
        // by a single `leftMouseUp`, *seems* to work consistently everywhere.
        // How's that for a hack?
        //
        // [N.B.: A previous implementation issued two clicks in a row (i.e.,
        // `pressDown()`, `release()`, `pressDown()`, `release()`), and that
        // worked better than one could reasonably expect. However, there were
        // rare cases where two clicks would actually be registered: for example,
        // in Safari, clicking the "close tab" button (the one embedded in the
        // tab itself) would result in two tabs being closed. Another example:
        // it wasn't possible to use Scoot to bring up it's menu bar: two clicks
        // would register, and it would immediately close.]
        //
        // [N.B.: This question [0] on Stack Overflow ("Simulating mouse clicks
        // on Mac OS X does not work for some applications") seems relevant, but
        // unfortunately none of the proposed strategies seem to work.]
        //
        // [0]: https://stackoverflow.com/q/2369806/15304124
        pressDown(button)
        pressDown(button)
        usleep(40000)
        release(button)
    }

I'd love to understand why this hack is necessary, and ideally modify the implementation to pressDown only a single time.

Would love to see this release on Linux also.

Legibility when using element-based navigation is (usually) OK-ish, but far from perfect.

As an example of a degenerate case, here's what HN looks like when element-based nav is active:

The combination of a small font size, with many unique anchors in close proximity, is problematic. It might make sense to ignore elements that are clustered too closely together (using size as a proxy for importance).

Originally posted by @mjrusso in #1 (comment)

Howdy! Wanted to point out that there is an issue with the grid-based navigation when MacOS Stage Manager is enabled. The grid is offset by the Stage Manager gutter (the left side of the screen where Stage Manager window groups are collected.) In addition to the gutter not being navigable with the grid, attempting to navigate to a grid ID will misplace the cursor by the same width of the gutter. For example, in the screenshot below, navigating to azw would put the cursor ~213px to the left of azw.

I have a proposal for new navigation mode

1. Screen is divided to 2x2 grid
2. Selecting one of grid cell (each grid has dedicated button)
3. Selected cell will be divided again to 2x2

User keeps selecting the cell until destination is reached. I built a POC for testing the viability of this (just simple app running in browser) and I think it has some potential.

Asking for comments:

• Have you already experimented with something like this before? How did you find it?
• Would this be simple to implement? (I am considering trying to make the PR myself, if this makes sense).

POC: https://arch-vile.github.io/4thSlot/

It can be hard to read grid and element labels in a variety of circumstances, but especially when Scoot is rendering on top of light backgrounds with dark text.

(This issue primarily affects grid-based nav, but the same technique should also be used to improve the drawing of bounding rects for elements when using element-based nav. For an entirely different legibility issue with element-based nav, see #2.)

Is there a (performant) way to modify the colours of the grid based on the content immediately underneath it?

Not sure if I haven't read the instructions properly yet but I couldn't see how to initiate a right click.
I guess it would be a little tricky as it would quite often bring up a context menu that may be hard to target.

Anyhow thanks for Scoot great idea!

From #25:

Currently, when Scoot is activated, it traverses the accessibility tree for the frontmost/focused application and builds the data structure used for element-based navigation, regardless of which navigation mode is requested. The thinking here was that once Scoot is on screen, you could swap between (for example) grid-based nav and element-based nav, without any delay.

If you've activated Scoot in the grid-based or freestyle nav modes, you should be able to use the requested mode immediately, without blocking on element-based nav becoming ready.

Either wait for element-based nav to be requested before doing the work of traversing the accessibility tree (and prepping the associated data structures, etc.), or do this work in a background thread (if possible).

Bringing Scoot to the foreground, with Zoom as the frontmost app, causes a crash. Note that this only seems to happen when actively in a Zoom meeting.

I can reproduce with Zoom 5.8.4 and 5.9.1.

The crash appears to be somewhere in AXSwift, but I haven't had a chance to dig any further yet.

First of all thanks for this super interesting tool! I have been trying to reduce my mouse usage and this definitely could be viable. Only been using this for a day now but I have an idea that I want to suggest.

It could be useful feature if Scoot would remember the clicked locations on each app (or just for all). And there would be a navigation mode that would show the most clicked locations and a shortcut to click/move to those. I can see some issues if you move windows around or use multiple displays but as an optional feature would help me at least.

Is it possible hide the grid after I reach the point? Currently have to press escape to quit the grid mode

I have an external display as the main display connected via USB-C to HDMI adapter to my 2017 Macbook Pro running OS X 12.1 with the built-in retina as an extended display. Launching grid navigation with Scoot 0.8 overlays both display grids on the external display and none on the built-in retina display. Relatedly, launching element navigation only shows targets when the focused window is on the external display.

Introducing the ability to create and execute macros would make Scoot a handy, lightweight automation tool.

This issue might be a bit tricky to debug, but is definitely something I think extremely limits the use of Scoot.

I'm using 2 monitors - the build in Mac display and an external monitor
When launching the grid mode while focused on the external monitor, the grid loads up on both displays almost instantaneously.
When launching it when focused on the build-in display, though, it can take up to 7 seconds at times to render the grid on both displays.

Steps to reproduce:

1. Click anywhere on the external display
2. Launch Scoot's grid mode with keyboard shortcut
3. Grid loads almost instantaneously on both displays
4. Press Esc to close grid mode
5. Click anywhere on the built-in display
6. Launch Scoot's grind mode with keyboard shortcut
7. Grid takes a while to render, making Scoot impractical to use

Scoot version: 0.12 (1)
MacOS version: 12.2.1
Model: MacBook Pro 13-inch 2017, 2.3GHz i5 & 16GB RAM

Thanks,
Orel

Currently, Scoot (globally) binds ⇧⌘J and ⇧⌘K to bring itself to the foreground (in element-based or grid-based nav modes, respectively).

You may be using these for other purposes, or just prefer different shortcuts, and it should be possible to change them.

There are times when I want to perform multiple clicks on UI elements. Currently, I have to toggle freestyle mode for every click since the overlaid UI element is automatically hidden after each click. It'll be nice if there's an option or settings for that or better yet, a key to click without hiding the overlaid UI.

In grid or freestyle mode, when I attempt to do a double left click with "=" all keys become un-responsive. The only exception seems to be CMD-SPACE, which still brings up Spotlight search. Running Debug > "Rebuild jump windows" returns full functionality.

System Details
MacBook Air (2020)
MacOS Sonoma 14.4.1
Scoot 1.2 (1)

Scoot is very opinionated about its colour scheme, but these colours may not be right for you -- maybe they're hard to read, or maybe you just don't like teal. It would be great if these could be customized.

Note that this may impact other legibility-related tickets, such as #4.

Issue #12 adds vi keybindings, but the bindings for j/k up/down are reversed. j should move the cursor down. k should move the cursor up (similar for related modifiers). man vi

First of all thank for this amazing contribution. I love using the app.

Steps to reproduce:

1. Open any .dmg file
2. Do CMD+shift+L
3. vim bindings doesn't work.

this is the only place I see this feature is not working so far. I was able to go mouseless pretty much all the time.

Ideally, Scoot should be able to find all elements on the screen, even if VoiceOver is off.

Relevant resources:

• https://www.electronjs.org/docs/latest/tutorial/accessibility
• electron/electron#7206
• electron/electron#10305
• https://www.chromium.org/developers/design-documents/accessibility#TOC-How-Chrome-detects-the-presence-of-Assistive-Technology
• https://bugzilla.mozilla.org/show_bug.cgi?id=1664992

This code sample also looks handy: https://github.com/rxhanson/Rectangle/blob/master/Rectangle/AccessibilityElement.swift

(Alternative title: Meta-Scoot: Scoot cannot "scoot" its own windows)

It isn't possible to interact with Scoot's windows (e.g., the Settings screen: #21, the About screen: a0fb6c2, etc.), using element-based navigation.

This is because of the following code, which prevents Scoot from "seeing" its own windows:

    /// The frontmost application at the moment when Scoot was most recently
    /// invoked. (If Scoot happens to be the frontmost app when it is invoked,
    /// the previous frontmost app is returned.)
    var currentApp: NSRunningApplication? {
        didSet {
            // Don't allow Scoot to be set as the current app; instead, retain
            // the previous value. (This behaviour provides a simple mechanism
            // for recomputing available elements when using element-based
            // navigation: simply re-invoke Scoot, even if it is frontmost.)
            guard currentApp != .current else {
                currentApp = oldValue
                return
            }
        }
    }

To fix this, we'll need to only ignore Scoot if NSApp.keyWindow (and NSApp.mainWindow?) is currently inputWindow.

I quickly prototyped this, and the technique itself mostly works, however when I click on a tab in the Settings screen, or pull up the color picker, etc., Scoot stops accepting any keyboard input. The commands are being received, however, because if I use my mouse to click, all the queued keyboard commands are immediately replayed.

Scoot currently allows scrolling to be simulated, but not more advanced gestures (like swiping, pinching, spreading, and rotating).

If technically possible, add support for simulating advanced gestures.

In element-based nav mode (first introduced in #1), Scoot will find all elements in the focused window of the frontmost app.

It would be better if Scoot could find all elements that are on all attached screen(s), across all apps.

Of course, windows can overlap, and we'll need to ignore any elements that are fully occluded by a window above it.

AXUIElementCopyElementAtPosition will find an element at the specific coordinate (if one exists), but you need to pass it an application reference, so it is not directly usable in this context (and, regardless, checking every possible x,y value on every screen seems wasteful from a performance perspective).

https://stackoverflow.com/a/56941105/ has some code that might be helpful (finding all windows on the screen, and ordering them relative to each other).

(Originally requested by @acicovic in this comment.)

Would love if we could set it to use vi key shortcuts!

This is a super cool project! Have always wanted to use my keyboard to control the mouse!!!

Sparkle: https://sparkle-project.org

Would be neat to do this entirely via GitHub (some ideas: sparkle-project/Sparkle#648)

I seem to randomly get beeps/sounds when using my mouse, clicking or moving my cursor after having used element based navigation, can we have an option to disable this beeping sound?

https://github.com/sindresorhus/LaunchAtLogin looks promising.