garywill / vimouse Goto Github PK

View Code? Open in Web Editor NEW

63.0 2.0 1.0 170 KB

Screen vision-recognition + vim/vimium. Let's do some tries to dump traditional mouse (experimental)

License: GNU General Public License v3.0

Python 100.00%

keyboard keyboard-mouse mouse vi vim vimium

vimouse's Introduction

Let's do some tries to dump traditional mouse. Inspired by Vim & Vimium.

viMouse

Use keyboard to click anywhere of screen. OpenCV based screenshot visual recognition.

The vi in the name here can mean "Vim", "Vision-recognition" or whatever

Dependencies:

python3 and: opencv , qt5 , numpy , pynput

Theoretically works cross-platform. Tested on Linux X11 and Windows. Welcome feedback the result on your machine.

This is still very simple and in alpha preview stage currently. Anything could change.

Usage

cd into folder and run python3 main.py to start.

Hit ctrl + meta to find clickable objects on screen and show keys

meta key = win key = super key

During keys shown, any key events are blocked from being sent to any other apps

To cancel, press ESC or ,(comma). To return to last step, press backspace
Press keys to move mouse to that position
It triggers click (if autoClick=True. Otherwise hit meta + ctrl to trigger click)

Edit file g.py and configure for your need: screenshotDelay, whole screen, autoClick ...

Algorithm (filtering, recognizing, parameters... ) can be improved. If you don't have a high contrast GUI, or some icons or buttons don't have clear contour, or two objects too close, it may fail to recognize.

We're expecting experts comming !

Other vi / keybinding projects

Keyboard clicking:

hunt-and-peck Simple vimium/vimperator style navigation for Windows applications based on the UI Automation framework.
warpd A modal keyboard driven interface for mouse manipulation.
vimac keyboard-driven navigation and control of the macOS Graphical User Interface (GUI)
vimium-everywhere OS-wide Keyboard navigation for Linux and Windows
keynav Control the mouse with the keyboard. only works on X11
Vimperator (discontinued) Make Firefox/Thunderbird look and behave like Vim
vimkey Browser extension. Use the keyboard trigger click button or open a link
Vimium browser extension that provides keyboard-based navigation and control of the web in the spirit of the Vim editor.
Vimium C A Customized Vimium having contextual mapping, global shortcuts, command sequences
vimb Vim-like web browser. The goal of Vimb is to build a completely keyboard-driven, efficient and pleasurable browsing-experience with low memory and CPU usage that is intuitive to use for Vim users.
Big Search A multiple-search-engine browser extension that has keyboard clicking feature on its UI

Edit text using vi-like keybinding on non-vi app:

gtk-vikb (discontinued) add vi keybindings to gtk (2/3) text areas
Firenvim Turn browser <textarea> into a Neovim client
wasavi Transforms browser <textarea> into a VI editor
emacs-anywhere
Emacs Everywhere
vime Using Vim as an input method editor for X11 applications
vim_ahk Based on AutoHotKey
tsf-vim (discontinued) Windows IME to edit text like vim editor

vimouse's People

Contributors

Stargazers

Watchers

Forkers

artemzdor

vimouse's Issues

crash after pressing ctrl+meta

app starting

starting key listener..
listening

, but after pressing ctrl+meta the following message appears in the console:

key listener stop or reset...
resetKeyPrsd()
showWindow()
create thread for QtApplication
WARNING: QApplication was not created in the main() thread.
key listener: NOW BLOCKING KEY EVENTS BEING SENT TO OTHER APPS, until key badges dismiss
QSettings::value: Empty key passed
QSettings::value: Empty key passed
QWidget subclass init finish
refresh()
Traceback (most recent call last):
File "/home/bwz/downloads/vimouse-master/tgui.py", line 158, in paintEvent
painter.setFont( QFont( 'Arial' , letterH, QFont.Bold) )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: arguments did not match any overloaded call:
QFont(): too many arguments
QFont(family: str, pointSize: int = -1, weight: int = -1, italic: bool = False): argument 2 has unexpected type 'float'
QFont(a0: QFont, pd: QPaintDevice): argument 1 has unexpected type 'str'
QFont(a0: QFont): argument 1 has unexpected type 'str'
QFont(variant: Any): too many arguments
zsh: IOT instruction (core dumped) python3 main.py

how i can fix it?

Error starting an application

Hello, the first time I ran the application I got this error:
starting key listener.. listening resetKeyPrsd() 148 2 showWindow() WARNING: QApplication was not created in the main() thread. QObject::moveToThread: Current thread (0x7fdacc000ee0) is not the object's thread (0x7fdacc23fa50). Cannot move to target thread (0x55bd23364d10) qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/paul/.local/lib/python3.11/site-packages/cv2/qt/plugins" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem. Available platform plugins are: xcb, eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx.

Reinstalled the opencv library:

pip uninstall opencv-python && pip install opencv-python-headless

After restarting I get the following error:

starting key listener.. listening resetKeyPrsd() 76 2 showWindow() WARNING: QApplication was not created in the main() thread. QWidget subclass __init__ 1300 500 QWidget subclass __init__ finish QWidget subclass paintEvent Traceback (most recent call last): File "/home/paul/Desktop/vimouse/viscreen.py", line 380, in paintEvent self.paintLabel(''.join(kpc['keyp']) , kpc['cord'][0] , kpc['cord'][1] ) File "/home/paul/Desktop/vimouse/viscreen.py", line 343, in paintLabel qp.fillRect( x-w/2-w_ex , y-h/2-h_ex , w+2*w_ex, h+2*h_ex , b_color ) TypeError: arguments did not match any overloaded call: fillRect(self, a0: QRectF, a1: Union[QBrush, QColor, Qt.GlobalColor, QGradient]): argument 1 has unexpected type 'numpy.float64' fillRect(self, a0: QRect, a1: Union[QBrush, QColor, Qt.GlobalColor, QGradient]): argument 1 has unexpected type 'numpy.float64' fillRect(self, x: int, y: int, w: int, h: int, b: Union[QBrush, QColor, Qt.GlobalColor, QGradient]): argument 1 has unexpected type 'numpy.float64' fillRect(self, a0: QRectF, color: Union[QColor, Qt.GlobalColor, QGradient]): argument 1 has unexpected type 'numpy.float64' fillRect(self, a0: QRect, color: Union[QColor, Qt.GlobalColor, QGradient]): argument 1 has unexpected type 'numpy.float64' fillRect(self, x: int, y: int, w: int, h: int, b: Union[QColor, Qt.GlobalColor, QGradient]): argument 1 has unexpected type 'numpy.float64' fillRect(self, x: int, y: int, w: int, h: int, c: Qt.GlobalColor): argument 1 has unexpected type 'numpy.float64' fillRect(self, r: QRect, c: Qt.GlobalColor): argument 1 has unexpected type 'numpy.float64' fillRect(self, r: QRectF, c: Qt.GlobalColor): argument 1 has unexpected type 'numpy.float64' fillRect(self, x: int, y: int, w: int, h: int, style: Qt.BrushStyle): argument 1 has unexpected type 'numpy.float64' fillRect(self, r: QRect, style: Qt.BrushStyle): argument 1 has unexpected type 'numpy.float64' fillRect(self, r: QRectF, style: Qt.BrushStyle): argument 1 has unexpected type 'numpy.float64' fillRect(self, x: int, y: int, w: int, h: int, preset: QGradient.Preset): argument 1 has unexpected type 'numpy.float64' fillRect(self, r: QRect, preset: QGradient.Preset): argument 1 has unexpected type 'numpy.float64' fillRect(self, r: QRectF, preset: QGradient.Preset): argument 1 has unexpected type 'numpy.float64' [1] 5726 IOT instruction (core dumped) python3.11 viscreen.py

wayland support

hello

please add a wayland support

garywill / vimouse Goto Github PK

vimouse's Introduction

viMouse

Usage

Other vi / keybinding projects

vimouse's People

Contributors

Stargazers

Watchers

Forkers

vimouse's Issues

crash after pressing ctrl+meta

Error starting an application

wayland support

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent