botcity-dev / botcity-framework-core-python Goto Github PK

What's the problem this feature will solve?
Adding the return types will make IDEs better provide users with autocomplete for methods and code in general.

Describe the solution you'd like
Types to be annotated where possible.

Additional context
Motivated by forum post https://community.botcity.dev/t/function-menu-select-error/646/2

Hello,

One question, i am trying to build a .exe with pyinstaller of the complete desktop automation, but i am having some problems when i execute, i think after the file.exe its complete, it doesnt find the images inside the resources folder.
Is there anyway i can work around this?

Best regards

Describe the bug
When executing the command python3 bot.py an error occurs in the PIL library, returning the following error:

ImportError: ImageGrab is macOS and Windows only

The steps described in the documentation for possible problems in the Linux system were performed.

Expected behavior
The bot should run and open the system's default browser.

Steps to Reproduce

1. BotCity template download for desktop
2. Install dependencies
3. Run the command python3 bot.py

Possible Solution
Researching the problem, I found a lib that replaces ImageGrab for Linux systems.

Suggestion: https://stackoverflow.com/questions/43520757/imagegrab-alternative-in-linux
Package: https://pypi.org/project/pyscreenshot/

My Platform

• OS Version: Ubuntu 20.04.6
• Python Version: 3.8.10
• Packages Version:
  
  

Additional context

I am trying to make my login in the BotStudio.exe, but every time i try to do this, this error shows up and i can't fix this, somebody can help?

Describe the bug
On MacOS, if using a resolution different than default the find command works properly but the coordinates are wrong as they are calculated based on the default screen resolution and not the one currently in use.

Expected behavior
Find works and the coordinates are adjusted to the user's selected monitor resolution.

Steps to Reproduce

• Configure the resolution to be different than the default for the display
• Perform a find
• Move the mouse to the element center
• Observe that it won't be in the proper location

Possible Solution

My Platform

• MacOS
• Display with 4K resolution
• System set up to half of the display capable resolution

Additional context

$ system_profiler SPDisplaysDataType
Graphics/Displays:

    Apple M2 Pro:

      Chipset Model: Apple M2 Pro
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 19
      Vendor: Apple (0x106b)
      Metal Support: Metal 3
      Displays:
        DELL P2715Q:
          Resolution: 3840 x 2160 (2160p/4K UHD 1 - Ultra High Definition)
          UI Looks like: 1920 x 1080 @ 30.00Hz
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Rotation: Supported

(temp) $ python -c "from botcity.core import DesktopBot; bot = DesktopBot(); print(bot.get_screenshot().size)"
(3840, 2160)

Description

When trying to type a special character or a string that contains it, the output is not as expected using the kb_type() and type_keys() methods. It is only possible to use special characters through the paste() method.

Description:

• Some screenshots from the project's README file are using an old version of Maestro.

The keyboard commands such as bot.enter(), bot.tab(), among others, have an option for the wait argument, where an integer value is passed to wait after the action is performed.

I would like to suggest a modification to insert this option to wait both before and after the action, something like wait_before and wait_after, which would assist in development instead of having to pass a bot.wait() before performing the action.

Describe the bug
When using the find_multiple method, passing a list of elements as a parameter, the following error is thrown:

Traceback (most recent call last):
 File "c:\Users\lsmac\Desktop\test-plugin-studio\test-plugin-studio\bot.py", line 113, in <module>
   main()
 File "c:\Users\lsmac\Desktop\test-plugin-studio\test-plugin-studio\bot.py", line 54, in main    
   elements = desktop_bot.find_multiple(elements_to_find)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\lsmac\AppData\Local\Programs\Python\Python311\Lib\site-packages\botcity\core\bot.py", line 212, in find_multiple
   results = pool.starmap(helper, paths)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\lsmac\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 375, in starmap
   return self._map_async(func, iterable, starmapstar, chunksize).get()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\lsmac\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 774, in get
   raise self._value
 File "C:\Users\lsmac\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 540, in _handle_tasks
   put(task)
 File "C:\Users\lsmac\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\connection.py", line 205, in send
   self._send_bytes(_ForkingPickler.dumps(obj))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\lsmac\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\reduction.py", line 51, in dumps
   cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

Expected behavior
It is expected that when executing the find_mutiple method, a dictionary will be returned with the label of each element passed as a parameter and their respective coordinates.

Steps to Reproduce
Below is the code that I am using:

elements_to_find = ["home", "workspace"]

elements = desktop_bot.find_multiple(elements_to_find)

print(elements)

Additional context
Apparently, the problem occurs on this line.

Describe the bug

find.all sometime returns duplicated objects on specific matching range
find.all in version 0.4.0 find.all returns the list of objects by matchscore.
In version 0.3.0 it was in the order they are found on the screen.

Box: x=1447, y=740, w=52, h=29, score=0.9999566078186035
Box: x=1102, y=740, w=52, h=29, score=0.967707633972168
Box: x=1620, y=740, w=52, h=29, score=0.9400887489318848
Box: x=1275, y=740, w=52, h=29, score=0.8545419573783875
Box: x=1274, y=740, w=52, h=29, score=0.846000075340271
Box: x=1533, y=641, w=52, h=29, score=0.82472825050354

objects 4 and 5 are the same

objects are shown in order of best score

Expected behavior
Box: x=1447, y=740, w=52, h=29, score=0.9999566078186035
Box: x=1102, y=740, w=52, h=29, score=0.967707633972168
Box: x=1620, y=740, w=52, h=29, score=0.9400887489318848
Box: x=1275, y=740, w=52, h=29, score=0.8545419573783875
Box: x=1533, y=641, w=52, h=29, score=0.82472825050354

Steps to Reproduce

cv2find.py edited to print score:

print(f"Box: x={x}, y={y}, w={needle_width}, h={needle_height}, score={result[y]}")
yield Box(x, y, needle_width, needle_height)

elements = self.find_all(label="arena_150", matching=0.81, waiting_time=1000, grayscale=False)
elements_list = list(elements)

Possible Solution
Fix duplicated entry

Please give find.all the option to list objects by score and also as it was in version 0.3.0 in the order they appeared on the screen (left-right-top-bottom)

for x, y in zip(matchx, matchy):
    yield Box(x, y, needle_width, needle_height)

My Platform

botcity-email-plugin 0.1.3
botcity-framework-base 0.4.3
botcity-framework-core 0.3.0+2.g54db502
botcity-framework-web 0.8.2
botcity-maestro-sdk 0.3.4
botcity-telegram-plugin 0.1.4

Additional context

Windows scale 100%, image scale 100%

needle / label / resource

What's the problem this feature will solve?
This new method would allow us to look for processes running on the machine easily and act upon them, executing operations such as terminate.

Describe the solution you'd like
I would like to have a method find_process, which would take keyword arguments that would allow me to find a process by any of the available keys related to a process returned by the psutil package method process_iter.

def find_process(**kwargs) -> Generator[Process, None, None]:
    processes = psutil.process_iter()
    for proc in processes:
        ...
        if match:
            yield proc

Additional context
This will be very useful for cleanup of processes.

What's the problem this feature will solve?
I am testing a DesktopBot automation, sometime when i use .browse("www.google.com.br"). THe bot popup the browser, sometime it doesn't. Is there a way that i can control it?

Describe the solution you'd like
I mean, pop it up if i want to or don't popup.

Description

• The find process method returns only a Process object, but the return type is List[Process]

Describe the bug
copy_to_clipboard() raises error:

/media/luiz/Dados/repo/android-test/venv/bin/python /media/luiz/Dados/repo/android-test/android.py
Traceback (most recent call last):
  File "/media/luiz/Dados/repo/android-test/android.py", line 72, in <module>
    bot.copy_to_clipboard(text='')
  File "/media/luiz/Dados/repo/android-test/venv/lib/python3.8/site-packages/botcity/core/bot.py", line 845, in copy_to_clipboard
    pyperclip.copy(text)
  File "/media/luiz/Dados/repo/android-test/venv/lib/python3.8/site-packages/pyperclip/__init__.py", line 659, in lazy_load_stub_copy
    return copy(text)
  File "/media/luiz/Dados/repo/android-test/venv/lib/python3.8/site-packages/pyperclip/__init__.py", line 336, in __call__
    raise PyperclipException(EXCEPT_MSG)
pyperclip.PyperclipException: 
    Pyperclip could not find a copy/paste mechanism for your system.
    For more information, please visit https://pyperclip.readthedocs.io/en/latest/index.html#not-implemented-error 

Expected behavior
Copy text to clipboard

Steps to Reproduce

from botcity.core import DesktopBot

bot = DesktopBot()
bot.copy_to_clipboard(text='')
print(bot.get_clipboard())

My Platform
* OS: Ubuntu 20.04
* Python Version: 3.8.10
* Packages Version: botcity-framework-core=0.2.8

Additional context

What's the problem this feature will solve?
pyautogui has too many bugs and it is impacting our ability to move forward and solve issues such as #23 .

Describe the solution you'd like
Replace pyautogui with pynput.

Describe the bug
As of now the documentation doesn't mention what type of exception is raised on methods.

Description

The screen_cut method expects the width and height of the screenshot region to be passed, but the method only works when we pass the width and height added to the initial value of x and y.

This is probably being caused by the way ImageGrab.grab() handles these parameters.