artfxdev / silex_client Goto Github PK

View Code? Open in Web Editor NEW

14.0 6.0 1.0 8.76 MB

Python library to execute silex actions

License: MIT License

Python 100.00%

library silex rez vfx websockets

silex_client's People

Contributors

Stargazers

Watchers

Forkers

michaelhaussmann

silex_client's Issues

Better errors with YAML action config files

When the commands key is empty:

double:
steps:
  init:
    index: 0
    label: "Logging"
    commands:

Actual:

dacite.exceptions.WrongTypeError: wrong value type for field "steps.commands" -
should be "typing.Dict[str, silex_client.action.command_buffer.CommandBuffer]" instead of value "None" of type "NoneType"

Expected: Error: no commands given

If I have a double.yml configuration file for an action in the config folder with the following:

# double.yml

test:
  steps:
    init:
      index: 0
      label: "Logging"
      commands:
        log_first:
          label: "First log"
          path: "silex_client.commands.utils.Log"
          parameters:
            message: "Initialization..."
            level: "info"

The root key is not equal to the filename.

Actual:

[action_query._initialize_buffer] Could not initialise the action: The resolved config is invalid (112)

Expected: Error: the top most key is not the same as the filename

When we have multiple commands with the same key name:

# double.yml

double:
steps:
  init:
    index: 0
    label: "Logging"
    commands:
      log_first:
        label: "First log"
        path: "silex_client.commands.utils.Log"
        parameters:
          message: "First log..."
          level: "info"

      log_first:
        label: "Second log"
        path: "silex_client.commands.utils.Log"
        parameters:
          message: "Second log..."
          level: "warn"

Actual (actionquery json):

{
  "name": "double",
  "uuid": "eee7388c26d411ecb599ab3705c91dc2",
  "status": 2,
  "steps": {
    "init": {
      "name": "init",
      "index": 0,
      "status": 2,
      "label": "Logging",
      "hide": false,
    "tooltip": "",
      "commands": {
        "log_first": {
          "path": "silex_client.commands.utils.Log",
          "name": "log_first",
          "label": "Second log",
          "hide": false,
          "tooltip": "",
          "parameters": {
            "message": {
              "label": "Message",
              "type": "str",
              "value": "Second log..."
            },
            "level": {
              "label": "Level",
              "type": "str",
              "value": "warn"
            }
          },
          "uuid": "eee7388d26d411ecb599ab3705c91dc2",
          "status": 2
        }
      },
      "uuid": "eee7388f26d411ecb599ab3705c91dc2"
    }
  },
  // ...
}

Expected: Warning: multiple command keys have the same name!

Clean shelf actions

Remove unused actions (proprosal, demo_async, stress-test and tester only in beta)
Add icons

Implement action queries protocol

Agree on a protocol to communicate the query, progression, update of an action

Implement serialisation of the action query object
Create an interface to send and reveive updates of an action

Implement the undo

Execute the undo on error
Implement the undo on the BuildOutputPath command

Create a versatile conform action

Guesse the conform type and execute the action
Create some conform for basic types like PNG, EXR, OBJ, FBX...
Make sure to ask if the conformed files should be merged with the existing files or replacing completely

Add some parameter types

Add a parameter to select from a list
Add a parameter to select from a range
Add a parameter to select some paths

Refactor conform commands to avoid repetition

Currently, all the conform configs are identical yamls with the conform type as only difference: https://github.com/ArtFXDev/silex_client/tree/dev/silex_client/config/conform

With action inheritance it would be better to avoid repeating ourselves.

On top of that, add video file formats for conform:

Rework the log redirection

The current implementation of log redirection through websocket won't work when executing multiple actions asynchronously

The handler should actually be here all the timr
The context manager will register a frame and the command into the handler
The handler will traverse the outerframes until it reaches a context manager's frame and store the log into the command
If the handler reaches a frame called by an update_websocket function, skip the redirection to prevent infinite loop

Add more control on the execution of the actions

Be able to cancel an action
Be able to undo an action
Be able to redo an action

Authentificate with gazu on initialisation

Use rez env for submitters commands

Use rez for render jobs on the farm:

rez env vray fady -- vray.exe ....

Add unit tests with websocket connection

Test to send 1000 messages and check they were all received
Test deconnections and reconnections
Test the modifications of the parameters from websocket

Create some basic cross dcc submitter commands

Create a command that wraps the submitter to submit jobs
Make sure to use the input field and return values to make is reusable in all the other DCCs

Conform files with identical names but different folders

Unclosed aiohttp client session on clear action

When sending a clear event on the /dcc/action route (you can kill the action from the interface), it stops the running action but the session is not closed:

$ rez env silex_client-dev -- silex action tester -c dev
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7feb2a515250>

Handle exceptions in command execution

When a command raise an exception, the status should update to error
When a command raise an exception, the buffer should store the error message
It would be cool is the buffer could capture all the logs, so we could store them in an elastic search database

Refactor the CommandOutput class

The InsertAction command should use a CommandOutput instead of a string for the variables parameter_path and outpu_path
The function that returns the actual value of the CommandOutput should be in the CommandOutput class itself

Wrong docstring for Tractor sumbitter

silex_client/silex_client/commands/submit_to_tractor.py

Line 20 in 397fb06

Put the given file on database and to locked file system

Implement undo commands

At some point we want to be able to cancel an action if something went wrong. For that we need to implement undo commands for file moves, publish...

Ctrl+C on Linux: Task was destroyed but it is pending!

I am running this action in the terminal:

$ rez env silex_client -- silex action tester

And pressing Ctrl+C to cancel the action is not being caught:

^CInterrupted by user
[SILEX]    [211022 10:46:58] INFO      |    [websocket_namespace.on_disconnect] Disconected from ws://127.0.0.1:5118 on /dcc/action (46)
Traceback (most recent call last):
  File "/home/josephhenry/packages/python/3.7.9/platform-linux/arch-x86_64/os-Pop-21.04/python/lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/josephhenry/packages/python/3.7.9/platform-linux/arch-x86_64/os-Pop-21.04/python/lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/josephhenry/Documents/artfx/pipeline/silex/git/silex_client/silex_client/cli/parser.py", line 118, in <module>
    main()
  File "/home/josephhenry/Documents/artfx/pipeline/silex/git/silex_client/silex_client/cli/parser.py", line 114, in main
    HANDLERS_MAPPING[subcommand](**args)
  File "/home/josephhenry/Documents/artfx/pipeline/silex/git/silex_client/silex_client/cli/handlers.py", line 77, in action_handler
    futures.wait([silex_context.ws_connection.stop()], timeout=None)
  File "/home/josephhenry/Documents/artfx/pipeline/silex/git/silex_client/silex_client/network/websocket.py", line 102, in stop
    futures.wait([future])
  File "/home/josephhenry/packages/python/3.7.9/platform-linux/arch-x86_64/os-Pop-21.04/python/lib/concurrent/futures/_base.py", line 301, in wait
    waiter.event.wait(timeout)
  File "/home/josephhenry/packages/python/3.7.9/platform-linux/arch-x86_64/os-Pop-21.04/python/lib/threading.py", line 552, in wait
    signaled = self._cond.wait(timeout)
  File "/home/josephhenry/packages/python/3.7.9/platform-linux/arch-x86_64/os-Pop-21.04/python/lib/threading.py", line 296, in wait
    waiter.acquire()
  File "/home/josephhenry/packages/python_engineio/4.2.1/88a70aca30cb79a278872594adf043dc6c40af99/python/engineio/client.py", line 36, in signal_handler
[SILEX]    [211022 10:46:58] INFO      |    [websocket_namespace.on_disconnect] Disconected from ws://127.0.0.1:5118 on /dcc       (46)
    return original_signal_handler(sig, frame)
  File "/home/josephhenry/packages/python_socketio/5.4.0/88a70aca30cb79a278872594adf043dc6c40af99/python/socketio/client.py", line 26, in signal_handler
    return original_signal_handler(sig, frame)
KeyboardInterrupt
Task was destroyed but it is pending!
task: <Task pending coro=<ActionQuery.execute.<locals>.execute_commands() running at /home/josephhenry/Documents/artfx/pipeline/silex/git/silex_client/silex_client/action/action_query.py:84> wait_for=<Future pending cb=[ActionQuery.async_update_websocket.<locals>.apply_update() at /home/josephhenry/Documents/artfx/pipeline/silex/git/silex_client/silex_client/action/action_query.py:206, <TaskWakeupMethWrapper object at 0x7f03819bc750>()]> cb=[_chain_future.<locals>._call_set_state() at /home/josephhenry/packages/python/3.7.9/platform-linux/arch-x86_64/os-Pop-21.04/python/lib/asyncio/futures.py:358]>

Move the execution of the actions in the event loop

Move the Context class in a "core" module
Create an EvenLoop class tah all the asyncio tasks can register to
Create a action executor task
Allow other packages to customise the executor task

Add the ability to connect the output of a command to the parameter input of an other

Find a syntax for the yaml config that allow to set the path of a command
Before the execution of a command, set the parameters value to the output of the specified command
When a parameter is linked to the output of an other command, set it to hide: True to avoid showing it to the UI

Refactor execute_in_thread

turn in into callable class
inherit from it in maya and houdini

Refactor the merge functions with jsondiff

Replace the use of the merge function with the jsondiff library

Create utils category for commands

create utils folder in commands
update yaml with new paths

Agree on environnement variables

Right now the environnement variables have différents names on the repos, WE should make them consistant

Implement socketio connection

Etablish a connection between the silex client and the websocket server

Initialize connection on startup
Implement the action query round trip
Synchronize the context between the server and the client

Add WS route to get the running actions

If the Socket server is disconnected while actions are running, it loses them when reconnecting and so the front-end.

Between the UI and SSS we have: /ui/getRunningActions which returns a dictionary of actions:

getRunningActions: WithCallback<
  ServerResponseWithData<{ [uuid: string]: Action }>
>;

then when the UI starts up, it sends that query to get the current running actions. In this case the either silex_client should send the actions to the SSS when reconnecting (we can use the reconnection event) or the other way around?

How to get the running actions?

I am trying to solve #96 and I want to get the current running actions in the context.

I am trying to use:

silex_client/silex_client/core/context.py

Line 48 in 921d1b0

self.running_actions: Dict[str, ActionQuery] = {}

But it's not used in any code, is it up to date?

The submit could be used either in a context or out of a context

Replace the use of rez ephemerals with custom script

Add gazu authentification
Guess the context from the task ID
Add a CLI tool to launch a dcc from a custom environment

Paramteters types for commands

separator / title type for organizing large amount of parameters
range for choosing start and end frame when exporting

Better error message when giving an action without the category

When trying to execute an action without the -c category argument:

$ rez env silex_client-dev -- silex action tester
[SILEX]    [211222 15:40:15] ERROR     |    [config.resolve_config] Could not resolve the action tester: The action does not exists (106)
Traceback (most recent call last):
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/cli/parser.py", line 140, in <module>
    main()
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/cli/parser.py", line 136, in main
    HANDLERS_MAPPING[subcommand](**args)
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/cli/handlers.py", line 48, in action_handler
    if not action.commands:
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/action/action_query.py", line 314, in commands
    return self.buffer.commands
AttributeError: 'ActionQuery' object has no attribute 'buffer'

Display a better message when the log level is not correct

Currently if we provide an incorrect log level (for logzero), there's an error:

$ SILEX_LOG_LEVEL="info" rez-env silex_client-dev -- silex action submit
Traceback (most recent call last):
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/cli/parser.py", line 9, in <module>
    from silex_client.cli import handlers
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/cli/handlers.py", line 15, in <module>
    from silex_client.utils.log import logger
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/utils/log.py", line 31, in <module>
    logger.setLevel(log_level)  # set default level
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/logging/__init__.py", line 1353, in setLevel
    self.level = _checkLevel(level)
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/logging/__init__.py", line 195, in _checkLevel
    raise TypeError("Level not an integer or a valid string: %r" % level)
TypeError: Level not an integer or a valid string: <function info at 0x7fde53894290>

We should either transform the variable to upper case or show a more descriptive error with logs.

Rez pip install package errors and warnings on Linux

Using Rez 2.100.0 and commit 1fc378b trying to install the package locally:

$ rez pip -i .

Everything is working well but I have the warnings:

21:49:20 WARNING  Metadata: missing: ['Author'], warnings: []
21:49:20 WARNING  Metadata: missing: ['Home-page', 'Author'], warnings: []
21:49:21 WARNING  Ignoring extras requested on 'python-socketio[client]==5.4.0' - this is not yet supported

And this error:

Traceback (most recent call last):
  File "/home/josephhenry/rez/bin/rez/rez", line 8, in <module>
    sys.exit(run_rez())
  File "/home/josephhenry/rez/lib/python3.7/site-packages/rez/cli/_entry_points.py", line 80, in run_rez
    return run()
  File "/home/josephhenry/rez/lib/python3.7/site-packages/rez/cli/_main.py", line 202, in run
    returncode = run_cmd()
  File "/home/josephhenry/rez/lib/python3.7/site-packages/rez/cli/_main.py", line 194, in run_cmd
    return func(opts, opts.parser, extra_arg_groups)
  File "/home/josephhenry/rez/lib/python3.7/site-packages/rez/cli/pip.py", line 84, in command
    extra_args=opts.extra)
  File "/home/josephhenry/rez/lib/python3.7/site-packages/rez/pip.py", line 357, in pip_install_package
    name_casings=dist_names
  File "/home/josephhenry/rez/lib/python3.7/site-packages/rez/utils/pip.py", line 396, in get_rez_requirements
    is_pure_python = is_pure_python_package(installed_dist)
  File "/home/josephhenry/rez/lib/python3.7/site-packages/rez/utils/pip.py", line 338, in is_pure_python_package
    wheel_data = setuptools_dist.get_metadata('WHEEL')
AttributeError: 'NoneType' object has no attribute 'get_metadata'

The error might be due to the missing tractor package but we need to make a package out of this.

Then if I try to run the env with silex_client, I get the following:

$ rez env silex_client
21:57:30 ERROR    PackageFamilyNotFoundError: package family not found: typing_extensions, was required by: aiohttp (searched: /home/josephhenry/packages; /home/josephhenry/.rez/packages/int; /home/josephhenry/.rez/packages/ext)

Finally trying to install typing_extensions with rez-pip gives me an error:

=> Here is the related issue: AcademySoftwareFoundation/rez#1159 (comment)

I am running Python 3.7.9 on Linux pop-os 5.13.0-7620-generic

Show right pattern in tractor tasks

Update the README

Explain the process to setup the development environment
Explain how to use the CLI

Create a parameter dataclass

Replace the parameter dictionary with a dataclass for safer data checking

Create the concept of ActionSteps

Handle KeyboardInterrupt exceptions

Avoid error message when pressing CTRL+C

Handle unable to get tractor config in Submitter

Send a clear message that the tractor in unreachable

authentification for database comunication

Get the authentification token from the websocket server on initialisation
authentificate with rest systeme (home made or lib)
Implement a context resolver, that guesses the shot, asset, sequence... according to a task ID

Authentification token : /auth on token
Returned data :
{
"data":
}

Return feedback on the GetReferences command

Print a resume on the references found

KeyboardInterrupt on Ctrl+C not handled on Linux

I am running the following action on Linux with rez:

$ rez-env silex_client-dev -- silex action submit

and when I hit Ctrl+C in the terminal I get:

^CInterrupted by user
Traceback (most recent call last):
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/cli/parser.py", line 140, in <module>
    main()
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/cli/parser.py", line 136, in main
    HANDLERS_MAPPING[subcommand](**args)
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/cli/handlers.py", line 76, in action_handler
    futures.wait([silex_context.ws_connection.stop()], timeout=None)
  File "/home/josephhenry/rez_packages/silex/silex-rez/packages/silex/silex_client/dev.0.1.0/silex_client/network/websocket.py", line 91, in stop
    futures.wait([future])
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/concurrent/futures/_base.py", line 301, in wait
    waiter.event.wait(timeout)
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/threading.py", line 552, in wait
    signaled = self._cond.wait(timeout)
  File "/home/josephhenry/rez_packages/python/3.7.9/platform-linux/arch-64bit/os-Pop-21.10/python/lib/threading.py", line 296, in wait
    waiter.acquire()
  File "/home/josephhenry/rez_packages/python_engineio/4.3.0/88a70aca30cb79a278872594adf043dc6c40af99/python/engineio/client.py", line 36, in signal_handler
    return original_signal_handler(sig, frame)
  File "/home/josephhenry/rez_packages/python_socketio/5.4.0/88a70aca30cb79a278872594adf043dc6c40af99/python/socketio/client.py", line 26, in signal_handler
    return original_signal_handler(sig, frame)
KeyboardInterrupt