timokoerber / laravel-one-time-operations Goto Github PK

Congrats on being featured in Laravel News! I've been trying to solve this problem for a while now, so my interested in this package was instantly piqued.

I've perused the source and don't see any sign of locking so that on deployment to multiple servers, the one-time process commands don't step on each other.

My current set up is a Kubernetes deployment with a few web server containers, and a few dedicated task worker containers. Since they're scalable (replicable) I can't configure individual instances to run different commands, so they've all got to do the same exact thing on deployment. (There are other "one-time" possibilities with Kubernetes, but I won't go into detail about my tribulations there. It's been frustrating to say the least.)

Laravel doesn't have any way (that I'm aware of) to run prevent running multiple instances of artisan migrate from stepping on each other either, so it's been a problem I'm trying to solve there as well. This package could be leveraged to run migrations, as strange as that sounds, so a feature enhancement here could solve that problem too.

Is there functionality like this that I'm missing, or any plans for something like this in the future?

Great job on this package! I have a suggestion for a feature for you to consider.
Operations seem to fit the scenario of data migrations. In most cases where I've handled tasks like that, I always try to add some sort of logging capability that shows progress and results. For example, if we're processing a large list of items filtered by some rule I like to output the result for each item processed.
Perhaps we could add a way to do that i.e. have some logging facility available inside an operation that then writes results in the DB table.

Hello.

This package is great, but today I run into small problem.

When running single operation with name provided
sail artisan operations:process 2023_11_10_170704_add_products_to_b2b_group.php

I got an error:
ERROR File 2023_11_10_170704_add_products_to_b2b_grou.php does not exist.

Note that file name in error message is missing last "p".

The problem most probably exists in /src/Commands/OneTimeOperationsProcessCommand.php on line 63 with rtrim function:

protected function proccessSingleOperation(string $providedOperationName): int
    {
        $providedOperationName = str($providedOperationName)->rtrim('.php')->toString();

I've found similar issue on stackoverflow: https://stackoverflow.com/questions/19473603/php-rtrim-php suggesting that we can use preg_replace("/(.+)\.php$/", "$1", $filename); or basename($filename, '.php') functions.

Dear Laravel Library Developer,

I hope this message finds you well. I am reaching out to request an additional feature in the One-Time Operations for Laravel package. Currently, all jobs are sent to the default queue by the library. However, it would be extremely useful if there was an option to specify the queue name for each job.

Therefore, I kindly request that you consider adding a "Queue Name" option to the library, which would allow users to specify the name of the queue that the job should be sent to.

Thank you for your time and consideration. I look forward to hearing back from you.

Best regards, fh32000

Why you have migrations and factory in completly different folders like in model in this case why this is not stored on same level like laravel have migrations/seeds/factories ?

Thank you for the effort for this package.

But I keep asking how this can be better then composer hooks in work with laravel commands...

Are there plans to bring support for Laravel 11?

It's currently not possible to override the stubs. I would expect that this would work the same as with the default Laravel stubs, by adding the already existing stub files into /stubs/... and make the adjustments as we like.

Could this be added :)?

I am reaching out to suggest an enhancement for the operations jobs and queue system, which presently relies on defining jobs as a string property. Transitioning to defining tags as an array could significantly improve the segregation of queues. Although the current system is flexible and efficient in managing tasks, I see an opportunity for enhancement.

Feature Proposal:

I propose the introduction of method-like tags that return an array of strings, similar to how jobs are handled in Laravel. This improvement would offer a more structured and intuitive approach to managing jobs and queues within Laravel applications. By implementing method tags, developers would achieve clearer insights into job responsibilities and invocation patterns, which in turn would enhance code readability and maintainability.

Benefits:

• Enhanced Readability: Utilizing array tags provides a more descriptive method for outlining each job's purpose, thereby simplifying the understanding of the codebase.
• Improved Maintainability: Array tagging facilitates easier updates and maintenance of job definitions by offering a clear view of the job's functionality.
• Increased Flexibility: This feature would enable developers to handle their queues and jobs in a more refined and adaptable way, easily adjusting to complex workflows.

I am convinced that incorporating this feature would significantly benefit the package, empowering developers to create more robust and efficient applications. I am keen to hear your views on this proposal and to discuss any potential considerations for its implementation.

Thank you for taking the time to consider my suggestion. I look forward to your feedback.

Hi,

Awesome package, i've been using migrations to run tasks that are part of deployment this is much better.
In some cases when deploying you need to run some logic before migrations are run and in some other cases after migrations has run.

Assuming the following is my deployment script, i will only be able to run the operations before the migrations but not after

php artisan operations:process
php artisan migrate

When laravel migrations are run two events are triggered MigrationsStarted and MigrationsEnded, so in this case we can use the MigrationsEnded event to run operations only after the migrations finished running.

If you are open to this it could be something like adding a $runAfterMigrations property that defaults to false on the OneTimeOperation class

If you are ok with this i can make a PR

I was really happy when I saw this package, but it did not really work.

I needed to add a user related folder to all users so for this to happen after a deployment this package seems perfect, but... the operation was executed, but nothing was created.

Here is the operation:

<?php

use TimoKoerber\LaravelOneTimeOperations\OneTimeOperation;

return new class extends OneTimeOperation
{
    /**
     * Determine if the operation is being processed asyncronously.
     */
    protected bool $async = true;

    /**
     * The queue that the job will be dispatched to.
     */
    protected string $queue = 'default';

    /**
     * A tag name, that this operation can be filtered by.
     */
    protected ?string $tag = null;

    /**
     * Process the operation.
     */
    public function process(): void
    {
        $users = \App\Models\User::all();
        foreach ($users as $user) {
            $user->makeUserDirectories();
        }
    }
};

here is the makeUserDirectories method

public function makeUserDirectories(): void
{
    // More entries, but the one below is the new
    Storage::disk('xxx')->makeDirectory('/zzzz/yyyy/xxx');
   
}

Execution:

php artisan operations:process           
INFO  Processing operations.  
2023_05_20_082512_fix_user_dirs ................................................................................. 7ms DONE
INFO  Processing finished.  

But the new directory was not created.

running the same code from a controller works

Sometimes I want to run an operation as part of a migration (before, during or after a specific migration). This package only supports running operations after all migrations are run, or before all migrations are run.

An example would be if I want to change the way data is stored, it needs to be an intermediate step to migrate the data, otherwise old data is lost.

There are a lot of instances of syncronously and asyncronously in this repo. They should be synchronously and asynchronously.

I know it's not a big deal, but my IDE is highlighting them as spelling mistakes.

It would be great to allow for the tracking of the "status" of an operation.

Currently, the only thing tracked about an Operation is its name, how it was dispatched, and when it was dispatched. It would be extremely useful to also track whether or not the operation job was successful or not.

A possible way would be to add a last_status column or something like that to the operations table. Then hook into Queue::after in the package Service Provider to update on whether or not the job finished with or without a failure.

Hi, I have an issue with this package on production, sometimes I got this error:
TimoKoerber\LaravelOneTimeOperations\OneTimeOperationManager::getDirectoryName(): Return value must be of type string, null returned
Note: configs are cached then execute php artisan operations:process
Anyone got this issue?
Thanks for your help!

I might drop a PR for this eventually, but it's something we'd love - the ability to drop a chain / batch of jobs or operations that would ALL need to run for the operation to be complete.

The use case:
We found that migrating 500k rows was timing out on a single job, and i don't like to have a single queue that has unlimited or a super-long timeout as it can cause problems with stuck jobs, so being able to create a "batch operation" would be cool as heck, even if it just was as simple as implemented a BatchOperation interface that defined a "singleOperation" function or similar.

Then those could be dispatched as callables on the queue using a custom job, OR dispatched as closures.

Then that would allow you to specify a "final" job that maybe does some tidying up with a "finally()" method or similar.

All defined within one file similar to how it is now, but gives more flexibility.

Will have a look at a PR if i can get some time IF you like the idea?

Howdy!

I am very interested in using this package but there are a few tweaks to the way it works that would make it perfect for my use-case. I believe a couple of changes to the way this is set up could allow folks like me to make those tweaks, but not break anything how it currently is.

If you want to let me know which of any of these you would be interested in, if any. I can open some PRs if you would like.

Non-Breaking

• Dispatch Events in key areas
  • Firing Events in some key areas is an easy way to allow folks to hook in with a listener to respond to certain points in the operations process
  • Some examples I can think of are:
    • When OneTimeOperationsProcessCommand starts
    • When OneTimeOperationsProcessCommand finishes
    • When OneTimeOperationProcessJob starts
    • If OneTimeOperationProcessJob succeeds
    • If OneTimeOperationProcessJob failed
  • This is perfect to help with CI/CD to do things like send Slack or email notifications if an operation fails, etc.

Mostly Non-Breaking

• Allow package users to provide their own classes
  • By making use of the Service Container and/or the config file for the package we could allow package users to override certain classes like the OneTimeOperationProcessJob for example
  • I believe we should be able to use the Service Container to resolve to the current classes, and give package users the ability to to re-bind in their app Service Providers. Which would make this a complete non-breaking change. But if not possible once dug into, another good route is defining which classes to use in the config, setting the current ones as the package defaults. This would be slightly "breaking" as it would require people to update their config file if they had published it already.

It might be useful when working with models with a large amount of records, the timeout of the default queue is 60 seconds, which might be not enough to process all the models.

I know its possible to change the queue with different timeout values, but it would be nice to do it with the default queue, instead of changing the queue configuration.

Would be nice to be able to add a delay on the queue,

Hi,

I have an enhancement suggestion.
Allow loading operations from multiple directories just like laravel migrations.
This can be done via the config file directory key maybe renamed to directories with a fallback to directory to avoid breaking change.
Also you could add a method called loadOperationsFrom() the same way laravel provides loadMigrationsFrom()

Personally for my use case, i have two places where i can load laravel migrations from.