Hey,

I'm just testing around with parallel cronjobs and have some weird behaviour. My crontab looks like this:

* * * * * /bin/sh /var/www/casamoda/casamodab2c/project/root/cron.sh cron.php -malways 1
* * * * * /usr/bin/env SCHEDULER_BLACKLIST='long_running_process' /bin/sh /var/www/magento/root/cron.sh cron.php -mdefault 1
* * * * * /usr/bin/env SCHEDULER_WHITELIST='long_running_process' /bin/sh /var/www/magento/root/cron.sh cron-slow.php -mdefault 1

Now, when I observe the running Processes, the cron.php executes the long_running_process, and the cron-slow.php executes all other processes.

I looked into your code and really can't explain myself, why this happens.

It's not a Problem with 2 cron processes, but it becomes one if you want to have more.

Maybe you have an idea?

Currently you are using substr to parse the error message. I think that this is easy to understand but does not enforce a good programming practice, as the type of the message is set implicitly via the message string. You could instead provide a constant or some sort of interfaces to set this type of message. This would also allow developers to use the Mage_Cron_Model_Schedule::STATUS_.*.

    if (strtoupper(substr($messages, 0, 6)) != 'ERROR:') {
            $this->setStatus(Mage_Cron_Model_Schedule::STATUS_SUCCESS);
    } else {
            $this->setStatus(Mage_Cron_Model_Schedule::STATUS_ERROR);
    }

Task could run in parallel (and in separation) by running in them in a separate fork (e.g. using Aoe_Threadi)

hey,

I would like to implement a new Event that is dispatched, when a cronjob is run by "run now" or "schedule now".

The Reason is, that we would like to get an email notification, when a cron job is run manually (in case it happened accidently)

I would implement the email notification in an own module, except you wish to have it in the scheduler, then I would make a pull Request, where you will be able to activate that feature in the system configuration.

Regards,
David

A job could have a flag that hides it by default from the timeline view.
The timeline view could have a button "Show hidden"

Tasks executed with a high frequency will quickly fill up the database - especially if you configured cron to keep 12h or 24h of tasks.
In most cases your not so much interested in seeing hundreds of tasks executed every 5 minutes, but you still want to keep to long timeframe to check if the nightly import worked successfully.
Feature: add a new configuration to manage how many successful tasks should be kept.

Since we upgraded to ffc09f1, some jobs fail as "gone" and are executed and completed two minutes later. Before the process ids were recorded the jobs were just marked complete at the time they were scheduled first. Do you have a guess if this happens because of the way your check is implemented or if this is a server / project issue?

Information on our setup:

• EE 1.13.0.2
• Aoe_Scheduler @ d17bd8a (last version without this problem: 2b853e1)

Cron schedule (filenames and pathes changed):

* * * * * /usr/bin/env SCHEDULER_WHITELIST='a,b,c,d' /bin/sh /path/to/magento/cron_1.sh cron_1.php -m=default  >> /path/to/log/scron.log 2>&1
* * * * * /usr/bin/env SCHEDULER_BLACKLIST='a,b,c,d' /bin/sh /path/to/magento/cron_2.sh cron.php -m=default  >> /path/to/logs/cron.log 2>&1

The timeline looks like that:

The failed job:

The message is:

Process "9227" on host "###" cannot be found anymore

The successful job:

First of all thanks for this great Extension.
I'm currently running into the "too late for schedule" issue. I have set the cron to be executed every minute. But it seems that I have some long running jobs every night. I tried to analyze this through using the timeline view. But unfortunately it's empty / shows nothing in the div. I have attached a screenshot, so that you can see what I'm talking about :)
I was also looking if some elements couldn't be loaded in Firebug, but there are no errors displayed.
Do you need any additional information?

I've noticed that the heartbeat/cron in not running if APC is enabled. and to run the cron i need to disaple APC. is this a known issue?

Allow configuring an expected user (e.g. "www-data") and
a) warn if the current user (via web or cli) doesn't match that configuration
or b) actually prevent the job from running with an error message
I guess the behavior (warning or fail) needs to be configurable.

Hello Fabrizio,

I have recently moved to another hosting provider and now from time to time getting this report:

a:4:{i:0;s:63:"SQLSTATE[HY000]: General error: 2006 MySQL server has gone away";i:1;s:1306:"#0 /var/www/vhosts/site/store/lib/Zend/Db/Adapter/Pdo/Abstract.php(305): PDO->beginTransaction()
#1 /var/www/vhosts/site/store/lib/Zend/Db/Adapter/Abstract.php(495): Zend_Db_Adapter_Pdo_Abstract->_beginTransaction()
#2 /var/www/vhosts/site/store/lib/Varien/Db/Adapter/Pdo/Mysql.php(219): Zend_Db_Adapter_Abstract->beginTransaction()
#3 /var/www/vhosts/site/store/app/code/core/Mage/Core/Model/Resource/Abstract.php(76): Varien_Db_Adapter_Pdo_Mysql->beginTransaction()
#4 /var/www/vhosts/site/store/app/code/core/Mage/Core/Model/Abstract.php(313): Mage_Core_Model_Resource_Abstract->beginTransaction()
#5 /var/www/vhosts/site/store/app/code/local/Aoe/Scheduler/Model/Observer.php(98): Mage_Core_Model_Abstract->save()
#6 /var/www/vhosts/site/store/app/code/core/Mage/Core/Model/App.php(1338): Aoe_Scheduler_Model_Observer->dispatch(Object(Varien_Event_Observer))
#7 /var/www/vhosts/site/store/app/code/core/Mage/Core/Model/App.php(1317): Mage_Core_Model_App->_callObserverMethod(Object(Aoe_Scheduler_Model_Observer), 'dispatch', Object(Varien_Event_Observer))
#8 /var/www/vhosts/site/store/app/Mage.php(447): Mage_Core_Model_App->dispatchEvent('default', Array)
#9 /var/www/vhosts/site/store/cron.php(46): Mage::dispatchEvent('default')
#10 {main}";s:11:"script_name";s:42:"/var/www/vhosts/site/store/index.php";s:4:"skin";s:5:"admin";}

Can you advise what might be wrong with it and a setting to look into?

Thank you

A user reported: (probably the schedule table is full and the timeline would be huge...)

I had this issue installed and running fine but now every time I go to Timeline View I get the following error:

Fatal error: Out of memory (allocated 272891904) (tried to allocate 83 bytes)

Have tried uninstalling/reinstalling, increased the php memory limit to 128M but no luck. Any ideas what I can try?

Currently only "scheduleNow" is supported, but it would be nice to also have "scheduleAt" or "scheduleIn" (in cli, web,...)

Currently the disable status is stored in a csl in the system configuration. This should be removed altogether since now this can be stored with a job overlay.
A data script should migrate this and the massactions also need to be updated.

Hey, I have been using this tool recently and it's pretty helpful for running cron's manually and debugging them. One thing I was thinking that would be nice to have, is some kind of option to tell my cron model to output debug information when manually run.

Imagine I have a long export process being triggered by a cron, now I won't want console output because it is being run on the server by the cron user.

If I am running the command manually through Aoe_Scheduler I want to output debug information such as the progress of the export.

How might you implement this feature?

I was thinking we could maybe just set a super global in the runNow function and I could just check that in my cron models, however that is not an optimal solution.

Any ideas?

If this is something you would be interested in and we agree on an implementation, i'll create a pull request :)

For now we have SQL query in Aoe_Scheduler_Model_Observer::generate() function. We need move it to resource model.
https://github.com/fbrnc/Aoe_Scheduler/blob/master/app/code/community/Aoe/Scheduler/Model/Observer.php#L188

This is just a documentation issue: symlinking cron.php to cron2.php will not work unless you are on Windows or have shell_exec disabled, due to this code in cron.php where the script spawns itself twice in "always" and "default" mode:

        } else if (!$isShellDisabled) {
            $fileName = basename(__FILE__);
            $baseDir = dirname(__FILE__);
            shell_exec("/bin/sh $baseDir/cron.sh $fileName -mdefault 1 > /dev/null 2>&1 &");
            shell_exec("/bin/sh $baseDir/cron.sh $fileName -malways 1 > /dev/null 2>&1 &");
            exit;
        }

Solution: Copy cron.php instead of symlinking to have two physically different files.

When users install an extension that comes with a scheduler task and uninstall it the xml configuration might still be there (or in the cache) resulting Aoe_Scheduler to throw an exception.

This exception should be catched and displayed as a meaningful warning instead.

$aoe = $.noConflict();
$aoe(function() {
$aoe(’.timeline-box’).scrollLeft($aoe(’.timeline-panel’).width());
... etc…

Some users are complaining that the see a blank page when clicking the scheduler menu items. This is because they installed the extension via modman and they did not change the configuration in Magento to allow symlinks for templates. Magento will then not load the needed blocks to render the pages.

A hint should be added to the readme file.

In Settings > System > Cron theres a typo:

scehduler should be "scheduler"

Aoe Scheduler complains of the above message despite having valid configuration for the above cronjob.

Cron error while executing enterprise_import_export_log_clean:

exception 'Mage_Core_Exception' with message 'No valid configuration found.' in magento/app/Mage.php:594
Stack trace:
#0 magento/app/code/community/Aoe/Scheduler/Model/Observer.php(36): Mage::throwException('No valid config...')
#1 magento/app/code/core/Mage/Core/Model/App.php(1338): Aoe_Scheduler_Model_Observer->dispatch(Object(Varien_Event_Observer))
#2 magento/app/code/core/Mage/Core/Model/App.php(1317): Mage_Core_Model_App->_callObserverMethod(Object(Aoe_Scheduler_Model_Observer), 'dispatch', Object(Varien_Event_Observer))
#3 magento/app/Mage.php(447): Mage_Core_Model_App->dispatchEvent('default', Array)
#4 magento/cron.php(73): Mage::dispatchEvent('default')
#5 {main}

Example:

• how many records to keep
• notifiy on success/failure

Hi,

from time to time (about every 4 weeks) a job hangs on status "Running".
It was never executed and thus has no executed, finished or last seen timestamp as well as no pid and host:

(Btw: the screenshot is particularly interesting as the job was created two hours in the future, see the Created at timestamp. This never has happened before, neither with this job "type" nor with any else. Now is the first time. We had hanging jobs with the "Running" status and correct timestamp before. Also, other jobs generated at the same time had the correct timestamp.)

The job doesn't get marked as failed, missed or something similar bei the cleanup procedures and thus blocks further executions of this job "type". There are no errors in the logs / reports.

We're using commit ddd9daf. The schedule is generated dynamically by a third-party extension by inserting XML nodes in the Magento XML config tree on-the-fly.

Please don't hesitate to ask if you need more details.

If a fatal error happens (e.g because of a syntax error) in the cron job class then we have a
running cron job without pid which is actually already dead, but Aoe_Scheduler doesn't know.

timeline view displays a blank page in admin when deploying with modman (symlinks).
copying the files to the appropriate location works.

When uninstalling the extension and the data in core_config_data may contain the cron job for nonexisting model.

A job should have an ETA
1.) determined by the history of the last jobs (Jenkins style)(
2.) The job updates itself with an ETA.
This ETA should be visualized in the timeline view.

If there are a lot of jobs to be executed, the scheduler tries to run multiple jobs in the same run. It it maybe possible to configure Aoe_Scheduler so that it only runs multiple jobs if the memory usage is less than say 500MB? If it exceeds that threshold it wont start a second job.

Allow configuring notifications

• on success
• on failure
• always
  Handlers
• email
• log entry
• admin notification (Magento backend)

Cron job scheduling and generation worked perfectly until we upgraded Aoe_Scheduler to the newest version (so that we can use the updates from July 18th and 19th).

In the last days we experienced that new jobs are not scheduled for 20 to 30 minutes. We don't get errors in the logs or high load on the server. It's not at the same time of the day.

It could be that the hiccup starts when/after the history cleanup runs. (Btw. the history cleanup runs twice because we have to cron groups set up).

Information on our setup:

• EE 1.13.0.2
• Aoe_Scheduler @ ffc09f1 (last version used before: 2b853e1)

Cron schedule (filenames and pathes changed):

* * * * * /usr/bin/env SCHEDULER_WHITELIST='a,b,c,d' /bin/sh /path/to/magento/cron_1.sh cron_1.php -m=default  >> /path/to/log/scron.log 2>&1
* * * * * /usr/bin/env SCHEDULER_BLACKLIST='a,b,c,d' /bin/sh /path/to/magento/cron_2.sh cron.php -m=default  >> /path/to/logs/cron.log 2>&1

The timeline looks like that:

That manic job at the top is enterprise_refresh_index.

There is no hint why the generation stops or starts again.

Our configuration:

We do have a long lifetime but we use that setting for a long time now. I thought about setting higher values for "Schedule Ahead for" and "History Cleanup Every" to see if this makes a difference.

If this problem persists we'll downgrade and check if it also happens with the old version.

Any ideas? Thanks!

Is it under the same AFL as Magento Community?

This issue was posted by Christian Haug to
http://www.fabrizio-branca.de/magento-cron-scheduler.html#comment-412503450

"in der Datei Schedule.php, Line 53 sollte noch $this mit an den callback übergeben werden, da manche callbacks das Schedule model als Parameter drin stehen haben."

Hi,
A Big Thank You for sharing such a useful module for Magento. I needed to run a cron job (and inspect all the jobs as well), so I found a good place to add my code would be your heartbeatTask::run() method. I am not sure what is going on, but this is the screenshot of sheduler timeline. (Click on image to see it better).

Here is my code (just in case, my code is causing it)

class Aoe_Scheduler_Model_HeartbeatTask {

    public function run() {

      date_default_timezone_set('Asia/Kolkata');
        $out = date('d.m.y h:i:s A')."\n";
        $old_time = time() - (30*24*60*60);//10min before timestamp
        $orders = Mage::getModel('sales/order')->getCollection()
          ->addFieldToFilter('status', 'pending')
          ->addFieldToFilter('cod_fee', array('null' => true))
          ->addFieldToFilter('created_at', array('lt' => $old_time))
          ->addAttributeToSelect('customer_email')
          ->addAttributeToSelect('created_at')
          ;

        file_put_contents('/home/vinayak/newcron6.txt', $out, FILE_APPEND);
      foreach ($orders as $order) {
        $out .= $order->getCustomerEmail()." @ ".$order->getCreatedAt()."\n";
//          $order->setState('pending_payment', true)->save();
        }
        file_put_contents('/home/vinayak/cron.txt', $out, FILE_APPEND);
        return true;
    }

}

BTW My code is not working correctly.
The executed at time for one of the job is "16:00:02" and finish at "21:30:02". This longish schedule turns up randomly.

Might it be because I am living in GMT+5:30 timezone?
Any insights would be helpful
Thanks!

Hi,

We have Varnish running with support by Nexcess Turpentine module.
However cannot make its crawler running - cron throws errors saying:'
Cron error while executing turpentine_crawl_urls:

exception 'Mage_Code_Exception' with message 'Too later for the schedule.' in .../app/Mage.php:595
etc.

Thank you!

Has anybody else experienced issues with certain processes running indefinitely? These processes were created in the future, scheduled in the past, and then executed in the future.. There's also several tasks buried within the running tasks that were missed or pending.
In addition I'm thinking the way AOE is displaying this is a bug of some sort, doesn't seem right.

Specs:
Magento 1.12.0.2 Enterprise
Latest version of AOE

If a job needs to be killed, but the job isn't running on the current server a "kill request" will be saved instead. Problem is that the server running that job will never see that kill request if there's only a single cron configured on that server.
We'd need a different mechanism (an a separate cron?) that checks and actively processes kill requests.

This feature was suggested by Matthias Zeis:

Allow adding a description text via XML (and via records) to every cron and show that information in the list view (and maybe in the title tag of the timeline view).

<crontab>
    <jobs>
        <aoe_queue>
            <description>Processes generic queue</description>
            <schedule>
                <config_path>system/aoe_queue/cron_expr</config_path>
            </schedule>
            <run>
                <model>aoe_queue/cron::processQueue</model>
            </run>
        </aoe_queue>
    </jobs>
</crontab>

This issue was original posted by Dan Strickland to:
http://www.fabrizio-branca.de/magento-cron-scheduler.html#comment-391020743

Hi! I can't seem to assign AOE Scheduler w/datails in my Permissions>Roles>Resources>Custom - It works fine in Web Services for API but have a user with no API access and needs standard Admin user/permission - I need to grant him access to the AOE scheduler but I only see "Scheduler" in Resources. When the user log's in, they only see System>Scheduler - but no details? I'm on CE 1.6.1. Yes, I have your lates 0.2.1 update as well.

User reported:

Hello
I’m interested b this extension but need some understanding to add a cron job in the list
Thanks

Readme should contain info on where to start when adding a new cronjob.
And how to schedule an existing cronjob.

A job could come with some extra configuration about what other task(s) to execute next

• on success
• on failure
• always.

This can already be done via the events that are being triggered, but it would be nice native support for this.

There used to be a Run Now action under the Schedule Configuration grid. What happend to it?

This is a strange one. I don't think that I saw it happen until I manually disable a cron using the scheduler interface, then re-enabled it.

After doing that, and then trying to generate a schedule for my cron task having a schedule of */15 * * * *, it wouldn't generate.

I tried several different options (*/10, */6, */12), and only */5 worked.

However, if I cleared the cron_schedule table manually and ran php cron.php manually to generate the schedule, my original */15 worked fine.

Seen that before?

If scheduler is configured to keep many hours of history the timeline can grow big (and will take a while to load). In this case the timeline could only show a slice of 6h? and provide buttons to move to an earlier or later "slice".

Note: How to deal with overlapping jobs at the borders?

A lot of users contact me saying that the scheduler isn't working. The main reason is that cron.sh was not added to the crontab. Or cron.sh is only called once an hour or even once a day.

This should be checked and a message should be displayed in the backend if a misconfiguration is detected.

Problem 1) For every option belonging to the Aoe menu I only get a text stating:
Scheduler is working. (Last execution: 0 minute(s) ago)

I can access the config, cant access the timeline etc. So when the hearthbeat is too old: I cant access any options, and when it is youger than 5 minutes I cant access any options

Problem 2) Also we see many messages in th elogs with "exception 'Mage_Core_Exception' with message 'Too late for the schedule.'"

(also mentioned on your site)

Many thanks!!

Hi,

we would like to use the parallel environment descriped here: https://github.com/AOEpeople/Aoe_Scheduler#running-multiple-scheduler-jobs-in-parallel

But it don't work.

Here is my own cron_multithread.sh:

#!/bin/bash

pathToMagentoRoot="`cd $(dirname $0)/.. && pwd -P`"
fileCronSh="${pathToMagentoRoot}/cron.sh"
## php files for each cron
cronExport='cron_export.php'
cronImport='cron_import.php'
cronDefault='cron.php'

## nice value for each cron
## -20 high - low 19
cronExportNiceValue='2'
cronImportNiceValue='2'
cronDefaultNiceValue='-1'

exportProcessList=(
    job1
    job2
)

importProcessList=(
    job3
    job4
)

####################################################
####################################################

## check whether the cron php derivates exist
if [ ! -e ${pathToMagentoRoot}/${cronExport} ]; then
    ln -s ${pathToMagentoRoot}/${cronDefault} ${pathToMagentoRoot}/${cronExport}
    echo "[INFO] cron file '${pathToMagentoRoot}/${cronExport} -> ${pathToMagentoRoot}/${cronDefault}' created"
fi

if [ ! -e ${pathToMagentoRoot}/${cronImport} ]; then
    ln -s ${pathToMagentoRoot}/${cronDefault} ${pathToMagentoRoot}/${cronImport}
    echo "[INFO] cron file '${pathToMagentoRoot}/${cronImport} -> ${pathToMagentoRoot}/${cronDefault}' created"
fi


####################################################
####################################################

whitelist=""
blacklist=""

case "${1,,}" in
    ###################
    ## EXPORT THREAD ##
    ###################
    "export")
        for process in ${exportProcessList[@]}; do
            whitelist="${process}, ${whitelist}"
        done
        settings="SCHEDULER_WHITELIST='${whitelist%??}'"

        cronFile=${cronExport}
        ;;


    ###################
    ## IMPORT THREAD ##
    ###################
    "import")
        for process in ${importProcessList[@]}; do
            whitelist="${process}, ${whitelist}"
        done
        settings="SCHEDULER_WHITELIST='${whitelist%??}'"

        cronFile=${cronImport}
        ;;


    ####################
    ## DEFAULT THREAD ##
    ####################
    *)
        for process in ${exportProcessList[@]}; do
            blacklist="${process}, ${blacklist}"
        done
        for process in ${importProcessList[@]}; do
            blacklist="${process}, ${blacklist}"
        done
        settings="SCHEDULER_BLACKLIST='${blacklist%??}'"

        cronFile=${cronDefault}
        ;;
esac

## cron call
echo /usr/bin/env ${settings} /bin/sh ${fileCronSh} ${cronFile} -mdefault 1

But always each thread would call all scheduled jobs. It ignores the blacklist and whitelist.

Here are my system cron calles

*   *   *   *   *   cd /MAGENTO/ROOT/shell && /bin/bash cron_multithread.sh export >> ../var/log/cron_export.log 2>&1
*   *   *   *   *   cd /MAGENTO/ROOT/shell && /bin/bash cron_multithread.sh import >> ../var/log/cron_import.log 2>&1
*   *   *   *   *   cd /MAGENTO/ROOT/shell && /bin/bash cron_multithread.sh default >> ../var/log/cron_default.log 2>&1

What I'm doing wrong?

When clicking (instead of hovering) on a task run you should get to a separate single view.
Sometimes there's too much information to display in a popup and the single view should allow to look at that information in better way.

There also should be a job single that shows a history of the runs (incl. future runs). From there you should also be able to get to the run single view.

Actions on single view:

• edit job
• schedule now
• kill
• (run now)