Zombies! about pa11y HOT 12 CLOSED

Wow! This caused node to eat up 100% CPU as well! This is neat.

My knee-jerk reaction would be to agree that this a bug in PhantomJS itself, but there might be some opposing signs. Running the following works:

$ phantomjs test.js

Where test.js contains the following:

var page = require('webpage').create();
page.open('http://prairieschooner.unl.edu/?q=fusion%2Fwork%2Fbus-driver', function () {
    page.includeJs('http://squizlabs.github.io/HTML_CodeSniffer/build/HTMLCS.js', function () {
        console.log('Page loaded + JS injected.');
        phantom.exit()
    });
});

Despite taking 10s on my machine, the above leads me to believe that there is a possibility that the issue is not with PhantomJS.

Also...

It seems that, inconsistently, using the console reporter causes the process to endlessly eat up CPU (requiring SIGKILL to be sent), and using the JSON reporter causes the following fatal error and an exit status of 134.

FATAL ERROR: JS Allocation failed - process out of memory
Abort trap: 6

I'm not really sure what to make of this.

@mfairchild365 What kind out output are you getting when the pa11y process dies?

from pa11y.

Wow @mfairchild365, thanks for spotting this! I've had an issue with the demo server running out of memory lately and it was on my to-do list to investigate, I have a feeling it's the same issue.

When running pa11y -r json http://prairieschooner.unl.edu/?q=fusion%2Fwork%2Fbus-driver, I get the same output as @whymarrh:

FATAL ERROR: JS Allocation failed - process out of memory
Abort trap: 6

I'll investigate today and see if there's a way we can tidy up zombies if an error like this occurs.

from pa11y.

After an initial investigation, pa11y is getting as far as this block before crashing. Changing it to send an empty array (change return window.HTMLCS.getMessages(); to return [];) stops the error happening, so I'm (probably naively) assuming that in this case, the list of messages is eating up all the memory.

I'm trying to work out if we can spot this before sending messages back to pa11y, and this doesn't fix the big issue which is that pa11y often does create zombie processes. You can create them easily by setting the timeout to a low number: pa11y -t 200.

I'll continue to investigate.

from pa11y.

Thanks for looking into this. :) So far my custom timeout wrapper looks like it is solving the problem, but it is implemented in PHP and is specific for my program. A solution at the source of the problem would be better.

@whymarrh I'm getting the same error. I also occasionally get an error like this:

phantom stderr: ExceptionHandler::GenerateDump waitpid failed:No child processes
PhantomJS has crashed. Please read the crash reporting guide at https://github.com/ariya/phantomjs/wiki/Crash-Reporting and file a bug report at https://github.com/ariya/phantomjs/issues/new with the crash dump file attached: /tmp/0496639c-3dc0-7367-0026f857-399d49b7.dmp

However, that dump file is not actually being created, so I can't report the bug.

@rowanmanning I'm not convinced that the list of messages is eating up the memory. If I run HTML_Codesniffer from the browser (FF) on that page, it finds just over 300 messages and does not use an excessive amount of memory. I've seen pages with more messages that never have this problem. Might it have something to do with how pa11y parses the results?

On a side note, I saw in another issue that you are working on a newer version of pa11y that does not use html_codesniffer. Where can I learn more about this?

from pa11y.

@mfairchild365 here, I posted today: #46

from pa11y.

Have there been updates on this?

from pa11y.

Hi @mfairchild365, I spent a long time trying to sort this a month or so ago, but got really frustrated and regular work-load increased. I have some time today so might give it another shot with a fresh pair of eyes.

I might try a different approach and attempt to identify and kill any zombie processes before pa11y exits.

Thanks for bringing back to my attention.

from pa11y.

Thank you @rowanmanning. Let me know if there is anything that I can do to help, this is continuing to be a real problem for us.

from pa11y.

@rowanmanning This is still an ongoing issue for us. I'm curious if you have made any progress or if there is anything that I can do to help.

from pa11y.

Hi @mfairchild365, sorry for very slack replies lately. I've managed to block out a day (today) to work on this, will feed back if I work something out or get stuck. If I don't get anywhere in the next hour or so I'll try implementing a short-term fix which kills any zombie processes before node exits.

from pa11y.

Holy shit I think I found a fix! Fresh eyes I guess. Basically the messages that come back from HTML Codesniiffer contain a DOM element, this obviously eats up loads of memory and I think in the case of this specific page (and others) there must be an infinite loop or something like that. I'm removing the DOM element before sending data back to node locally, and it's working. PR incoming, would be great if you can test it out.

I'd still like to investigate what exactly it is about the page in question, but that can come later today.

from pa11y.

Closing, fixed in #63

from pa11y.