find_probable_cause_of_failure() is bad at fetching logs about mrjob HOT 8 CLOSED

Would it be more appropriate to call out to the ssh utility or to add a dependency on something like paramiko?

from mrjob.

I'm refactoring the S3 and SSH log fetcher functionality to subclass LogFetcher in a new submodule.

from mrjob.logfetch.ssh import SSHLogFetcher
# etc.

This will probably also involve breaking a lot of S3-related code out of EMRJobRunner, which probably isn't a bad thing since that class is currently a couple thousand lines long.

from mrjob.

Yup, that sounds good. Another good way to approach this is to start out by building a standalone utility (in mrjob.tools.emr) that fetches and analyzes logs, and then patch it into EMRJobRunner.

And please, use scp; don't add another library dependency. :)

from mrjob.

Can do. My current strategy is to copy any relevant functions (ls/get from S3, local, and SSH + dependencies) into instance methods and helpers for fetchers so that logfetch can be used independently. Then I will write a tool around it, verify by hand on various cases, add mocking for SSH + automated tests, and finally insert it into EMRJobRunner, removing redundant functions.

from mrjob.

Sounds like a good plan.

from mrjob.

New info: logs have slightly different paths on S3 vs local. Here's a quickref I'll put in the comments:

S3 location             Local location
/daemons                / (root)
/jobs                   /history
/node                   <not present>
/steps                  /steps
/task-attempts          /userlogs

from mrjob.

I believe this can be closed unless it also encompasses a log fetching/parsing refactor.

from mrjob.

Yup, thanks!

from mrjob.