Getting stdout from tool about edalize HOT 11 CLOSED

This would be great. I have a couple of use cases I would like to have supported here

1. As a developer, creating a python tool that uses Edalize, I would like to access stdout/stderr independently of each other
2. As a user of FuseSoC I would like to run FuseSoC (and thereby Edalize) with minimal output in the normal case, but have access to the full output if something goes wrong
3. As a user of FuseSoC I would like to set a verbose mode and a silent mode

2 or 3 is probably most relevant for FuseSoC, but I'm mentioning it here so that the design in Edalize enables this

from edalize.

It would be a really nice feature!!

from edalize.

Is it on your roadmap? We would like use this feature in TerosHDL.

from edalize.

In the past I've used tool output to help automated CI flows make decisions on build and test outcomes and sometimes it has been necessary to use stdout or stderr. I could definitely make use of this feature if it were available.

from edalize.

It sounds like there are multiple uses for this. Just started looking into it hoping it would be a simple hack but after reading up a bit I'm not so sure anymore. The documentaion warns against capturing stdout when using subprocess.check_call and recommends using subprocess.run instead, which would break Python 2. Perhaps this is the time that Python2 support is dropped, or maybe it works fine with check_call. Does anyone have experience with this?

from edalize.

Sadly this is stretching my knowledge quite far. I'd be wary of using subprocess.check_call because of the warning that the child process will block if it produces too much for stdout. These tools can produce an awful lot of stdout.

I know we've used subprocess.Popen() in the past. Briefly looking at the documentation it looks like Popen is a more flexible option but I don't know what the implications/drawbacks are.

from edalize.

There is a subprocess.check_output() -- however the entire stdout / stderr will be stored in memory.

from edalize.

It's a lot of output for sure, but it can't realistically be more than a couple of megabytes which is a drop in the ocean when it comes to EDA tools. But I have no idea what is "too much" output that will cause the process to block. kilobytes of data? Megabytes? Not a clue. Don't think check_output will be enough here since there are likely users who wants to separate stdout and stderr.

Still, we need to move forward with this. My original idea was to just create new file-like objects with open, pass them through the constructor and use one for each of stdout and stderr. Then the user can do whatever with those things. Print or write to a file. Perhaps that is just fine despite the scary warnings from the subprocess docs. I'll give it a shot

from edalize.

Since we're off Python 2 the Python 3.5+ subprocess.run interface should be available. The capture_output argument is newer, but it reads like the stdout and stderr arguments do the same thing. I think all of these are just wrappers for subprocess.Popen so all of the earlier warnings might still apply, but perhaps it would offer a cleaner interface.

from edalize.

Yeah, reading this conversation again I'm not sure we should be afraid of these warnings. Let's just go with the easiest solution and patch if necessary

from edalize.

I want this!

from edalize.