gg force --engine lambda failing after emptying bucket about gg HOT 10 CLOSED

Removing ~/.cache/gg/remotes and all its content should fix the issue. gg will reupload the missing files.

If you look at the error message, it says that the blobs VziaXgzsiNzeCIBBjDSZ9oHqywVIPYTBP5.ksiphPP_000000016 and VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160 are missing from the bucket. In fact, gg didn't try to upload those file because it has done that before (according to a cache entry in ~/.cache/gg/remotes) and it's not aware that you emptied the bucket!

This reminds me we were meant to create a gg bucket-sync or a similar tool to resync the local index and the remote bucket, but never got around to it.

from gg.

that fixed the issue! Now I'm able to see the thunks in the bucket. thanks!

I'm running another script and this time it is giving me a different error:

$ gg force --engine lambda output.txt
→ Loading the thunks...  done (0 ms).
No files to upload.
gg-force: no execution engine could execute T_iyxy6SOlvutbNC9MrsSTZH8sNaxcqEZyICDFtu0aWk000002b4
$ gg force output.txt
→ Loading the thunks...  done (0 ms).
$

Can I see the execution logs somewhere? Or is there some other way to debug this?

from gg.

This is probably because the total size of the input files to that thunk exceeds 230 MiB. That's a hard-coded limit in the lambda engine, since Lambdas come with 500 MiB of disk space (gg reserve the rest of the space for the output).

You have a couple options: either change your thunk to use smaller files as input, or, change the code and increase the maximum limit. Just be careful of the total 500 MiB disk space and the fact that your output needs to fit on disk as well.

from gg.

When running a gg job, it's possible that some thunks exceed the limitations of the engine. In that case you can run gg force like this:

gg force --engine lambda --fallback-engine local thunk_name

If none of the engines listed by --engine can run the thunk, it will be run locally (of course, it will need to copy all of the dependencies to the local machine). Of course, in your case here, I think you have a single thunk that you wanna run, so, this may not be so useful.

from gg.

Oh, my script + files were 266MB XD. Changing the limit fixed the issue :). The fallback option will be quite useful later i think!

Looks like the lambda workers don't seem to have libpython3.7m.so.1.0 for nuitka to work with:

$ gg infer argtest 65 i.txt o.txt --arg 23 --inputfile j.txt --outputfile p.txt
$ gg force --engine lambda o.txt 
→ Loading the thunks...  done (0 ms).
No files to upload.
argtest: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
std::exception
 `TMDchyW8zrDeCv8W2XKHvrTClA8mMZQzoZxo_941Odv0000001b5': process exited with failure status 127

gg-force: execution failed: TMDchyW8zrDeCv8W2XKHvrTClA8mMZQzoZxo_941Odv0000001b5

Will see if they have some something else I can use. I'll take a look at this tomorrow.

from gg.

@sadjad can gg be told to upload the libpython.so file as well?

from gg.

Oh, that's the issue with Nuitka: it relies on the availability of Python libraries on the host. Let me check something first.

from gg.

I think for now the quickest solution might be going back to PyInstaller. There's a way to have gg to include the required libraries, such as libpython.so, but it's a little painful. Meanwhile, I'm gonna try to figure out if there's way to use Nuitka with Lambda.

from gg.

Cool, I'll try the pyinstaller route tomorrow. Thanks for looking at Nuitka!

from gg.

Looks like running a pyinstaller binary on lambda also has a similar problem:

[7] Error loading Python lib '/tmp/_MEIovTa5m/libpython3.7m.so.1.0': dlopen: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/_MEIovTa5m/libpython3.7m.so.1.0)

I'm starting a new thread [#48] for this since this thread refers to the solved empty bucket issue.

from gg.