Getting segmentation fault while trying to run Gpipe example code about lingvo HOT 10 CLOSED

I am running into a similar issue (Segmentation fault) just trying to run the image/mnist example described in the /README.md file. Installed everything directly (not using docker).
tensorflow version: 1.14.1-dev20190412.
g++ version: (Debian 7.3.0-5) 7.3.0 (Only 4.8 is supported)

mnist_segfault.txt

Instead I did the example with docker, and that works perfectly fine.

from lingvo.

It's strange that you would be getting SEGV without any output. Try setting the environment variable TF_CPP_MIN_LOG_LEVEL to zero and rerun. Also, please rerun within GDB and tell us where the program crashed.

from lingvo.

Hi, I know how to use gdb normally, but I do not know how to use in this case. Can you help me with that!

from lingvo.

Just run the program with gdb --args. Then run at the prompt. Once it crashes, type where to display all stack locations of all threads.

from lingvo.

Commands I am running:
bazel build -c dbg //lingvo:trainer
gdb --args bazel-bin/lingvo/trainer --run_locally=gpu --mode=sync --model=lm.one_billion_wds.OneBWdsGPipeTransformer --logdir=/tmp/mnist/log --logtostderr --worker_split_size=4

I am facing the following problem while trying to do run in gdb:
"/home/guest/lingvo/bazel-bin/lingvo/trainer": not in executable format: File format not recognized
(gdb) run
Starting program: --run_locally=gpu --mode=sync --model=lm.one_billion_wds.OneBWdsGPipeTransformer --logdir=/tmp/mnist/log --logtostderr --worker_split_size=4
No executable file specified.
Use the "file" or "exec-file" command.

But when I do a "file /home/guest/lingvo/bazel-bin/lingvo/trainer" it shows: Python script, ASCII text executable

from lingvo.

Oh, right, sorry. It's a bit more complicated.

OK, first, let's try to see the logs by running env TF_CPP_MIN_LOG_LEVEL=1 /home/guest/lingvo/bazel-bin/lingvo/trainer.

from lingvo.

Hi, after running the above command env TF_CPP_MIN_LOG_LEVEL=1 /home/guest/lingvo/bazel-bin/lingvo/trainer I got the following, I am attaching it in a text file:
output_log.txt

Hi, can we have a common time so that we can solve this problem at one go, its taking a lot of time.

from lingvo.

Hi , I went and did gdb python ./core, and did a where in gdb, so this was the output. I did this maybe something will be related to this:

gdb-where.txt

from lingvo.

Hi any update/solution for the above.

from lingvo.

If docker worked fine then it's probably some package version issue. Unfortunately we're not going to try to support all configurations.

Hi , I went and did gdb python ./core, and did a where in gdb, so this was the output. I did this maybe something will be related to this:

gdb-where.txt

Since there's a reference to /usr/include/c++/7 is it possible you are using g++ version 7 (check g++ --version). If so it might be that lingvo does not work with g++7 like #63 (comment) implies.

from lingvo.