Internal torch error: Could not run 'aten::empty.memory_format' with arguments from the 'CUDA' backend about syntaxdot HOT 9 CLOSED

Could you perhaps show add the command you are using to start finetuning. Additionally, what is the output for

echo $LD_LIBRARY_PATH

and

ls -l /content/drive/MyDrive/SyntaxDot/libtorch/lib/

I think this error could happen when you have built SyntaxDot against the CUDA version and are running against the CPU version or vise versa. The best thing to do during the build is to set LIBTORCH to your downloaded libtorch directory. So I think in your case:

export LIBTORCH=/content/drive/MyDrive/SyntaxDot/libtorch
cd syntaxdot
cargo build --release

By default, the tch Rust library downloads a non-GPU version of libtorch. You then run into the issue that the compiled binary is not properly linked against the libtorch CUDA libraries.

(I am sorry that linking against libtorch CUDA is such a mess. If libtorch with CUDA wasn't so large, I would've provided prebuilt binaries.)

from syntaxdot.

Hey, thanks for the answer.
So, !/root/.cargo/bin/syntaxdot prepare /content/drive/MyDrive/SyntaxDot/syntaxdot.conf /content/drive/MyDrive/SyntaxDot/tuebadz-conllu-new.conllu works.

I might have to add that I am trying to run the finetuning process on Google Colab, where I was able to install an instance of rustup to cargo install syntaxdot. The problem is, that I cannot really create a rustup environment in which I could use Python, therefore, I had to improvise. However, without GPU, I cannot train the model therefore, I am trying to use Google Colab.

Thanks!

from syntaxdot.

No solution yet?

from syntaxdot.

Did you set the LIBTORCH environment variablebefore building SyntaxDot, to ensure that it's built against the GPU version? You'd need to set it to the main libtorch directory, I think in your case it would be:

export LIBTORCH=/content/drive/MyDrive/SyntaxDot/libtorch

from syntaxdot.

yes, I have to use os.environ, but yes, putting it before build worked... however, I now get the following error after starting training:

thread 'main' panicked at 'Head out of bounds', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/udgraph-0.7.0/src/graph.rs:452:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

from syntaxdot.

Interesting. That is a bug, however, that should only happen if the input data has incorrect dependencies (heads that do not exist). If the input does not have dependencies at all, remove the biaffine layer from the syntaxdot configuration.

from syntaxdot.

Actually, I already removed the biaffine layer from the configuration. Nevertheless, the error occurs. I am using the tuebadz conllu data, and changed the misc column so that only the named entity tags are available (e.g. NE=B-PER). So, what else could I do to solve the issue?

from syntaxdot.

The bug is that this should be an error, not an assertion. The error means that the input data is invalid. The CoNLL-U data is read as graphs. If you get this error, it means one token has a CoNLL-U head field that has an incorrect head. E.g. if you have a token like

32      ze      ze      PRON    PRON-pers       Person=3|PronType=Prs   35      nsubj   _        _

Which indicates that 35 is its head, but there is no 35th token. As a consequence, it is not possible to construct a dependency graph.

from syntaxdot.

Thanks for the answer. Also, I noticed, that \d+-\d+ for words like "im" (in dem), "ins" (in das), "übers" (über das) are not allowed, even though it is a valid coNLL-U formatting.

from syntaxdot.