tugraz-isds / systemds Goto Github PK

A fix on the conditional that is now not applied in the outer product template, needs more work to get it right (see discussion on commit 4d085ad#comments)

Since changing to docker container, it seems like our buildin function tests fail again.

Last time we fixed it by updating the R version in the Github action, now we have to do it inside the docker container.

https://github.com/tugraz-isds/systemds/runs/527598022?check_suite_focus=true

I would prefer some other way to shutdown the server than shutdown. Fist of all a check is missing since we only start a subprocess if no SystemDS instance is running yet, secondly if we need a explicit shutdown it makes sense to also have an explicit start. As I have already suggested in my PR it would make sense to use a ContextManager (See https://book.pythontips.com/en/latest/context_managers.html). We should make sure that we are happy with our decision since it will likely stay for longer, I hoped that we find a way to automatically shutdown the java server process on exit of the python program, but a ContextManager would also be a good idea.

Originally posted by @kev-inn in #125 (comment)

The function JMCL test sometimes fails, sometimes not.

https://github.com/tugraz-isds/systemds/runs/527598141

I looked at the bug 🐛 @Shafaq-Siddiqi mentioned. I didn't find where to fix it, but I'll leave some more information for somebody to fix it:

I reduced the DML that produces the bug to

F = read($X, data_type="frame", format="csv")
A = cbind(F, as.frame(matrix(1, nrow(F), 1)))
print(toString(A))

To produce the error uncomment the spark test invocation in BuiltinMiceTest.java:49 and replace the content of src/test/scripts/functions/builtin/mice.dml with the three lines of DML above.

Upon running the test, the check for frame block dimensions in FrameBlock.java:1002 will now fail with org.tugraz.sysds.runtime.DMLRuntimeException: Incompatible number of rows for cbind: 98 (expected: 49)

So the block is split and the column to append is not. This results in the dimension mismatch. This is as far as I got. I didn't find where the split happens. I tried specifying dimensions explicitly in the read() function (that gave other errors, which I'll investigate another time) and in an MTD file. That did not help though :-/ Furthermore, the problem seems to occur only with "real" frame data, not with matrices converted to frames with as.frame().

Originally posted by @corepointer in #116 (comment)