OpenBSD - MoarVM panic: Memory allocation failed about app-rakubrew HOT 4 CLOSED

Hm. The error points quite clearly at an out of memory situation.
Can you determine at what memory use the build roughly errors? (e.g. by looking at top/htop/...)
Do you know whether OpenBSD has any mechanisms to limit the resources a process can use and whether any such limit is in effect?

from app-rakubrew.

@Veraellyunjie How much RAM does the computer have you are building on? The Rakudo CORE.c.setting compilation (the part where it failed) does require quite some RAM (>1 GB).

from app-rakubrew.

@patrickbkr Have 16 GB RAM

from app-rakubrew.

Success. Your @patrickbkr cue made me look in the right direction.

Can you determine at what memory use the build roughly errors? (e.g. by looking at top/htop/...)

I recall it was 1 GB back then.

Now, for measuring memory consumtion of a successful build I tried:

/usr/bin/time -l rakubrew build  # -l - The contents of the rusage structure are printed.
time: rakubrew: No such file or directory

/usr/bin/time -l /opt/raku/rakubrew/bin/rakubrew build
The shell hook required to run rakubrew in either 'env' mode or with the 'shell' command seems not to be installed.
Run 'rakubrew init' for installation instructions if you want to use those features,
or run 'rakubrew mode shim' to use 'shim' mode which doesn't require a shell hook.

Is there a way to run rakubrew via a wrapper programm like /usr/bin/time or torsocks?

while :; do ps -x -o rss,maxrss,vsz,command >> /tmp/rakubrew_build.ps.log; sleep 1; done
grep moar /tmp/rakubrew_build.ps.log | sort -h -k 1
grep moar /tmp/rakubrew_build.ps.log | sort -h -k 2
grep moar /tmp/rakubrew_build.ps.log | sort -h -k 3
Maximum in KB:
1803816 for resident set size; 1803816 for max resident set size (dunno the difference); 1934780 for virtual size.

Do you know whether OpenBSD has any mechanisms to limit the resources a process can use and whether any such limit is in effect?

datasize in /etc/login.conf and /etc/login.conf.d/${class}

MoarVM seems to honor datasize-cur soft limit - even when datasize-max hard limit is also set and is higher, MoarVM still stops at datasize-cur.

Here are defaults for OpenBSD 7.1 amd64:
Without having added yourself to a class:

default:\
...
        :datasize-cur=1024M:\
...

root is like daemon:

daemon:\
...
        :datasize=4096M:\
...

It is common to add a user to staff:

staff:\
        :datasize-cur=1536M:\
...

Note raised limit for compiling ports:

pbuild:\
...
        :datasize-cur=8192M:\
...

So, I raised my user's class' datasize-cur to 8192m first, 2022.06 release did compile and measurements showed almost 2G.
I lowered datasize-cur to 2048m and 2022.04 release did compile. Similar memory usage.

If you are on OpenBSD and experience similar problem,
As regular user: ulimit -S -d, ulimit -H -d. If -Hard limit is higher and seems sufficient, raise your -Soft limit: ulimit -S -d $(ulimit -H -d).
As root:
usermod -L <login class> <username> - make the user belong to a class
edit /etc/login.conf.d/${class} or /etc/login.conf appropriately
Print current limit: su -c <login class>, ulimit -d

Memory exhaustion check:

while :; do ps auxw | { sleep 0.01; grep loop ;} ; sleep 0.1; done

somehow, ps doesn't show rakudo command name at the beginning of the line - it starts with arguments and at the end there is (rakudo) in parentheses, which may not be shown due to line length, so grep loop

raku -e 'my @a.[0] = 0; loop { @a.push: @a[^@a.elems/5].Str }'

Better one-liners to test memory exhaustion are welcome.

The internet shows similar issues to be rather frequent, due to OpenBSD policy of tighter limits.
So if rakubrew team find it appropriate, I'd suggest adding to rakubrew webpage and built-in help:

OpenBSD note:
For building Raku from source you may need to bump up `datasize` system limit to `2048M` or higher.
As regular user, try: `ulimit -S -d $(ulimit -H -d)` - consult `man 1 ksh` http://man.openbsd.org/ksh#ulimit
As root: `:datasize-cur=2048M:` in `login.conf`, then relogin - consult `man 5 login.conf` http://man.openbsd.org/login.conf#CAPABILITIES

and maybe a check in code:

if ($^O eq 'openbsd') {
	require BSD::Resource;
	BSD::Resource->import;
	my ($soft, $hard) = getrlimit("RLIMIT_DATA");
	if ($soft < 2097152) {
		my $warning = <<~"END";
			system limit of $soft on datasize is likely to be insufficient for building Raku.
			2048M was reported to be required for `moar-2022.06` on OpenBSD 7.1 amd64.
			If build fails due to lack of memory, raise system limit on the size of the data area:
			as regular user, try: `ulimit -S -d $(ulimit -H -d)` - consult `man 1 ksh`
			as root: `:datasize-cur=2048M:` in `login.conf`, then relogin - consult `man 5 login.conf`
			END
		warn $warning;
	}
}

Thanks

from app-rakubrew.