Internal mmap syscall fails about sgx-lkl HOT 3 CLOSED

Thanks for the explanation. I figured most of this out myself, after noticing the buried message of my heap setting being ignored due to the missing signing key. One thing that tripped me though, is that looking at https://github.com/lsds/sgx-lkl/blob/master/src/main/sgx_lkl_run.c it seemed like the default heap size should be plenty (>700MB). But I guess this is ignored, unless a signing key is provided?

Anyhow, my little proof of concept now works (HTTPS Python web server using the RA-TLS certificate and key).

from sgx-lkl.

I've noticed that playing around with SGXLKL_CMDLINE="mem=xxx" helps somewhat, since the default size is ~12MB. However, at around 80MB the same thing happens again: the internal mmap fails, albeit much earlier, when it tries to set aside said 80MB of kernel memory. I'm assuming that its internal pool cannot satisfy the request. How do I increase that internal memory pool, if my application needs more memory?

from sgx-lkl.

Hi Thomas,

thanks for reporting this! There are a few things to say here.

First of all, as you have noticed there are currently two separate memory pools, one managed by the kernel, and one managed by an internal allocator/a custom mmap implementation (which can be found in src/sgx/enclave_mem.c). After enclave initialization, the enclave size is fixed. The enclave size is roughly determined by the size of the code and data sections of libsgxlkl.so, plus the enclave heap size specified via SGXLKL_HEAP, plus stack/TCS/SSA/TLS pages for each enclave thread.

Currently, when the kernel inside the enclave boots up it will allocate a chunk of memory via mmap of the size specified by the kernel boot mem parameter or the size specified by the BOOT_MEM variable in the main Makefile (12 MB by default). This memory is taken from the SGXLKL_HEAP pool which is managed by the enclave_mman.c code. While this memory is initially intended for serving allocations during the kernel boot process, this memory is never released (via unmap) to our internal allocator again. Instead it is gifted to the kernel-internal allocators to serve requests by kmalloc, vmalloc, etc. Allocations by the application itself will continue to be served by the custom SGX-LKL allocator.

Obviously having two distinct allocators leads to fragmentation and can lead to out of memory issues like the ones you see even when theoretically there is still enclave memory available. We are currently working on unifying these two allocators into one so that in the future we 1) won't have the mentioned fragmentation anymore, and 2) won't require application-/workload-specific adjustments to the size of the boot memory pool.

In the meantime, and to answer your question, you will have to adjust the amount of kernel memory manually. As the kernel memory is taken from the internal memory pool, you will have to both increase the amount of kernel memory (via SGXLKL_CMDLINE) and the enclave heap size. In your case, it seems like the 80MB allocation fails because the enclave heap size is too small. If you don't specify SGXLKL_HEAP at all, the enclave heap size will be determined based on what will fit into the EPC, e.g. it will be below ~<96 MB. The issue here might have been that you did not provide an enclave signing key via SGXLKL_KEY. In that case, the SGXLKL_HEAP argument will be ignored, and the default will be used. Every enclave has to be signed, but when the default enclave heap size is used, no key needs to be specified for SGX-LKL because a "default" signature is generated as part of the standard build process for the default enclave size. Anyway, I would consider this to be a bug as a provided SGXLKL_HEAP value should never be silently ignored. This definitely will be fixed.

So, maybe you can try the following:

SGXLKL_CMDLINE="mem=80M" SGXLKL_HEAP=500M SGXLKL_KEY=../../build/config/enclave_debug.key SGXLKL_TAP=sgxlkl_tap0 RATLS_AESMD_IP=10.0.1.254 SGXLKL_VERBOSE=1 build/sgx-lkl-run apps/python-https-ratls/sgxlkl-miniroot-fs.img /usr/bin/python /https-server.py

Also, for debugging these kind of issues, setting SGXLKL_TRACE_MMAP=1 might be helpful. It will log every mmap/mremap/munmap call to the internal allocator and gives you insights into the amount of free/used enclave heap memory during execution.

To summarize, there are a number of changes we should make:

1. Update the README to clarify how memory allocation works and how the enclave size/amount of memory available to the application can be increased. We also should add a paragraph on enclave signing.
2. Don't silently ignore the SGXLKL_HEAP parameter when no signing key is provided.
3. Unify the two memory pools/allocators.

The first two should be easy to fix and we should be able to do that within the next few days. We have already done work on the unified allocator, but we are not quite there yet. I assume we can merge those changes soon as well, so that in cases like this one you won't have to manually adjust the kernel memory size anymore.

from sgx-lkl.