Limit context memory use about futhark HOT 4 OPEN

I propose two new C API functions:

void futhark_context_config_set_max_memory_unused(const char *space, int64_t bytes);
void futhark_context_config_set_max_memory_usage(const char* space, int64_t bytes);

The former function limits how many unused bytes are allowed to be in the free list(s), and the latter how many may be allocated total (actually there will be some fine print regarding CPU memory in particular, as that is difficult to count when stack-allocating). The space argument is used to indicate which memory space we are considering (e.g. you can limit GPU memory without limiting CPU memory). We could ditch that for simplicity, and then the limit would apply generally to all memory spaces.

from futhark.

I'm not completely sure what you are asking for. There are two ways you can limit memory:

1. Put an upper bound on how much free memory the Futhark runtime system is allowed to keep in the free list.
2. Put an upper bound on how much memory Futhark is allowed to allocate in total.

Option (1) means a Futhark program can still eat up all the system memory, but option (2) means programs might unexpectedly fail.

I guess both are useful in different circumstances.

from futhark.

I was primarily thinking of total memory, but for my purposes either would be sufficient. If I could only choose one of them, 1 seems more flexible, in the sense that the same setting could be used for any program without causing any crashes. Option 2 does seem attractive in the context of sharing hardware though.

I'm also a bit curious how limiting the memory use could affect performance if there is sufficient L3-cache to store all necessary data. Could limiting the total memory use to below the size of the cache actually improve performance by reducing cache misses?

from futhark.

If you limit the total memory usage, it just means more programs will fail with OOM errors. It will not affect the performance of programs that actually run. The only point where Futhark makes decisions based on memory availability is when it manages the free list.

from futhark.