[Bug]: OOM when running stdlib level 8 and 6 about ballerina-lang HOT 4 CLOSED

So far I have observed fallowing problems related to memory with respect to BIR serialization logic

1. Every rec atom we create is not actually defined in the BIR. We create various atoms as part of the various type checks (such a Cloneable type). When we serialize the package we don't serialize these atoms but we still level "holes" in the atoms indices (since they are determined by the number of rec atoms defined at that point). As a workaround for this I introduced the ability to "compact" the rec atom indices before serialization by removing nulls. Unfortunately this reduce memory consumption of packages separated by one level only. (One possible improvement would be to do an analysis on the package before serialization and compact the indices before serialization). I managed to fix out of memory issues in level 6 and 7 this way. This was fixed by "re-indexing" the rec atoms as well write the BIR. Basically for each given rec atom index we will calculate a new index by the order in which we see it when serializing the BIR. This should ensure the BIR there are no holes in RecAtoms (assuming we serialize only one package).
2. Lets call BDDs with a single atom and left true, middle and and right false simple BDDs. Most BDDs are of this simple variety (These represent BDDs that are of single atomic type). However for each of these we spend the memory for 3 fields that hold pointers to left, right and middle (actual values are singletons in this case). Instead introduced a different subclass that only has the atom and waste no memory to store fields to left, right and middle
3. All mapping and list atoms store their "members" as cells (in order tot properly represent mutability) which consumes considerable amount of memory.
4. Different maps have fields with the same name, type and mutability combinations in most of the out of memory examples. Currently we create separate field objects for each (fixing 3 should reduce the overhead of this however)

from ballerina-lang.

#43095 Implements 1 and 2. I experimented with a solution to fix 3 (for mapping only) but of that to work either we need to update the type checking logic in MappingOps to work with them. Trying to create temperary cell atoms that are not part of environment failed. However given now we can remove Atoms from Env when we start to run out of memory this shouldn't be a major problem.

At point where we run out of memory we are holding a considerable number of byte arrays with the content of BIR files. Ideally we should have dropped those byte arrays by that point given those BIRs have been successfully unmarshalled. Given we don't have an efficient serialization format to represent SemTypes having lot of semtypes in the serialized file is exacerbating this issue. Therefore I have decided to temporarily disable semtype serialization for most cases (except for Fininte types)

from ballerina-lang.

Fixed with #43095

from ballerina-lang.

This issue is NOT closed with a proper Reason/ label. Make sure to add proper reason label before closing. Please add or leave a comment with the proper reason label now.

      - Reason/EngineeringMistake - The issue occurred due to a mistake made in the past.
      - Reason/Regression - The issue has introduced a regression.
      - Reason/MultipleComponentInteraction - Issue occured due to interactions in multiple components.
      - Reason/Complex - Issue occurred due to complex scenario.
      - Reason/Invalid - Issue is invalid.
      - Reason/Other - None of the above cases.

from ballerina-lang.