The documentation makes it seem that the normal way a cursor should work cannot impose a StoragePreference::NONE on the storage if the object has a defined preference.

Quote:

    /// Create a cursor without a storage preference override, at position 0.

Though in this example this happens exactly as it shouldn't.

        let (obj, _info) =
            os.open_or_create_object_with_pref(name.as_bytes(), StoragePreference::FAST)?;
        let mut cursor = obj.cursor();

        with_random_bytes(&mut client.rng, OBJECT_SIZE / 2, 8 * 1024 * 1024, |b| {
            cursor.write_all(b)
        })?;

If the default preference is FASTEST everything will end up in this tier.

This has been observed sporadically when many nodes are migrated. In observed cases, the free storage counter overstepped the total counter which should not happen at all, but some construction may lever this behavior. Needs further investigation.

In the current implementation the migration unpacks packed leafs as we need to modify. This is unnecessary overhead, rather we should modify the packed leaf in place using the binary search for keys and write the leafs out as is. Here we need to take care that parents of this leaf do receive the updated position.

For large objects it is advantageous to asynchronously perform migrations without user intervention. It may be advisable to perform this at a reduced rate for policy initiated migrations to remain responsive to operations from clients.

Once a dataset has been closed we are not able to modify the handle anymore and it should return a closed error. This has not been done yet.

For some things like objects it would be nice to know before hand if the data we are writing will fit on to the device, this has been exempted from the space accounting implemented as this does only reflect the actual on-disk state of the filesystem, which might not be the current actual state. For this reason we might want to keep an estimation of the future space requirements of data in the cache to know if storage will be available in the future.

For now a "fix" could be the assumption that all data of the cache will be written on whatever storage one wants to write data to, or faster. This will keep some space unusable and be overly careful but includes all cases in which errors might occur.

The determination of how much space will be required when writing to disk is easy as this is already part of the cache, but how this is distributed in the then storage is not trivial and also depends on the order of these objects in the tree. Determining this beforehand requires modification of the fallback semantics which leads to many more implications not fully explored yet.