How to store chunk data in memory for reuse? about noa HOT 12 CLOSED

Excellent, that makes a lot of sense. Lets see what I can do with this! Thanks ;)

from noa.

Ahh yes, I completely forgot about the quirkiness of certain comparisons in javascript! Thanks for the reminder.

from noa.

Ok, so, as after digging through the library and testing, I found out that you really need to do two primary things to store the chunk data in memory for reuse.

1. Prevent the chunk from being overwritten with null or removed from memory when the chunk exceeds the boundary to get picked up by chunk removal, but do allow the meshes to get removed
2. Add the getChunk(world,x,y,z) function in to check and see if a chunk in memory exists before generating new chunk data to then be written into the designated chunk coordinate.

So, I've gotten the chunks to load from memory and not get deleted when exceeding the boundary, however, the meshes are invisible unless you place a block down, then they render. This is something a bit beyond the scope of what I understand about the library right now, however I'm definitely going to do some more digging into the code tomorrow to figure out why this is.

This the code I added to the bottom of the requestNewChunk() function:

if(getChunk(world,x,y,z) != null) {
noa.world.setChunkData(id, getChunk(world,x, y,z))
}else{
world.emit('worldDataNeeded', id, chunk.array, x, y, z);
}

I also modified the getChunk():
return world._chunkHash.get(mi, mj, mk) || null

from noa.

Hi, thanks for trying out the library!

So the idea here is, different applications will create/store/reuse world data in different ways. To make this possible, noa emits events when it's creating or disposing chunks of world data to be rendered, and trusts that the client application will (A) save any data it wants from chunks that are being disposed, and (B) fill up new chunks with data (newly-created data or data that was stored somewhere). The intention that it should be fairly easy to do this without having to fork or edit noa itself.

The specific events and APIs are:

• noa.world#chunkBeingRemoved => (id, array, userData)
• noa.world#worldDataNeeded => (id, array, x, y, z)
• noa.world.setChunkData(id, array, userData)

In the above, id is a unique string specifying the chunk, array is an instance of ndarray containing the voxel IDs for that chunk, and userData is any user-specified value you want.

(Note that none of these events or APIs give you a copy of the actual Chunk object that noa uses internally. That's intended to be an internal thing that the client application shouldn't need to know about.)

So the simplest way to have persistent world data is something like this:

noa.world.on('chunkBeingRemoved', function (id, array, userData) {
    // when chunk is removed, store data for later
    myDataStorage.storeForLater(id, array, userData)
})


noa.world.on('worldDataNeeded', function (id, array, x, y, z) {
    if (myDataStorage.containsChunk(id)) {
        // for new chunks, fill with data from storage if it exists
        myDataStorage.fillArrayFromStorage(id, array)
    } else {
        // otherwise create fresh data
        myWorldGenerator.fillArrayWithNewData(array, x, y, z)
    }
    // send the data to noa to be rendered
    noa.world.setChunkData(id, array, userData)
})

Side note: the data ndarrays are quite large, so it will save a lot of memory if you compress them when storing them for later. I use a module called voxel-crunch for this, and highly recommend it.

Let me know if anything is unclear!

from noa.

Ive been struggling with this error :(

Uncaught TypeError: Cannot read property '0' of null
    at Chunk.initData (bundle.js:7695)
    at World.setChunkData (bundle.js:6971)
    at storeContainsChunk (bundle.js:3085)
    at World.<anonymous> (bundle.js:3113)
    at World.EventEmitter.emit (bundle.js:493)
    at requestNewChunk (bundle.js:7162)
    at processChunkQueues (bundle.js:7102)
    at World.tick (bundle.js:6927)
    at Engine.tick (bundle.js:3496)
    at GameShell.onTick (bundle.js:4536)

It's almost like the saved chunk arrays are unreadable by the engine when it's getting parsed from chunk storage. I'm not using the encoder/decoder yet until I get a working version.

Here's my code:

var chunkStore = [];
var storeContainsChunk = (id) => {
    for(var i in chunkStore){
      if(chunkStore[i].id == id) {
          console.log('store contains chunk!');
          console.log(chunkStore[i].array);
          // noa.world.setChunkData(id, voxelDecode(chunkStore[i].array, new Uint32Array(chunkStore[i].array.length)));
          noa.world.setChunkData(id, chunkStore[i].array);
          return true;
      };
    } 
    return false;
}
var storeAwayChunk = (id,array,userData = null) => {
    chunkStore.push({
      id: id,
      // array: voxelEncode(array),
      array: array,
      userData: userData
    });
}

noa.world.on('chunkBeingRemoved', function (id, array, userData) {
  console.log('+++++++++++++++++++++++++++++++++++++++++')
  console.log(array);    // when chunk is removed, store data for later
    // myDataStorage.storeForLater(id, array, userData);
    storeAwayChunk(id,array,userData);
});

// add a listener for when the engine requests a new world chunk
// `data` is an ndarray - see https://github.com/scijs/ndarray
noa.world.on('worldDataNeeded', function (id, data, x, y, z) {
  if(!storeContainsChunk(id)){  
      // populate ndarray with world data (block IDs or 0 for air)
      for (var i = 0; i < data.shape[0]; ++i) {
        for (var k = 0; k < data.shape[2]; ++k) {
          var height = getHeightMap(x + i, z + k)
          for (var j = 0; j < data.shape[1]; ++j) {
            var b = decideBlock(x + i, y + j, z + k, height)
            if (b) data.set(i, j, k, b)
          }
        }
      }
      noa.world.setChunkData(id, data)
  }
});

Any ideas??

from noa.

--edit note: this is not a proper solution!

Here's the updated removeChunk function:

// remove a chunk that wound up in the remove queue
function removeChunk(world, i, j, k) {
    var chunk = getChunk(world, i, j, k)
    var id = chunk.id;
    var arr = chunk.array;
    console.log(arr);
    var usrdat = chunk.userData;
    world.emit('chunkBeingRemoved', id, arr, usrdat);

    world.noa.rendering.disposeChunkForRendering(chunk)
    // chunk.dispose()
    setChunk(world, i, j, k, 0)
    unenqueueID(chunk.id, world._chunkIDsInMemory)
    unenqueueID(chunk.id, world._chunkIDsToMesh)
    unenqueueID(chunk.id, world._chunkIDsToMeshFirst)
    // when removing a chunk because it was invalid, arrange for chunk queues to get rebuilt
    if (chunk.isInvalid) world._lastPlayerChunkID = ''
}

THANKS ANDY ;)

from noa.

Hey, reopening to answer your questions.

About data arrays becoming null - when the engine disposes a chunk, it cleans up its internal data structures so that they don't stick around in memory, and part of that is that it nulls out the data array. So the best thing would be to leave the engine code as-is, and when you're storing data copy the data you want (rather than just keeping a reference to the data array).

The best way to do this incidentally would be to use the voxel-crunch module I mentioned earlier. The code for this is as simple as something like:

var VoxelCrunch = require('voxel-crunch')

var storeAwayChunk = (id,array,userData = null) => {
    var voxelData = array.data
    var compressed = VoxelCrunch.encode(voxelData)
    // now "compressed" is a compressed copy of the chunk's voxel data,
    // suitable for storing and decoding later
}

from noa.

Question 2, about the userData argument - this is so that you can have multiple "worlds" of voxel data - like in a game if the player teleports to a new level, and you want that level to be populated with separate voxel data. The userData is just a label you can apply to each chunk's data so that you know which world it's associated with.

More discussion of this, with sample code, in #13 .

from noa.

Ahh ok, I figured chunk.dispose() was simply terminating the chunk itself and this didn't matter since the chunk would be cleared from the array anyway.

So, here's where I am, correct me if I'm wrong, but from what I can tell due to the way that EventEmitter works, I think creating a new reference is not possible outside of the removeChunk() function because chunk.dispose() is being invoked before world.on(chunkBeingRemoved) can execute.

Is EventEmitter running processes in a different thread? I'm not familiar with it.

So when this following is the removeChunk() function:

// remove a chunk that wound up in the remove queue
function removeChunk(world, i, j, k) {
    var chunk = getChunk(world, i, j, k);
    world.emit('chunkBeingRemoved', chunk.id, chunk.array, chunk.userData);
    world.noa.rendering.disposeChunkForRendering(chunk)
    chunk.dispose()
    setChunk(world, i, j, k, 0)
    unenqueueID(chunk.id, world._chunkIDsInMemory)
    unenqueueID(chunk.id, world._chunkIDsToMesh)
    unenqueueID(chunk.id, world._chunkIDsToMeshFirst)
    // when removing a chunk because it was invalid, arrange for chunk queues to get rebuilt
    if (chunk.isInvalid) world._lastPlayerChunkID = ''
}

and this following is the chunkBeingRemoved handler:

noa.world.on('chunkBeingRemoved', function (id, array, userData) {
  // compareCompression(array);
  var copiedArray = array;
  var copiedId = id;
  var copiedUserData = userData;
  console.log(copiedArray); // <---------------------------this will console.log() to null
    // when chunk is removed, store data for later
    // myDataStorage.storeForLater(id, array, userData);
    storeAwayChunk(copiedId,copiedArray,copiedUserData);

});

Then the newly copied array shows null.

I also tried adding in a new array reference in removeChunk() before passing the data into chunkBeingRemoved but to no avail.

from noa.

Hi, I'm not sure what's causing your issue but I can't reproduce it. E.g. when I add this:

noa.world.on('chunkBeingRemoved', function(id, array, userData){
	console.log(array)
})

..to /docs/test/index.js and then run it (with npm test), it always logs out the ndarray reference, not null.

Can you try minimally reproducing the problem using the example worlds (/docs/test or docs/hello-world/) as a starting point?

(Incidentally EventEmitter doesn't do anything asynchronous, it just calls all registered event listeners (synchronously) in the order they were added, so chunk removal event handlers should be getting executed before the chunk is disposed.)

from noa.

I'm still getting this issue... I tried cloning the repo again and using the plain test environment and I still got it. When I get some time, I'm going to take a look and see if I'm adding some other code in by accident that may be affecting the test environment. ;(

On a side note, in regard to the chunk data compression, I did try roaming throughout the world without it for around 10 mins loading probably a hundred or so chunks and the effect on performance was negligible(not using compressed chunks yet because I've had some issues using the compression library you showed me).

Compression is working, but the data is not the same when it's decoded. It's not saving the voxels that were changed:

function compareCompression(ndarray){
  var encodedArray = voxelEncode(array);
  var decodedArray = voxelDecode(encodedArray, new Uint16Array(array.length));
  console.log(array == decodedArray);
}

for some reason compareCompression(ndarray) is equating to false ;(

from noa.

Not sure what to say about the chunk disposal stuff - it's possible there's something wonky about my setup but AFAIK there are other people using the library without seeing the issue you're having. If you find more info please let me know.

About the compression question, please type [1,2,3] == [1,2,3] into your JS console, and then google comparing arrays in JS. ;)

from noa.