vramfs
Unused RAM is wasted RAM, so why not put some of that VRAM in your graphics card to work?
vramfs is a utility that uses the FUSE library to create a file system in VRAM. The idea is pretty much the same as a ramdisk, except that it uses the video RAM of a discrete graphics card to store files. It is not intented for serious use, but it does actually work fairly well, especially since consumer GPUs with 4GB or more VRAM are now available.
On the developer's system, the continuous read performance is ~2.4 GB/s and write performance 2.0 GB/s, which is about 1/3 of what is achievable with a ramdisk. That is already decent enough for a device not designed for large data transfers to the host, but future development should aim to get closer to the PCI-e bandwidth limits. See the benchmarks section for more info.
Requirements
- Linux with kernel 2.6+
- FUSE development files
- A graphics card with support for OpenCL 1.2
Building
First, install the OpenCL driver for your graphics card and verify that it's
recognized as an OpenCL device by running clinfo. Then install the libfuse3-dev
package or build it from source. You will also need pkg-config and OpenCL
development files, (opencl-dev, opencl-clhpp-headers package or equivalent),
with version 1.2 of the OpenCL headers at least.
Just run make to build vramfs.
If you want to debug with valgrind, you should compile with the minimal fake OpenCL implementation to avoid filling your screen with warnings caused by the OpenCL driver:
- valgrind:
make DEBUG=1
Mounting
Mount a disk by running bin/vramfs <mountdir> <size>. The mountdir can be
any empty directory. The size is the disk size in bytes. For more information,
run bin/vramfs without arguments.
The recommended maximum size of a vramdisk is 50% of your VRAM. If you go over that, your driver or system may become unstable because it has to start swapping. For example, webpages in Chrome will stop rendering properly.
If the disk has been inactive for a while, the graphics card will likely lower its memory clock, which means it'll take a second to get up to speed again.
Implementation
The FUSE library is used to implement vramfs as a user space file system. This eases development and makes working with APIs such as OpenCL straightforward.
Basic architecture
When the program is started, it checks for an OpenCL capable GPU and attempts to allocate the specified amount of memory. Once the memory has been allocated, the root entry object is created and a global reference to it is stored.
FUSE then forwards calls like stat, readdir and write to the file system
functions. These will then locate the entry through the root entry using the
specified path. The required operations will then be performed on the entry
object. If the entry is a file object, the operation may lead to OpenCL
cvEnqueueReadBuffer or cvEnqueueWriteBuffer calls to manipulate the data.
When a file is created or opened, a file_session object is created to store
the reference to the file object and any other data that is persistent between
an fopen and fclose call.
VRAM block allocation
OpenCL is used to allocate blocks of memory on the graphics card by creating
buffer objects. When a new disk is mounted, a pool of disk size / block size
buffers is created and initialised with zeros. That is not just a good practice,
but it's also required with some OpenCL drivers to check if the VRAM required
for the block is actually available. Unfortunately Nvidia cards don't support
OpenCL 1.2, which means the cvEnqueueFillBuffer call has to be simulated by
copying from a preallocated buffer filled with zeros. Somewhat interestingly, it
doesn't seem to make a difference in performance on cards that support both.
Writes to blocks are generally asynchronous, whereas reads are synchronous. Luckily, OpenCL guarantees in-order execution of commands by default, which means reads of a block will wait for the writes to complete. OpenCL 1.1 is completely thread safe, so no special care is required when sending commands.
Block objects are managed using a shared_ptr so that they can automatically
reinsert themselves into the pool on deconstruction.
File system
The file system is a tree of entry_t objects with members for attributes like
the parent directory, mode and access time. Each type of entry has its own
subclass that derives from it: file_t, dir_t and symlink_t. The main file
that implements all of the FUSE callbacks has a permanent reference to the root
directory entry.
The file_t class contains extra write, read and size methods and manages
the blocks to store the file data.
The dir_t class has an extra unordered_map that maps names to entry_t
references for quick child lookup using its member function find.
Finally, the symlink_t class has an extra target string member that stores
the pointer of the symlink.
All of the entry objects are also managed using shared_ptr so that an object
and its data (e.g. file blocks) are automatically deallocated when they're
unlinked and no process holds a file handle to them anymore. This can also be
used to easily implement hard links later on.
The classes use getter/setter functions to automatically update the access,
modification and change times at the appropriate moment. For example, calling
the children member function of dir_t changes the access time and change
time of the directory.
Thread safety
Unfortunately most of the operations are not thread safe, so all of the FUSE
callbacks share a mutex to ensure that only one thread is mutating the file
system at a time. The exceptions are read and write, which will temporarily
release the lock while waiting for a read or write to complete.
Benchmarks
The system used for testing has the following specifications:
- OS: Ubuntu 14.04.01 LTS (64 bit)
- CPU: Intel Core i5-2500K @ 4.0 Ghz
- RAM: 8GB DDR3-1600
- GPU: AMD R9 290 4GB (Sapphire Tri-X)
Performance of continuous read, write and write+sync has been measured for different block allocation sizes by creating a new 2GiB disk for each new size and reading/writing a 2GiB file.
The disk is created using:
bin/vramfs /tmp/vram 2G
And the file is written and read using the dd command:
# write
dd if=/dev/zero of=/tmp/vram/test bs=128K count=16000
# write+sync
dd if=/dev/zero of=/tmp/vram/test bs=128K count=16000 conv=fdatasync
# read
dd if=/tmp/vram/test of=/dev/null bs=128K count=16000
These commands were repeated 5 times for each block size and then averaged to produce the results shown in the graph. No block sizes lower than 32KiB could be tested because the driver would fail to allocate that many OpenCL buffers. This may be solved in the future by using subbuffers.
Although 128KiB blocks offers the highest performance, 64KiB may be preferable because of the lower space overhead.
Future ideas
- Implement RAID-0 for SLI/Crossfire setups
License
The MIT License (MIT)
Copyright (c) 2014 Alexander Overvoorde
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to
deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
sell copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent