dotnwat / gassyfs Goto Github PK

gassyfs - distributed in-memory file system over RDMA

License: GNU Lesser General Public License v2.1

Makefile 0.62% C++ 22.84% C 7.96% Shell 68.56% Lua 0.01%

gassyfs's Introduction

gassyfs - distributed in-memory file system over RDMA

GassyFS is a FUSE-based file system that stores data in distributed remote memory. Remote memory is managed and accessed using GASNet, which supports RDMA over a wide variety of high-performance network interconnects, as well as supporting slower network access methods such as UDP that are useful for development. The Gassy file system is intended to be used in a manner analagous to tmpfs, but when the amount of RAM needed exceeds that of a single node.

Testing

The file system is regularly tested with a variety of workloads. The following workloads are tested for each travis-ci.org build:

Run the Tuxera POSIX test suite (test/posix.sh)
Build Git and run unit tests (test/git.sh)
samtools

An additional set of larger workloads (in addition to those listed above) are run prior to each release:

Build the Linux kernel (test/kernel.sh)
Build the Ceph storage system (test/ceph.sh)
Multiple configurations of iozone (test/iozone.sh)
Build PostgreSQL and run tests (test/postgres.sh)

gassyfs's People

Contributors

Stargazers

Watchers

Forkers

michaelsevilla ivotron wenqicao cloudxtreme huangyibo doytsujin

gassyfs's Issues

fs: use async read/write

fs: don't cap read size

update makefile to support IBV conduit

fs: allocate space with flexible sized extents

fs: review truncate offset edge cases

fs: create ram-based backend

fs: du reports incorrect space usage

When empty du -sh . reports zero correctly. Other scenarios appears to just count directories.

nwatkins@issdm-36:~/gassyfs/blah$ dd if=/dev/zero of=xyz bs=1M count=1
nwatkins@issdm-36:~/gassyfs/blah$ ls -l
total 0
-rw-rw-r-- 1 nwatkins nwatkins 1048576 Sep  7 11:01 xyz
nwatkins@issdm-36:~/gassyfs/blah$ du -sh .
0       .
nwatkins@issdm-36:~/gassyfs/blah$ mkdir asdf
nwatkins@issdm-36:~/gassyfs/blah$ du -sh .
512     .
nwatkins@issdm-36:~/gassyfs/blah$ cp -a ../rocksdb .
nwatkins@issdm-36:~/gassyfs/blah$ du -sh .
44K     .

fs: implement efficient truncate with extent abstraction

integration tests

Coverage
Valgrind
Param sweep

move kernel inode cache refcount into index_cache

fs: use fuse_getgroups for permission checks

fuse_req_getgroups

integration tests: add rocksdb

There is a blocker currently in which rocksdb doesn't play nice with a custom tmp directory.

flexible storage backend (gasnet, local ram)

We need a flexible mechanism for dealing with different storage backends. For example, with GASNet-[Large|Fast] GASNet performs memory allocation on our behalf and informs us of the address space ranges on each node. With GASNet-Everything we have to allocate and manage our own ranges, and presumably there could me multiple per node. Likewise, a local RAM-based backend could be a special case of this with a local mmap rather than using malloc for each byte.

store policies in inode

We need to get the Lua script into the inode. The pNFS guys said that they defined the layout_hint on a create()... but I'm not sure how they did this without an ioctl.

fs: handle holes on read path that fall into extent

related to the truncate optimization where a logical hole overlaps an allocated extent.

fs: return iterator from allocate_space to avoid extra lookup

fs: support st_birthtime on mac

fs: implement hardlink (needed by ceph compilation)

git clone --recursive https://github.com/ceph/ceph.git
cd ceph
./autogen.sh
./configure --with-debug --with-librocksdb-static
make -j10

Then later

ln: failed to create hard link `.libs/librbd_replay.lax/lt110-libcommon_crc_la-crc32c_intel_fast_asm.o' => `.libs/librbd_replay.lax/libcommon.a/libcommon_crc_la-crc32c_intel_fast_asm.o': Function not implemented
ln: failed to create hard link `.libs/librbd_replay.lax/lt111-libcommon_crc_la-crc32c_intel_fast_zero_asm.o' => `.libs/librbd_replay.lax/libcommon.a/libcommon_crc_la-crc32c_intel_fast_zero_asm.o': Function not implemented
ln: failed to create hard link `.libs/librbd_replay.lax/lt112-libcommon_crc_la-sctp_crc32.o' => `.libs/librbd_replay.lax/libcommon.a/libcommon_crc_la-sctp_crc32.o': Function not implemented
ln: failed to create hard link `.libs/librbd_replay.lax/lt113-linux_version.o' => `.libs/librbd_replay.lax/libcommon.a/linux_version.o': Function not implemented

documentation update

inode reference counting scheme
locking
build instructions

fs: support FUSE_SET_ATTR_CTIME

This doesn't seem to work on Linux

fs: investigate 'writeback_cache' option in libfuse bleeding edge

fs: support fallocate (noop stub)

No need to actually implement this right now because it doesn't affect correctness, as long as there is space available. Wait until new allocator to implement this fully.

ci: add travis-ci script

Apparently one may be able to use UML to mount a FUSE file system:
https://travis-ci.org/vi/execfuse/builds/47789978

fs: support write_buf

fs: support retreive_reply

fs: evaluate options for write_buf optimizations

doc: clean-up and document code

support osx build

fs: configure a caching strategy

Check out the fuse file handle flags and different mount options.

test: add kernel compile test

fs: wrong modes returned

Testing true for executable when it is not.

nwatkins@node:~/gassyfs/blah/ceph/src$ touch x
nwatkins@node:~/gassyfs/blah/ceph/src$ if [ -x x ]; then echo "executable"; fi
executable
nwatkins@node:~/gassyfs/blah/ceph/src$ ls -lsd x
0 -rw-r--r-- 1 nwatkins zlog-PG0 23448 Sep 10 10:18 x

Normal behavior:

pn1215552:~ nwatkins$ ls -lsd x
0 -rw-r--r--  1 nwatkins  staff  0 Sep 10 10:18 x
pn1215552:~ nwatkins$ if [ -x x ]; then echo "executable"; fi
pn1215552:~ nwatkins$ chmod +x x
pn1215552:~ nwatkins$ ls -lsd x
0 -rwxr-xr-x  1 nwatkins  staff  0 Sep 10 10:18 x
pn1215552:~ nwatkins$ if [ -x x ]; then echo "executable"; fi
executable

test: create benchmarking script

fs: extending file doesn't allocate blocks

git clone --recursive https://github.com/ceph/ceph.git
cd ceph
./autogen.sh
./configure --with-debug
make -j10
make check

set_size: ino=57357 old=0 new=65536
bad read block ino=57357 blkid=0 blkssize=0 offset=0 readsize=65536 fsize=65536
gassy: gassy.cc:445: ssize_t Gassy::Read(FileHandle*, off_t, size_t, char*): Assertion `blkid < blks.size()' failed.
*** Caught a fatal signal: SIGABRT(6) on node 0/10
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
bash: line 1: 242544 Aborted                 (core dumped) './gassy' '__AMUDP_SLAVE_PROCESS__' 'issdm-36:59823' 'blah'

legal: consider alternative licenses

add some fuse options (eg atomic o trunc) to default flags

fs: symlinks not created correctly during git checkout

The first batch below is inside a gassy mount and the symlink files are zero length and mode=0, even though it does show the link which is probably what is causing the problem.

nwatkins@node:~/gassyfs/blah/ceph$ ls -l src/include/rados
total 0
l--------- 1 nwatkins zlog-PG0      0 Sep 10 09:34 buffer.h -> ../buffer.h
l--------- 1 nwatkins zlog-PG0      0 Sep 10 09:34 crc32c.h -> ../crc32c.h
-rw-r--r-- 1 nwatkins zlog-PG0 102629 Sep 10 09:34 librados.h
-rw-r--r-- 1 nwatkins zlog-PG0  40149 Sep 10 09:34 librados.hpp
-rw-r--r-- 1 nwatkins zlog-PG0    929 Sep 10 09:34 librgw.h
l--------- 1 nwatkins zlog-PG0      0 Sep 10 09:34 memory.h -> ../memory.h
l--------- 1 nwatkins zlog-PG0      0 Sep 10 09:34 page.h -> ../page.h
-rw-r--r-- 1 nwatkins zlog-PG0    465 Sep 10 09:34 rados_types.h
-rw-r--r-- 1 nwatkins zlog-PG0    894 Sep 10 09:34 rados_types.hpp

nwatkins@node:~/gassyfs/blah/ceph$ ls -l ../../ceph/src/include/rados
total 160
lrwxrwxrwx 1 nwatkins zlog-PG0     11 Sep 10 09:35 buffer.h -> ../buffer.h
lrwxrwxrwx 1 nwatkins zlog-PG0     11 Sep 10 09:35 crc32c.h -> ../crc32c.h
-rw-r--r-- 1 nwatkins zlog-PG0 102629 Sep 10 09:35 librados.h
-rw-r--r-- 1 nwatkins zlog-PG0  40149 Sep 10 09:35 librados.hpp
-rw-r--r-- 1 nwatkins zlog-PG0    929 Sep 10 09:35 librgw.h
lrwxrwxrwx 1 nwatkins zlog-PG0     11 Sep 10 09:35 memory.h -> ../memory.h
lrwxrwxrwx 1 nwatkins zlog-PG0      9 Sep 10 09:35 page.h -> ../page.h
-rw-r--r-- 1 nwatkins zlog-PG0    465 Sep 10 09:35 rados_types.h
-rw-r--r-- 1 nwatkins zlog-PG0    894 Sep 10 09:35 rados_types.hpp

dotnwat / gassyfs Goto Github PK

gassyfs's Introduction

gassyfs - distributed in-memory file system over RDMA

Testing

gassyfs's People

Contributors

Stargazers

Watchers

Forkers

gassyfs's Issues

Recommend Projects

Recommend Topics

Recommend Org