This is the implementation of a distributed file system that described in the paper: https://cs.uwaterloo.ca/~mtabebe/publications/abebeWatDFS2019SIGCSE.pdf
Below is a diagram that traces the steps (indicated by numbered arrows) that a file system operation (such as reading a file) invoked by an application (e.g., the cat command) will take to reach WatDFS, as well as the actions that WatDFS should take. The dashed lines indicate separate processes, the thick gray lines separate machine boundaries, and the thin gray lines separate user space from the kernel.
I have implemented both the client and server components with features including mutual exclusion, file freshness check and atomic read & write ...
I have implemented two global structure variable to keep track of the state and metadata of each file on client side.
struct fileMetadata {
int client_mode;
int server_mode;
time_t tc;
};
fileMetadata
tracks the fh
and file access flags information. It also keeps track of the last access time of current file. Any open or modification to a file, might change the metadata.
struct file_state {
time_t cacheInterval;
char *cachePath;
std::map<std::string, struct fileMetadata > openFiles;
};
It is important to know the syncrhonization status of a file on both client and server side and we better need a openFiles
map data structure to record files opened by the client side. CachePath
is simply the path of a file and cacheInterval
is mainly used to check the freshness. Overall, we say this structure is the global state of all target files (userdata
).
On server side, i decide to create a global structure to keep track of 3 states of a file (read only, write and read, closed). And also a map to track all fils opened from client side. Those are intialized during server call to watdfs_open
struct file_critical_section{
bool read; // true for r and false for w
rw_lock_t * lock ;
};
std::map<std::string, struct file_critical_section> fileMutex;
int download_to_client(struct file_state *userdata, const char *full_path, const char *path);
- initalize
stat buffer
on server side and client will trigger aopen rpc call
to read file information from server and copy to the local side if the file on server exists. - Call
open
system call to open file from local side. this might results in creation (mknod
system call) of new file if it does not exist. - After local file is opened, truncate the local file based on the required size limit from server. and then trigger
read rpc call
to read from server so that we can then write all data and stored in the local file usingpwrite
system call function. - Update file metadata accordingly.
int push_to_server(struct file_state *userdata, const char *full_path, const char *path);
-
same as above.
-
open rpc call
is triggered to open the file on server. If not exist, we will create a new one and then open on the server. -
We will open the target open file that already exists through system call
open
and then read all data throughpread
. -
Trigger
truncate rpc call
to truncate the server file based on size information of local file. So it gets ready for receiving the data. -
Now, it is ready to write the data from local to server through
write rpc call.
-
Update metadata accordingly.
-
Try to get the attribute of this file on server and test to see if above steps are functioned correctly.
Atomic Transfer:
In addition to above download
and upload
explanations, the client side is forced to try to achieve and hold the lock_to_read
before and during the file downloading to client and achieve and hold the lock_to_write
before and during the file uploading to server process. Locks are released once the download and upload period is each done.
With this implementation feature, file is either transfer completed or not at all as others wont be able to interrupt at the middle of transferring.
Mutual Exclusion:
Recall that client side has a map structure to keep track of all opened files by client. When a open
call is trigger to a file, we will first examine if it is opened already by others. This can prevent the sitation where mutiple files are trying to access the same file at the same time. This ensures also mutual exclusions.
Recall that the server also have a map to track all files opened. To ensure the mutual exclusions of server write, if the file is already opened for writing, any futher open call will result in an error code returned.
The key idea to check if the operatio on file satisfies the freshness condition at time T for some freshness interval t. ( (T โ Tc) < t || T_client == T_server )
Tc: last time the entry was validated by client.
T_client: last time that the file was modified by client.
T_server: last time that the file was modified by the server.
I have followed strcitly as described on the A3-spec and will not give more details about exactly how to check freshness as those are exact sentences on the spec. So for my implementation, freshness is all checked at the time of client's getattr, read, write, utimens and truncate
When a fsync is issued, the client's file will be uploaded to server. All time metadata are updated accordingly. The last acces time of a file (saved in fileMetadata
structure) is set to current time after open or freshness check.
#define LOCKERR -600;
#define UNLOCKERR - 700;
Those are returned when a call is failed to get the lock or fail to unlock.
For other error codes, -EMFILE
is returned when mutiple files are required to access/open a single file. -errorno
is returned for any system call failure.
I have implemented multiple python function to do the file manipulation and test each piece of my client-server model implementation.
basic_open_check.py
: create new and open file with multiple flags for read/write flag. Try to write to a file under different open scenarios and see if error returned. Then close the file. Also test for upload.
basic_upload_truncate.py
: open existing file on local and truncate it and then close the file.
basic_fsync_utimens_check.py
: (1) issue fsync after open and write operations. (2) issue utimens and check for file metadata.
complex_mutual_exclusion.py
: open two client and trigger write operations. See if there is error.
complex_freshness_check.py
:do the normal read and write but including a sleep function for 10s
Note that there are more testing scripts that I have implemented but to keep the manual concise, i would not list all of them. But from above, clearly, you can see that I test the model from two different approaches/perspective: (1) do a series of file system call from single client. (2) do complex calls from multile clients.