This repository contains the starter code for project 3 (15-440, Fall 2021). These instructions assume you have set your $GOPATH to point to the repository's root P3/ directory.

Note that if you choose to test your implementation on AFS cluster, you need to manually install Go 1.17. For more instruction about setting up AFS, please check the README file in the P0 repo.

The starter code for this project is organized roughly as follows:

bin/                               Student-compiled binaries

sols/                              Staff-compiled binaries
  darwin_amd64/                    Staff-compiled Mac OS X executables
    crunner                        Staff-compiled TribClient-runner
    trunner                        Staff-compiled TribServer-runner
    lrunner                        Staff-compiled Libstore-runner
    srunner                        Staff-compiled StorageServer-runner

  linux_amd64/                     Staff-compiled Linux executables
    (see above)

src/github.com/cmu440/tribbler/
  tribclient/                      TribClient implementation
  tribserver/                      TODO: implement the TribServer
  libstore/                        TODO: implement the Libstore
  storageserver/                   TODO: implement the StorageServer

  util/                            Util functions
    keyFormatter.go                Format the key posted to storage server

  tests/                           Source code for official tests
    proxycounter/                  Utility package used by the official tests
    tribtest/                      Tests the TribServer
    libtest/                       Tests the Libstore
    storagetest/                   Tests the StorageServer
    stresstest/                    Tests everything

  tests_cp/                        Source code for official tests (checkpoint)
    (subset of above)

  rpc/
    tribrpc/                       TribServer RPC helpers/constants
    librpc/                        Libstore RPC helpers/constants
    storagerpc/                    StorageServer RPC helpers/constants

  sh_cp/                           Shell scripts to run the test for a checkpoint implementation
  sh/                              Shell scripts to run the tests for a final implementation

To and compile your code, execute one or more of the following commands from src/github.com/cmu440/tribbler:

go install github.com/cmu440/tribbler/runners/srunner
go install github.com/cmu440/tribbler/runners/lrunner
go install github.com/cmu440/tribbler/runners/trunner
go install github.com/cmu440/tribbler/runners/crunner

Check that the resulting binaries are located in the $GOPATH/bin directory: this should be the case if $GOBIN has never been set. If you have set $GOBIN to some other location, you need to also point $GOBIN to $GOPATH/bin. To simply check that your code compiles (i.e. without creating the binaries), you can use the go build subcommand to compile an individual package as shown below:

# Build/compile the "tribserver" package from tribbler/.
go build ./tribserver

# A different way to build/compile the "tribserver" package.
go build github.com/cmu440/tribbler/tribserver

If at any point you have any trouble with building, installing, or testing your code, the article titled How to Write Go Code is a great resource for understanding how Go workspaces are built and organized. You might also find the documentation for the go command to be helpful. As always, feel free to post your questions on Piazza.

To run and test the individual components that make up the Tribbler system, we have provided four simple programs that aim to simplify the process. The programs are located in the P3/src/github.com/cmu440/tribbler/runners/ directory and may be executed from anywhere on your system (after go install-ing them as discussed above). Each program is discussed individually below:

The srunner (StorageServer-runner) program creates and runs an instance of your StorageServer implementation. Some example usage is provided below (this assumes you are in the $GOPATH/bin directory; note you will have to run the commands in separate terminals or in the background with &):

# Start a single master storage server on port 9009.
./srunner -port=9009

# Start the master on port 9009 and run two additional slaves.
./srunner -port=9009 -N=3
./srunner -port=9008 -master="localhost:9009"
./srunner -port=9007 -master="localhost:9009"

For additional usage instructions, please execute ./srunner -help or consult the srunner.go source code.

The lrunner (Libstore-runner) program creates and runs an instance of your Libstore implementation. It enables you to execute Libstore methods from the command line, as shown in the example below:

# Create one (or more) storage servers in the background.
./srunner -port=9009 &

# Execute Put("thom", "yorke")
./lrunner -port=9009 p thom yorke
OK

# Execute Get("thom")
./lrunner -port=9009 g thom
yorke

# Execute Get("jonny")
./lrunner -port=9009 g jonny
ERROR: Get operation failed with status KeyNotFound

Note that the exact error messages that are output by the lrunner program may differ depending on how you implement your Libstore. For additional usage instructions, please execute ./lrunner -help or consult the lrunner.go source code.

The trunner (TribServer-runner) program creates and runs an instance of your TribServer implementation. For usage instructions, please execute ./trunner -help or consult the trunner.go source code. A simple example is:

./trunner -master localhost:9001

In order to use this program for your own personal testing, your Libstore implementation must function properly and one or more storage servers (i.e. srunner programs) must be running in the background.

The crunner (TribClient-runner) program creates and runs an instance of the TribClient implementation we have provided as part of the starter code. For usage instructions, please execute ./crunner -help or consult the crunner.go source code. As with the above programs, you'll need to start one or more Tribbler servers and storage servers beforehand so that the TribClient will have someone to communicate with.

Last but not least, we have also provided pre-compiled binaries (i.e. they were compiled against our own reference solutions) for each of the programs discussed above. The binaries are located in the P3/sols/ directory and have been compiled against both 64-bit Mac OS X and Linux machines. Similar to the staff-compled binaries we provided in project 1, we hope these will help you test the individual components of your Tribbler system.

The tests for the checkpoint are provided as bash shell scripts in the P3/src/github.com/cmu440/tribbler/sh_cp directory. For example, to run the libtest.sh test, simply execute the following from tribbler/:

./sh_cp/libtest.sh

If you get a permission issue, you can change the file permission by running chmod -R +x sh_cp. Note that these bash scripts link against both your own implementations as well as the test code located in the P3/src/github.com/cmu440/tribbler/tests_cp/ directory. Note that a few of these tests will also run against the staff-solution binaries discussed above, thus enabling us to test the correctness of individual components of your system as opposed to your entire Tribbler system as a whole.

To run all checkpoint tests, execute

./sh_cp/runall.sh

The tests for the whole project are provided as bash shell scripts in the P3/tests directory. For example, to run the libtest.sh test, simply execute the following from tribbler/:

./sh/libtest.sh

If you get a permission issue, you can change the file permission by running chmod -R +x sh_cp. Note that these bash scripts link against both your own implementations as well as the test code located in the tribbler/tests/ directory. Similarly, a few of these tests will also run against the staff-solution binaries discussed above, thus enabling us to test the correctness of individual components of your system as opposed to your entire Tribbler system as a whole.

To run all final tests, execute

./sh/runall.sh

If you and your partner are still confused about the behavior of the testing scripts (even after you've analyzed its source code), please don't hesitate to ask us a question on Piazza!

To submit your code to Gradescope, create a tribbler.zip file containing your implementation as follows:

cd $GOPATH/src/github.com/cmu440
zip -r tribbler.zip tribbler

Note: if tribbler.zip already exists and contains files that you have since deleted from your filesystem, the above command will not remove those files from tribbler.zip. Instead, you will need to delete tribbler.zip and run the command again. This is a common source of mysterious go fmt failures on Gradescope.

Before you begin the project, you should read and understand all of the starter code we provide. To make this experience a little less traumatic, fire up a web server and read the documentation in a browser by executing the following command:

godoc -goroot=$GOPATH &

If you don't have godoc already, you may have to run:

go get -v golang.org/x/tools/cmd/godoc

Then, navigate to localhost:6060/pkg/github.com/cmu440/tribbler in a browser (note that you can execute this command from anywhere in your system, assuming your $GOPATH is set correctly).