diff --git a/blob5001.tgz b/blob5001.tgz new file mode 100644 index 0000000000000000000000000000000000000000..c2e643e8573c83ef1d97714067d0866d16be5725 Binary files /dev/null and b/blob5001.tgz differ diff --git a/p4.md b/p4.md index 32c41dc8d41ef89008e4973558e787754e00a1a6..8bc3e6d1755a22908ae83200cdc0f5d8c8a26726 100644 --- a/p4.md +++ b/p4.md @@ -175,7 +175,12 @@ Pulled by :5558 from localhost:5557: 0 blobs using 1 RPCs, 0.044928375 secs Note that this is show counts of blobs and RPCs used for each pull request, though the video does not. Yours should do this. + ### Server command line args: +- **-b <dir>** : Directory where the blobdirectory should be + placed. CHANGES: your blob directory should be called + `<dir>/blob<port>`, where the dir is specified like `-b /tmp/`, and + the port is specified w/ `-s <port>`. For example: "/tmp/blob5001". - **-s <serverport>** : Bare integer. - **-S <serv1:port1,serv2:port2...>** : Comma-separated list of <servname:portno>. If specified, this takes precendence over @@ -183,9 +188,9 @@ request, though the video does not. Yours should do this. debugging, whereas the server names from `compose.yaml` are used in the containerization test. - **-p <period>**: Changes the anti-entropy period. -- **-a**: The code you use to implement *pull* requests should spin - off new gothreads to handle *path* and *get* calls in parallel. This - should significantly improve your performance w/o impacting correctness. +- **-D <treedepth>**: Changes the tree depth. Remember that a depth + *N* counds all levels, including root and leaves (i.e. height+1). + ### Antientropy @@ -319,21 +324,218 @@ counts of blobs pulled and RPCs as described above. ## Notes - The maximum size of gRPC message is 4MB by default. We should not be running up against this limit in my testing. +- `async` NOT TESTED. + +## Testing + +The following will be my testing procedure. Yours should look very +similar, and accept the same command line arguments as below, or you +will not get credit. + + +First thing we test is probing your merkle tree. To make this concrete, grab [this file](blob5001.tgz), and then untar it in `/tmp/`, so that you now have a directory `/tmp/blob5001` that has 95 blobs in it. + +**Test 1 (10 pts):** Then fire up an ubiserver, which should NOT wipe out the blobdir you just uncompressed, and instead should treat it as its own. +``` +io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 & +[1] 46194 +io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5002] +serving gRPC on :5001, host "io.lan", servers [localhost:5002], /tmp/blob5001/ + +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc + 96 97 6373 +``` + +**Test 2 (10 pts):** The above showed that your ubiserver can find my blobs. Next, create and probe a merkle tree: +``` +io:~/818/projects/p4/solution> go run cli.go -s :5001 build +95-sig tree on :5001: sha256_32_MEKYINRK35PJ67GPQZRFM2X4KUOTOU756ZJWNDCCY6U4LYEHN3YQ==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 path last E +sigs: 3 +combined: sha256_32_2VAB4CK4VUVO36JU5HPLONSF7MGG64DPMN4TRFQURUC5ZOOSBYRA==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 path last EM +sigs: 2 +combined: sha256_32_CN757EJBFCJFXNORM5QSS76GGB5TC2QAKTE3GV5MAQZAQDSG3UQQ==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 path last EMW +sigs: 1 +combined: sha256_32_B5XMMYDZ2WFBHXRRICT7WMRSYEMTYGINA6UBVICJAAU6GPJ2RSBA==== + "sha256_32_EMWG6JUKVXSLYUE4UWUR6HJIITXFJFF4S5DGQLEJ5QDLWFW3CSLQ====" +``` + +**Test 3 (10 pts):** Now do the same w/ a tree of height 2: +``` +killall ubiserver +io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 -D 2 & +[1] 47308 +io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5002] +serving gRPC on :5001, host "io.lan", servers [localhost:5002], /tmp/blob5001/ + +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc + 96 97 6373 +io:~/818/projects/p4/solution> go run cli.go -s :5001 build +95-sig tree on :5001: sha256_32_75LWMMDVKOOBFSXIFSDYFSP6PELM3YUZ4YKTHYH4LD5HAIITLN4A==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 path last "" +sigs: 95 +combined: sha256_32_75LWMMDVKOOBFSXIFSDYFSP6PELM3YUZ4YKTHYH4LD5HAIITLN4A==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 path last E +sigs: 3 +combined: sha256_32_N3C5MBRDCCJLLX3BBPEOUU2TVWSOMOD7DLN5DYILHEG52EVBMSUA==== + "sha256_32_EMORFEZQVOUTK5JX66U7KR3M4IEHQK6WKV2DY6FBZORBSHKDQF6Q====" + "sha256_32_EMWG6JUKVXSLYUE4UWUR6HJIITXFJFF4S5DGQLEJ5QDLWFW3CSLQ====" + "sha256_32_ETBRHI7Y53CYS6IRZW5GETFFK3MG4RBD66JYJEQZS3HBNZ7SY3QQ====" +``` + +**Test 4 (10 pts):** Now start up another ubiserver and see if we can sync: +``` +io:~/818/projects/p4/solution> killall ubiserver +No matching processes belonging to you were found +io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 & +[1] 47967 +io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5002] +serving gRPC on :5001, host "io.lan", servers [localhost:5002], /tmp/blob5001/ + +io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 1000 & +[2] 47998 +io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5001] +serving gRPC on :5002, host "io.lan", servers [localhost:5001], /tmp/blob5002/ + +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc + 96 97 6373 +io:~/818/projects/p4/solution> go run cli.go -s :5002 list | wc + 1 2 7 +io:~/818/projects/p4/solution> go run cli.go -s :5001 pull "localhost:5002" +Pulled by :5001 from localhost:5002: 0 blobs using 215 RPCs, 0.079132334 secs +io:~/818/projects/p4/solution> go run cli.go -s :5002 pull "localhost:5001" +Pulled by :5002 from localhost:5001: 95 blobs using 215 RPCs, 0.096492958 secs +io:~/818/projects/p4/solution> go run cli.go -s :5002 pull "localhost:5001" +Pulled by :5002 from localhost:5001: 0 blobs using 1 RPCs, 0.054424625 secs +io:~/818/projects/p4/solution> go run cli.go -s :5001 pull "localhost:5002" +Pulled by :5001 from localhost:5002: 0 blobs using 1 RPCs, 0.0569965 secs +``` +We: +- verified that the blobs were only in 5001's blobstore +- tried to pull from 5002 (empty) to 5001. This took many RPC even + though it received nothing, as every level of the trees are + different. +- pulled from 5001 to 5002 and saw many blobs moving, still w/ many + rpcs +- repeated the above, this time took only 1 RPC to see there was + nothing needed from 5001 +- pulling from 5002 to 5001 brought nothing, but only took a single + RPC this time + + +**Test 5 (15 pts):** Last, we need to check anti entropy. The above had anti-entropy turned +off by setting a period of 20 minutes, so the first anti-entropy never +happened. +``` +io:~/818/projects/p4/solution> killall ubiserver +No matching processes belonging to you were found +io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 10 & +[1] 49441 +io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5001] +serving gRPC on :5002, host "io.lan", servers [localhost:5001], /tmp/blob5002/ + go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 10 & +[2] 49467 +io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5002] +serving gRPC on :5001, host "io.lan", servers [localhost:5002], /tmp/blob5001/ + +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + 1 2 7 + 1 2 7 +``` + +Both sides are empty. Now read sampledir into 5001: +``` +io:~/818/projects/p4/solution> go run cli.go -a -s :5001 put sampledir +sha256_32_ITNRHUSBBLCJFBHKUB2ESG24XNUUPF3KH34CRO22JWOIZPZRC5OA==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + 96 97 6373 + 82 83 5435 +io:~/818/projects/p4/solution> pulled 95 blobs using 214 rpcs from localhost:5001 +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + 96 97 6373 + 96 97 6373 +``` + +**Test 6 (15 pts):** Now load a real big directory into the 5002 and make sure that 5001 +can grab it as well: +``` +io:~/818/projects/p4/solution> go run cli.go -a -s :5002 put sampledis-7.2.1 +sha256_32_UGPT64YNNMEPL3SVEJJHCR2M5QC3EYD5YKOEPOSFJVDJRVSTFYEA==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + 96 97 6373 + 4715 4716 315848 +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + 96 97 6373 + 4715 4716 315848 +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +pulled 4619 blobs using 5361 rpcs from localhost:5002 + 4715 4716 315848 + 4715 4716 315848 +``` + +**Test 7 (30 pts):** + +I will then verify that the containerization works. With the +`Dockerfile` and `compose.yaml` you have been given, I will bring up +three hosts: +``` + docker compose up +``` + +In another window I will load `sampledir` into one container and +`sampledis-7.2.1` into another: +``` +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc; go run cli.go -s :5002 list | wc; go run cli.go -s :5003 list | wc + 1 2 7 + 1 2 7 + 1 2 7 +io:~/818/projects/p4/solution> go run cli.go -s :5001 put sampledir +sha256_32_ITNRHUSBBLCJFBHKUB2ESG24XNUUPF3KH34CRO22JWOIZPZRC5OA==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc + 96 97 6373 + 1 2 7 + 1 2 7 +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc + 96 97 6373 + 96 97 6373 + 96 97 6373 +io:~/818/projects/p4/solution> +io:~/818/projects/p4/solution> go run cli.go -s :5003 put sampledis-7.2.1 +sha256_32_UGPT64YNNMEPL3SVEJJHCR2M5QC3EYD5YKOEPOSFJVDJRVSTFYEA==== +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc + 96 97 6373 + 3209 3210 214946 + 4715 4716 315848 +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc + 96 97 6373 + 4715 4716 315848 + 4715 4716 315848 +io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc + 4715 4716 315848 + 4715 4716 315848 + 4715 4716 315848 +``` +Done! + +For your convenience, all these commands are condensed [here](p4cmds.txt). + ## Grading I will use the following rough guide to assign grades: -- 50 pts: creating merkle tree and allow correct navigating using - *build* and *path* commands from `cli`. -- 25 pts: anti-entropy -- 25 pts: containerization -- ?? pts: prometheus and grafana +- 70 pts: from the first six tests +- 30 pts: containerization -Create a `README.md` file in your `p4` directory describing -what works, what doesn't, and any divergences from this -document. +Bonus! +- prometheus and grafana ## Video walkthrough  ## Submit +Create a `README.md` file in your `p4` directory describing +what works, what doesn't, and any divergences from this +document. + Commit to the repository. diff --git a/p4cmds.txt b/p4cmds.txt new file mode 100644 index 0000000000000000000000000000000000000000..4c4f5a16cddd2e0014780c7d028c36af04a254f2 --- /dev/null +++ b/p4cmds.txt @@ -0,0 +1,54 @@ +# Test 1 +go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 & +go run cli.go -s :5001 list | wc + + +# Test 2 +go run cli.go -s :5001 build +go run cli.go -s :5001 path last E +go run cli.go -s :5001 path last EM +go run cli.go -s :5001 path last EMW + +# Test 3 +killall ubiserver +go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 -D 2 & +go run cli.go -s :5001 list | wc +go run cli.go -s :5001 build +go run cli.go -s :5001 path last "" +go run cli.go -s :5001 path last E + +# Test 4 +killall ubiserver +go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 & +go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 1000 & +go run cli.go -s :5001 list | wc +go run cli.go -s :5002 list | wc +go run cli.go -s :5001 pull "localhost:5002" +go run cli.go -s :5002 pull "localhost:5001" +go run cli.go -s :5002 pull "localhost:5001" +go run cli.go -s :5001 pull "localhost:5002" + +# Test 5 +killall ubiserver +go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 10 & +go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 10 & +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +go run cli.go -a -s :5001 put sampledir +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + +# Test 6 +go run cli.go -a -s :5002 put sampledis-7.2.1 +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + +# Test 7 +docker compose up +go run cli.go -s :5001 list | wc; go run cli.go -s :5002 list | wc; go run cli.go -s :5003 list | wc +go run cli.go -s :5001 put sampledir +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc +go run cli.go -s :5003 put sampledis-7.2.1 +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc + diff --git a/p4cmds.txt~ b/p4cmds.txt~ new file mode 100644 index 0000000000000000000000000000000000000000..76eaddd79d2a04c6849d874bd43e731429802b42 --- /dev/null +++ b/p4cmds.txt~ @@ -0,0 +1,55 @@ +# Test 1 +go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 & +go run cli.go -s :5001 list | wc + + +# Test 2 +go run cli.go -s :5001 build +go run cli.go -s :5001 path last E +go run cli.go -s :5001 path last EM +go run cli.go -s :5001 path last EMW + +# Test 3 +killall ubiserver +go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 -D 2 & +go run cli.go -s :5001 list | wc +go run cli.go -s :5001 build +go run cli.go -s :5001 path last "" +go run cli.go -s :5001 path last E + +# Test 4 +killall ubiserver +go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 & +go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 1000 & +go run cli.go -s :5001 list | wc +go run cli.go -s :5002 list | wc +go run cli.go -s :5001 pull "localhost:5002" +go run cli.go -s :5002 pull "localhost:5001" +go run cli.go -s :5002 pull "localhost:5001" +go run cli.go -s :5001 pull "localhost:5002" + +# Test 5 +killall ubiserver +go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 10 & +go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 10 & +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +go run cli.go -a -s :5001 put sampledir +sha256_32_ITNRHUSBBLCJFBHKUB2ESG24XNUUPF3KH34CRO22JWOIZPZRC5OA==== +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + +# Test 6 +go run cli.go -a -s :5002 put sampledis-7.2.1 +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc + +# Test 7 +docker compose up +go run cli.go -s :5001 list | wc; go run cli.go -s :5002 list | wc; go run cli.go -s :5003 list | wc +go run cli.go -s :5001 put sampledir +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc +go run cli.go -s :5003 put sampledis-7.2.1 +go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc +