Skip to content
Snippets Groups Projects
Commit 0d176064 authored by Peter J. Keleher's avatar Peter J. Keleher
Browse files

auto

parent 62c42cc4
No related branches found
No related tags found
No related merge requests found
File added
......@@ -175,7 +175,12 @@ Pulled by :5558 from localhost:5557: 0 blobs using 1 RPCs, 0.044928375 secs
Note that this is show counts of blobs and RPCs used for each pull
request, though the video does not. Yours should do this.
### Server command line args:
- **-b <dir>** : Directory where the blobdirectory should be
placed. CHANGES: your blob directory should be called
`<dir>/blob<port>`, where the dir is specified like `-b /tmp/`, and
the port is specified w/ `-s <port>`. For example: "/tmp/blob5001".
- **-s <serverport>** : Bare integer.
- **-S <serv1:port1,serv2:port2...>** : Comma-separated list of
<servname:portno>. If specified, this takes precendence over
......@@ -183,9 +188,9 @@ request, though the video does not. Yours should do this.
debugging, whereas the server names from `compose.yaml` are
used in the containerization test.
- **-p <period>**: Changes the anti-entropy period.
- **-a**: The code you use to implement *pull* requests should spin
off new gothreads to handle *path* and *get* calls in parallel. This
should significantly improve your performance w/o impacting correctness.
- **-D <treedepth>**: Changes the tree depth. Remember that a depth
*N* counds all levels, including root and leaves (i.e. height+1).
### Antientropy
......@@ -319,21 +324,218 @@ counts of blobs pulled and RPCs as described above.
## Notes
- The maximum size of gRPC message is 4MB by default. We should not be
running up against this limit in my testing.
- `async` NOT TESTED.
## Testing
The following will be my testing procedure. Yours should look very
similar, and accept the same command line arguments as below, or you
will not get credit.
First thing we test is probing your merkle tree. To make this concrete, grab [this file](blob5001.tgz), and then untar it in `/tmp/`, so that you now have a directory `/tmp/blob5001` that has 95 blobs in it.
**Test 1 (10 pts):** Then fire up an ubiserver, which should NOT wipe out the blobdir you just uncompressed, and instead should treat it as its own.
```
io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 &
[1] 46194
io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5002]
serving gRPC on :5001, host "io.lan", servers [localhost:5002], /tmp/blob5001/
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc
96 97 6373
```
**Test 2 (10 pts):** The above showed that your ubiserver can find my blobs. Next, create and probe a merkle tree:
```
io:~/818/projects/p4/solution> go run cli.go -s :5001 build
95-sig tree on :5001: sha256_32_MEKYINRK35PJ67GPQZRFM2X4KUOTOU756ZJWNDCCY6U4LYEHN3YQ====
io:~/818/projects/p4/solution> go run cli.go -s :5001 path last E
sigs: 3
combined: sha256_32_2VAB4CK4VUVO36JU5HPLONSF7MGG64DPMN4TRFQURUC5ZOOSBYRA====
io:~/818/projects/p4/solution> go run cli.go -s :5001 path last EM
sigs: 2
combined: sha256_32_CN757EJBFCJFXNORM5QSS76GGB5TC2QAKTE3GV5MAQZAQDSG3UQQ====
io:~/818/projects/p4/solution> go run cli.go -s :5001 path last EMW
sigs: 1
combined: sha256_32_B5XMMYDZ2WFBHXRRICT7WMRSYEMTYGINA6UBVICJAAU6GPJ2RSBA====
"sha256_32_EMWG6JUKVXSLYUE4UWUR6HJIITXFJFF4S5DGQLEJ5QDLWFW3CSLQ===="
```
**Test 3 (10 pts):** Now do the same w/ a tree of height 2:
```
killall ubiserver
io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 -D 2 &
[1] 47308
io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5002]
serving gRPC on :5001, host "io.lan", servers [localhost:5002], /tmp/blob5001/
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc
96 97 6373
io:~/818/projects/p4/solution> go run cli.go -s :5001 build
95-sig tree on :5001: sha256_32_75LWMMDVKOOBFSXIFSDYFSP6PELM3YUZ4YKTHYH4LD5HAIITLN4A====
io:~/818/projects/p4/solution> go run cli.go -s :5001 path last ""
sigs: 95
combined: sha256_32_75LWMMDVKOOBFSXIFSDYFSP6PELM3YUZ4YKTHYH4LD5HAIITLN4A====
io:~/818/projects/p4/solution> go run cli.go -s :5001 path last E
sigs: 3
combined: sha256_32_N3C5MBRDCCJLLX3BBPEOUU2TVWSOMOD7DLN5DYILHEG52EVBMSUA====
"sha256_32_EMORFEZQVOUTK5JX66U7KR3M4IEHQK6WKV2DY6FBZORBSHKDQF6Q===="
"sha256_32_EMWG6JUKVXSLYUE4UWUR6HJIITXFJFF4S5DGQLEJ5QDLWFW3CSLQ===="
"sha256_32_ETBRHI7Y53CYS6IRZW5GETFFK3MG4RBD66JYJEQZS3HBNZ7SY3QQ===="
```
**Test 4 (10 pts):** Now start up another ubiserver and see if we can sync:
```
io:~/818/projects/p4/solution> killall ubiserver
No matching processes belonging to you were found
io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 &
[1] 47967
io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5002]
serving gRPC on :5001, host "io.lan", servers [localhost:5002], /tmp/blob5001/
io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 1000 &
[2] 47998
io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5001]
serving gRPC on :5002, host "io.lan", servers [localhost:5001], /tmp/blob5002/
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc
96 97 6373
io:~/818/projects/p4/solution> go run cli.go -s :5002 list | wc
1 2 7
io:~/818/projects/p4/solution> go run cli.go -s :5001 pull "localhost:5002"
Pulled by :5001 from localhost:5002: 0 blobs using 215 RPCs, 0.079132334 secs
io:~/818/projects/p4/solution> go run cli.go -s :5002 pull "localhost:5001"
Pulled by :5002 from localhost:5001: 95 blobs using 215 RPCs, 0.096492958 secs
io:~/818/projects/p4/solution> go run cli.go -s :5002 pull "localhost:5001"
Pulled by :5002 from localhost:5001: 0 blobs using 1 RPCs, 0.054424625 secs
io:~/818/projects/p4/solution> go run cli.go -s :5001 pull "localhost:5002"
Pulled by :5001 from localhost:5002: 0 blobs using 1 RPCs, 0.0569965 secs
```
We:
- verified that the blobs were only in 5001's blobstore
- tried to pull from 5002 (empty) to 5001. This took many RPC even
though it received nothing, as every level of the trees are
different.
- pulled from 5001 to 5002 and saw many blobs moving, still w/ many
rpcs
- repeated the above, this time took only 1 RPC to see there was
nothing needed from 5001
- pulling from 5002 to 5001 brought nothing, but only took a single
RPC this time
**Test 5 (15 pts):** Last, we need to check anti entropy. The above had anti-entropy turned
off by setting a period of 20 minutes, so the first anti-entropy never
happened.
```
io:~/818/projects/p4/solution> killall ubiserver
No matching processes belonging to you were found
io:~/818/projects/p4/solution> go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 10 &
[1] 49441
io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5001]
serving gRPC on :5002, host "io.lan", servers [localhost:5001], /tmp/blob5002/
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 10 &
[2] 49467
io:~/818/projects/p4/solution> And we are up..."io.lan"....[localhost:5002]
serving gRPC on :5001, host "io.lan", servers [localhost:5002], /tmp/blob5001/
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
1 2 7
1 2 7
```
Both sides are empty. Now read sampledir into 5001:
```
io:~/818/projects/p4/solution> go run cli.go -a -s :5001 put sampledir
sha256_32_ITNRHUSBBLCJFBHKUB2ESG24XNUUPF3KH34CRO22JWOIZPZRC5OA====
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
96 97 6373
82 83 5435
io:~/818/projects/p4/solution> pulled 95 blobs using 214 rpcs from localhost:5001
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
96 97 6373
96 97 6373
```
**Test 6 (15 pts):** Now load a real big directory into the 5002 and make sure that 5001
can grab it as well:
```
io:~/818/projects/p4/solution> go run cli.go -a -s :5002 put sampledis-7.2.1
sha256_32_UGPT64YNNMEPL3SVEJJHCR2M5QC3EYD5YKOEPOSFJVDJRVSTFYEA====
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
96 97 6373
4715 4716 315848
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
96 97 6373
4715 4716 315848
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
pulled 4619 blobs using 5361 rpcs from localhost:5002
4715 4716 315848
4715 4716 315848
```
**Test 7 (30 pts):**
I will then verify that the containerization works. With the
`Dockerfile` and `compose.yaml` you have been given, I will bring up
three hosts:
```
docker compose up
```
In another window I will load `sampledir` into one container and
`sampledis-7.2.1` into another:
```
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc; go run cli.go -s :5002 list | wc; go run cli.go -s :5003 list | wc
1 2 7
1 2 7
1 2 7
io:~/818/projects/p4/solution> go run cli.go -s :5001 put sampledir
sha256_32_ITNRHUSBBLCJFBHKUB2ESG24XNUUPF3KH34CRO22JWOIZPZRC5OA====
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
96 97 6373
1 2 7
1 2 7
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
96 97 6373
96 97 6373
96 97 6373
io:~/818/projects/p4/solution>
io:~/818/projects/p4/solution> go run cli.go -s :5003 put sampledis-7.2.1
sha256_32_UGPT64YNNMEPL3SVEJJHCR2M5QC3EYD5YKOEPOSFJVDJRVSTFYEA====
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
96 97 6373
3209 3210 214946
4715 4716 315848
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
96 97 6373
4715 4716 315848
4715 4716 315848
io:~/818/projects/p4/solution> go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
4715 4716 315848
4715 4716 315848
4715 4716 315848
```
Done!
For your convenience, all these commands are condensed [here](p4cmds.txt).
## Grading
I will use the following rough guide to assign grades:
- 50 pts: creating merkle tree and allow correct navigating using
*build* and *path* commands from `cli`.
- 25 pts: anti-entropy
- 25 pts: containerization
- ?? pts: prometheus and grafana
- 70 pts: from the first six tests
- 30 pts: containerization
Create a `README.md` file in your `p4` directory describing
what works, what doesn't, and any divergences from this
document.
Bonus!
- prometheus and grafana
## Video walkthrough
![p4 walkthrough](p4.mp4)
## Submit
Create a `README.md` file in your `p4` directory describing
what works, what doesn't, and any divergences from this
document.
Commit to the repository.
# Test 1
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 &
go run cli.go -s :5001 list | wc
# Test 2
go run cli.go -s :5001 build
go run cli.go -s :5001 path last E
go run cli.go -s :5001 path last EM
go run cli.go -s :5001 path last EMW
# Test 3
killall ubiserver
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 -D 2 &
go run cli.go -s :5001 list | wc
go run cli.go -s :5001 build
go run cli.go -s :5001 path last ""
go run cli.go -s :5001 path last E
# Test 4
killall ubiserver
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 &
go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 1000 &
go run cli.go -s :5001 list | wc
go run cli.go -s :5002 list | wc
go run cli.go -s :5001 pull "localhost:5002"
go run cli.go -s :5002 pull "localhost:5001"
go run cli.go -s :5002 pull "localhost:5001"
go run cli.go -s :5001 pull "localhost:5002"
# Test 5
killall ubiserver
go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 10 &
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 10 &
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
go run cli.go -a -s :5001 put sampledir
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
# Test 6
go run cli.go -a -s :5002 put sampledis-7.2.1
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
# Test 7
docker compose up
go run cli.go -s :5001 list | wc; go run cli.go -s :5002 list | wc; go run cli.go -s :5003 list | wc
go run cli.go -s :5001 put sampledir
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
go run cli.go -s :5003 put sampledis-7.2.1
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
# Test 1
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 &
go run cli.go -s :5001 list | wc
# Test 2
go run cli.go -s :5001 build
go run cli.go -s :5001 path last E
go run cli.go -s :5001 path last EM
go run cli.go -s :5001 path last EMW
# Test 3
killall ubiserver
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 -D 2 &
go run cli.go -s :5001 list | wc
go run cli.go -s :5001 build
go run cli.go -s :5001 path last ""
go run cli.go -s :5001 path last E
# Test 4
killall ubiserver
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 1000 &
go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 1000 &
go run cli.go -s :5001 list | wc
go run cli.go -s :5002 list | wc
go run cli.go -s :5001 pull "localhost:5002"
go run cli.go -s :5002 pull "localhost:5001"
go run cli.go -s :5002 pull "localhost:5001"
go run cli.go -s :5001 pull "localhost:5002"
# Test 5
killall ubiserver
go run ubiserver.go -b /tmp/blob -s :5002 -S localhost:5001 -p 10 &
go run ubiserver.go -b /tmp/blob -s :5001 -S localhost:5002 -p 10 &
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
go run cli.go -a -s :5001 put sampledir
sha256_32_ITNRHUSBBLCJFBHKUB2ESG24XNUUPF3KH34CRO22JWOIZPZRC5OA====
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
# Test 6
go run cli.go -a -s :5002 put sampledis-7.2.1
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc
# Test 7
docker compose up
go run cli.go -s :5001 list | wc; go run cli.go -s :5002 list | wc; go run cli.go -s :5003 list | wc
go run cli.go -s :5001 put sampledir
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
go run cli.go -s :5003 put sampledis-7.2.1
go run cli.go -s :5001 list | wc ; go run cli.go -s :5002 list | wc ; go run cli.go -s :5003 list | wc
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment