p2.md

Project 2: Separation of concerns: RPCs

Due: Sept 24, 2023, 11:59:59 pm, v1.01

Recent changes

• Clarified that the server should listen to an address with just :<port>, no host.

Big Picture

This project requires you to use gRPC to split your code from Project 1 into a command-line interface and a blobserver. You will also build a discrete lockserver that responds to gRPC lock and unlock requests. the system.

gRPC

gRPC (go bindings) is a powerful, open-source remote procedure call (RPC) package that works across heterogeneity of machines and languages. gRPC interfaces are defined using protocol buffers, which are also used to serialize messages.

Setup

Download the starter files for this project here.

• Install the protobuf compiler.
• Install the golang bindings.
• Update your paths so the protoc compiler can find the plugins.

For example, on a mac:

   brew install protobuf`
   protoc --version     # at least 24.3
   
   go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
   go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2

   export PATH="$PATH:$(go env GOPATH)/bin    # for bash

Helloworld

You can download the entire grpc-go repo as specified in the quickstart guide, but it is not necessary for this project. The supplied files have a version of the helloworld example w/ altered go.mod and package statements to make it completely local. You can go through the quickstart helloworld example completely in this directory.

The command to rebuild the protocol buffers for the altered directory would be (from the helloworld directory):

 protoc --go_out=. --go_opt=paths=source_relative	\
    --go-grpc_out=. --go-grpc_opt=paths=source_relative	\
    pb/helloworld.proto

• Learn how gRPC works in Introduction to gRPC and Core concepts.
• Work through the Basics tutorial.
• Explore the API reference.

Big Picture

There are essentially two parts to this project, both of which that rely on using gRPCs.

The first part is taking your simple blob store and splitting it to a distinct blob store in p2/ubi/server.go and command-line tools p2/cli.go and p2/lock.go. The command-line tools understand files and directories, the server is just a dump blob server.

The second part is building a lock server that will allow shared and exclusive locking of any random string. You do not have to do locking of prefixes of the path.

Each of these locks should be implemented with sync.RWMutex, and acquire requests for paths already locked with incompatible modes would result in the gRPC request being blocked until the situation changes.

Details: gRPC protocol definition

pb/comm.proto will define the object services that your servers will respond to. I gave you the service definition, and the "message" definitions for the BlobPutRequest. Define the others similarly.

Then, compile the protocol w/ the protoc command in the comment. This will create files comm.pb.go and comm_grpc.pb.go. The latter defines the routines that use object reflection to generate communication stubs that call the functions you write in your server.

The service definition is:

service Blob {
    rpc Put (BlobPutRequest) returns (BlobPutReply) {}
    rpc Get (BlobGetRequest) returns (BlobGetReply) {}
    rpc List (BlobListRequest) returns (BlobListReply) {}

    rpc Lock (LockRequest) returns (LockReply) {}
    rpc Unlock (UnlockRequest) returns (UnlockReply) {}
}

You will need to define server routines corresponding to each of the above. For example, the put command function will look like:

func (s *GRPCserver) Put(ctx context.Context, in *pb.BlobPutRequest) (*pb.BlobPutReply, error) {

       ....
	   
	return &pb.BlobPutReply{Sig: sig, ErrorString: ""}, nil
}

Most of this stuff works seamlessly. One oddity is that when you start up your server (in server.go), the actual startup is here:

	lis, err := net.Listen("tcp", ServerAddress)
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	s := grpc.NewServer()
	pb.RegisterBlobServer(s, &GRPCserver{})

The GRPCserver is a placeholder structure that does not need to define any properties or methods, except the UnimplementedBlobServer method. From ubi/grpcServe.go

type GRPCserver struct {
	pb.UnimplementedBlobServer
}

Details: Locks

Implement the locks using Sync.RWMutex from the standard library. The path used to choose which lock to lock/unlock is an arbitrary string, and each path must have a distinct lock.

This means you need to create RWMutex's at runtime. You can store them in a map for easy lookup. Though I won't be testing high-speed concurrent locks, your implementation should work w/ even when multiple clients are trying to lock a path at the same time, and:

• the path hasn't been accessed before, so no lock is pre-existing
• you must ensure that only one RWMutex is create per lock (no overwriting!).

This is complicated by the fact that gRPC is multi-threaded; each incoming request is given it's own server go-thread and runs concurrently with all other requests

Command-line parameters

The server:

   go run server.go [-d] [-s <:port>] 

The clients:

   go run cli.go [-d] [-s <host:port>] (desc <sig>|get <sig> <destination>|put <source>|list|lock <path>|unlock <path>)
   go run lock.go [-d] [-s <host:port>] (lock <path>|rlock <path>|unlock <path>|runlock <path>)

Testing

The testscript from before is a good place to start, but you also need locking behavior identical as shown in the video.

Notes

• Use the context to set RPC timeouts to 60 seconds.
• Your server main should spawn off your http server in a new goroutine, and be always available.
• gRPC communication requires that the server listen to an address w/o the host, e.g. ":50050", in order to work across the network.

Grading

I will assign grades vaguely as follow:

• 70 pts - distributing the blobserver
• 30 pts - lockserver

Video walkthrough

p2 walkthrough

Submit

Submit by copying into your distro and pushing to gitlab.