Skip to content
Snippets Groups Projects
Select Git revision
  • main default protected
1 result
Blame Permalink
Code owners
Assign users and groups as approvers for specific file changes. Learn more.
p3.md 15.48 KiB

Project 3: Crypto

Due: Oct 8, 2023, 11:59:59 pm, v1.02

v1.02 changes:

  • added putraw command and text describing how to verify a signed binding.
  • changed go run blob.go to cli. Create cli w/ go build -o cli cli.go.

Overview

This project has the following tasks:

  1. Generate public/private RSA key pairs pete.
  2. Use public-key crypto to sign and validate signatures
  3. Build signed bindings of permanent names to changing values.
  4. Add a new dump command to see the raw bindings.
  5. Add a new verifybinding command to verify bindings.

Setup

Download files here.

New Packages

The new crypto packages we will be using include:

  • pem: PEM (Privacy Enhanced Mail) encoding is the de facto format for storing and sending cryptographic keys and certificates. This is just encoding, like base32, and is not encryption.
  • x509: Widespread standard for public key certificate formats. We will not be using certificates, but we will be using routines from this package.
  • rand: random numbers
  • rsa: RSA (Rivest-Shamir-Adleman) public keys.

Task 1: Generate a public/private keypair

Signing, verifying, and using JSON blobs (including recipes) is going to be the centerpiece of this project. Public key crypto is asymmetric: they come in pairs with a public key and a private key. For this project, we will use the private key just to sign JSON blobs, and the public key to verify the signatures. As you might expect, public keys are not secret, in fact they need to be widely disseminated in order to be used. Private keys are secret.

  • cli genkeys: Should generate a 2048-bit RSA private key using the rsa routine GenerateKey, and write to output file key.private. Write the corresponding public key to key.public.

To write the public key:

  • create a pem.Block specifying Type "RSA PUBLIC KEY" and passing the results of x509.MarshalPKCS1PublicKey to Bytes.
  • encode this using pem.EncodeToMemory
  • write the results to a file.

The resulting public key should look something like this:

-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEAsIYRztz29n63uOAKjnvlmysxfu4YjOq4erx6FsAdHLp+8BLF3YjX
WbJB9IcSsik9oRQ9+Abv9jdc7qHz/gTmdxiBx55isbGWAjLeADSxorumQVqJqaXi
/PNATUCOozdOEzDjxv/OXxOc4uZqjj7i+AbncPFTt22WuD40U0UEKo8Uv4JJtryS
bYvvXTO2R6UEMusxhz2eKA1BdqIl4EEeNhjUpXEk9P6fJ6QXc2dm7sOP42gfhoL5
tLpXM9quAkpVv/L1vwIVpBZTgtsFHfHz06p7ST77+E6lua10km9EVcdfBlwo0uvf
7GAP/V9jBDVUrTfBlZIrKksm4sv27pmt0wIDAQAB
-----END RSA PUBLIC KEY-----

Write the private key out similarly, substituting "private" for "public" where appropriate:

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAsIYRztz29n63uOAKjnvlmysxfu4YjOq4erx6FsAdHLp+8BLF
3YjXWbJB9IcSsik9oRQ9+Abv9jdc7qHz/gTmdxiBx55isbGWAjLeADSxorumQVqJ
qaXi/PNATUCOozdOEzDjxv/OXxOc4uZqjj7i+AbncPFTt22WuD40U0UEKo8Uv4JJ
trySbYvvXTO2R6UEMusxhz2eKA1BdqIl4EEeNhjUpXEk9P6fJ6QXc2dm7sOP42gf
hoL5tLpXM9quAkpVv/L1vwIVpBZTgtsFHfHz06p7ST77+E6lua10km9EVcdfBlwo
0uvf7GAP/V9jBDVUrTfBlZIrKksm4sv27pmt0wIDAQABAoIBAQCjqNlv2guaA2gg
no8LBTGeSFCOZv8Z/zBkksP6vcj/7qRrvmv2db17KrrdlDGwPm4mHpdJklz+ANxH
Xp3RYAewWWdRhLXqG0SB61lKBZC/2IxOwTsqr2MgtAJk3TqhdlALmONFh7t1CMEG
2Svnj8RswjHn08aTJKZ9USbAIRe4v/7dgKalffpnro4gQAhcBHhQSqKD+cmUebXI
mOApIjz1waywHSS69DrzJpNG755kJBd4q5ZYwXA6m3jFDTrfI9RLD6gJDtoVpMiE
h293dRS9w/A564sCznUFjMWUQFcHB8PhlDG0SzjStO5eE3f845UjLdfa7HZW7d1S
wLGS7EMxAoGBANqcSPE0aEHyemjR3DcaYv5e2oBDtCDcRKP85PDSnp47haXo0nBe
+iTJL6BCg6sohgF0Ft3xdj87dgn05Mo2NyyXbsdmjj32uQwvMkVPtNCN9KU4FVL0
11M/1ruRsYa6NNQ61zXwj8/MwepM3tOFJU+rofsNp8gpBiyTTxevAwcrAoGBAM63
Cv/lV4EU7njpPt0aIoCQoOQZGreTlqdQ/GgdA/1K+1LCw3dsEhzHZmmkrqQm3Vcr
wPpx0uXPjygWVzyfhFn/nDz4iNyXile3Bi+gCux2iwAWDzoJHkfP7bSNupI1uyuz
cub+D9lxzD5s4FgD2TfNSDjzV9hmlYeJC2zfnJ/5AoGBAMN0Rhc6dxi3VlCPiafO
tMGfRxa08cELj2dbPco/Vcg9iZG75yLHGDl5k1ZjAdwvABkelS9cqw9/91qRlVli
PdRllIs9m2G1TN+i9vxXdl+c/CYYTaB8/mQVSMUtTx8ZLxCthytX6QyukpYMopFV
kvV3i/ytydxBKZ8DGg0f0cI/AoGAB64vX3Ci/q3LanyoFEj7TTGSeMciAf4e09qj
di7VzhxyGBIadx5x0dXqzTQMNRcolCuRAP0nq9g5ZnDmDt+SaFGh+XX2h9OtlTK8
rRpSLZT99yParvpVwK9OEq5NZ09NxALn2wNHjXm37/3VnA+Qi406CLup4OV22tFR
tlrck+kCgYA4Y9dre8u9NYgnK/hPW7xze5vsCKEkRaH8+rnP2jdWYBPs3luvhL+t
Ijm3EBrPsJSNfZMSRNtqeKsd/ZUxAxy/LXKwTP2k6WjpXkLUVlYkgn25M4YHZIes
nERuoaKxLGuM+xAfNole2SPaakNYeWizS+y1Dm5iquHNHHjWH56XtA==
-----END RSA PRIVATE KEY-----

Reading the keys back in requires:

  • reading data from the file
  • decoding the pem encoding
  • x509 parsing (rather than marshalling)

Task 2: Sign and Verify

Signing

Cryptographic signing is accomplished by computing a hash of the material to be signed, and encrypting that hash with a private key. We encrypt a hash rather than the actual text because public key crypto works only over small amounts of data, limited by key size.

For this project we want to include our signatures in the JSON text, meaning that we have a recursion problem: we can not sign something that includes the signature resulting from the signing. The answer is is that we will sign everything except the signature, and then just add the signature on the end.

We define signing on anything in JSON. Let's start with an example file recipe:

io> cli put blob.go
sha256_32_TFBQWYJ7DGDPXPMYUQI7XJCF73FYCB27DP75EPMLBAHXT4JYATEA====
io> cli desc last
{
    "Name": "blob.go",
    "Size": 3228,
    "Mode": 420,
    "ModTime": "2023-09-15T21:46:37.753398955-04:00",
    "IsDir": false,
    "Version": 1,
    "PrevSig": "",
    "ChildSigs": null,
    "ChildNames": null,
    "DataBlockSigs": [
        "sha256_32_ZXF7E5X2VWUXNMY2ERBBEVG2KEKCB6KYMVBYFDQ5EQTTQBOMEUUA===="
    ]
}

Signing this blob gives us:

io> cli -q key.private sign last
sha256_32_63W7EPA6FACLUPU6N2NPVE2IQBUXV7UKIYNXVHRZEKGZJG7EX3EA====
io> cli desc last
{
    "Name": "blob.go",
    "Size": 3228,
    "Mode": 420,
    "ModTime": "2023-09-15T21:46:37.753398955-04:00",
    "IsDir": false,
    "Version": 1,
    "PrevSig": "",
    "ChildSigs": null,
    "ChildNames": null,
    "DataBlockSigs": [
        "sha256_32_ZXF7E5X2VWUXNMY2ERBBEVG2KEKCB6KYMVBYFDQ5EQTTQBOMEUUA===="
    ]
,
"signature": "RMNDXPYJUASFBSYJ4XQDLLXG64SOEH32CTNMZE3VRXSRJXXHTZO74MRO3MBHRUJKSYMWRCFH22RB3PAGVU4PARQK223II2BFGDUBCMHRD5R4BOAOH2MK7LO446HRKVQHQX3ZT542DSTVSB6ZRA7ASSZLNXPS6WR3VKOED5OEGB7YG45X3LFTZPJUUBG7YRGT7NEXECCOPUNSETU2B2V45KGRSE3MNAU2E3WT62F36AH7YJR2IIVFECPBDJHMMQ7LW6EPQ5PHG4UXPRQJZ4GC37KHB4F44VQIOIMSMRFHQNILRM3DYW3KYQQUH2Y64TKCQAK6NSWCRGWAJO5RP32USYU7AJCDLKS6U6IW5GGKSPJLN55Y2ON2WKAVZVTEOUDMRPPZFVBTDQX63IIETRI53QZPKA======"
}

The signed blob differs only at the end. The procedure for generating the signature should be followed exactly:

  1. read in the private key
  2. trim all whitespace off the end of the JSON
  3. remove the last character (which should be '}')
  4. take the SHA256 hash
  5. call rsa.SignPKCS1v15 on the hash result: this gives the raw signature
  6. encode the signature using base32
  7. add ,\n\"signature\": \", the encoded signature, and "\n} to the end of the truncated original blob

Verification is mostly the same, but in reverse. Left as an exercise for the reader:

io> cli -p key.public verify last
Verification of "sha256_32_63W7EPA6FACLUPU6N2NPVE2IQBUXV7UKIYNXVHRZEKGZJG7EX3EA====" w/ key.public succeeded

Task 3: Mutable signed bindings

Our series of projects so far describes a write-only store. Blobs can be removed, but the blob corresponding to a hash can not be changed. We'd like to support permanent names, called bindings, that can point to different content over time.

For example, you might define a root binding that points to the current root of a file system stored in your blobstore. You might define a status binding that shows the current status of your system. In either case the binding binds the Name to a Value, where Value is the sig containing the current content.

type Binding struct {
	Name          "status"
	Value         "sha256_32_63W7f...."
	Date          "Tue Nov 10 23:00:00 UTC 2009"  // layout time.UnixDate 
	PublicKeyHash ".....something....."  // discussed below
}

Ordinary file recipes are protected in the sense that their names are derived from their content; there is no way to modify either without detection.

Since a binding is designed to evolve over time, we need to have a different way of securing it. We therefore sign the JSON Binding, and include the public key in the JSON. Including the public key has two uses:

  • First, public keys are publicly bound to individual principals, thereby identifying the binding's creator.
  • Second, the binding is signed by the included public key's private key counterpart. Therefore a binding can be validated by using the included public key to verify the signature.

The following command creates a binding:

  • -p <publickey.fname> -q <privatekey.fname> binding <raw binding sig>

Say, for example, that we wish a permanent link to our server's status page. We can create an initial page as follows:

io:p3> cli put status.html
sha256_32_ZQOZMQ7KQ3LHR3OCHSBIUIUITUM3HJRNGG6WMTXHSENUQN54PIZA====
io:p3> cli desc last
{
    "Name": "status.html",
    "Size": 75,
    "Mode": 420,
    "ModTime": "2023-09-15T21:46:37.754750286-04:00",
    "IsDir": false,
    "Version": 1,
    "PrevSig": "",
    "ChildSigs": null,
    "ChildNames": null,
    "DataBlockSigs": [
        "sha256_32_7JXCJT52ARJ7UWE5WOYUEHOP2LZOTSC6KFBZDOHIH7SIX37BQ7VA===="
    ]
}

This can also be retrieved at:

http://localhost:8000/sha256_32_7JXCJT52ARJ7UWE5WOYUEHOP2LZOTSC6KFBZDOHIH7SIX37BQ7VA====

The above work (w/ the http server listening to port 8000), but it's unwieldy and immutable. We can use a binding to make the status mutable and easier to type/remember:

io:p3> cli -p key.public -q key.private binding status last
status
io:p3> cli desc status
{
    "Name": "status.html",
    "Size": 75,
    "Mode": 420,
    "ModTime": "2023-09-15T21:46:37.754750286-04:00",
    "IsDir": false,
    "Version": 1,
    "PrevSig": "",
    "ChildSigs": null,
    "ChildNames": null,
    "DataBlockSigs": [
        "sha256_32_7JXCJT52ARJ7UWE5WOYUEHOP2LZOTSC6KFBZDOHIH7SIX37BQ7VA===="
    ]
}

Also, note that the binding is automatically resolved to it's Value. This means that http://localhost:8000/status now shows the current status page.

io:p3> curl 'localhost:8000/status'
<center>
<h1>Things are looking pretty Bleak.</h1>

We do not (yet) have a way to view an existing binding.

The public key property

The PublicKey property of the binding is the hash of the public key corresponding to the private key used to create the signature. A hash can be used to verify something, but it can not be reversed to get back the public key on which it is based.

Luckily, we have a means of mapping hashes back to hashed data: our store. Your code binding creation code, therefore, should also:

  • write the PEM-encoded public key to the store
  • put the resulting hash in the binding as the value for "PublicKeyHash"

New command: dump

Recall we can not view the binding we inserted using desc, as it will automatically resolve to the binding's value.

Create a new function dump. dump is identical to desc except that it retrieves blobs in a raw mode: returning the raw binding text rather than resolving through to the binding Value.

Implementation of the dump command requires an extra raw parameter to be added to BlobGetRequest. When raw is false, the server's local blob read should check to see if the blob is a Binding (format is JSON, and can successfully be unmarshalled into a Binding struct) and resolve this binding by returning the blob pointed to by the binding's Value property, rather than the binding itself.

If raw is true, the binding itself should be returned.

io:~/818/projects/p3/solution> cli dump status
{
    "Name": "status",
    "Value": "sha256_32_55IEZUD6FH4FHUP5YCGXIP7SDGHZDY2RCKEV3EU45JUBLABVOZUA====",
    "Date": "Sat Sep 30 21:42:21 EDT 2023",
    "PublicKeyHash": "sha256_32_VDFWNRKFDLDLFMN52A54NSYPPVKR2EXMUFGREHKNPGGCLIMY3SWQ===="
,
"signature": "MFPBBVFRDCZHGBW3OIPZ2WUX43K7HIF23CVAHKFW75BSWBKAXJETNKAMYU6GKZ53N2KWYJ76K56QTH2EZMA46D6Y6QB3QFKEFEWWBNIQBVL63LIZQEV4SBXONJCT3YNAL7GOE5BPCQHMAC7ROGQ7AW3NUEMWLM5S65UIWWOJ22GIOYQNJOUR6FVBI3X5X7I55TCDI3QP2NREQG77B3YNBD6ZQSBLBSYYYJARYMA5DGBFGCO7ZXUEFXKGNR75ZE6HOZPD7VKLQ7S76WDZI7H6DVD56NA5DZH5HNUA7E4ZDYSYACDTKWPREOS3RJMICQ6GAJ3HJ2WYYKJJEZ4EXAD4LSV4KYMVUAX3LED2OOYRVYHDN5C6BN73GRXN4A5UKR5QGZJ5JCTOW2XOEUFYTTGVUAL6VY======"}

New command: putraw

You can check your verification by using the following public key from above. Get the public key hash of the actual key file contents, not the recipe pointing to it, but defining a new command putraw <filename> that send a blob to the remote store, returns, and prints out the resulting hash.

This is not what we want, as this hash is of the public key file's recipe:

io:~/818/projects/p3/solution> go run cli put key.public
sha256_32_JUWSBINGL4PNJIKTUOSCJDXBHVBSDZYORGYTCCA52MP7W7QFZGIA====

This is what we want: the hash of the data in the key file itself:

io:~/818/projects/p3/solution> go run cli putraw key.public
sha256_32_VDFWNRKFDLDLFMN52A54NSYPPVKR2EXMUFGREHKNPGGCLIMY3SWQ====

Note that this matches the public key hash in the status binding above. You can verify the signature now by creating

Updating a binding

A binding is updated by replacing it with a new binding using the same name. We can update the current status shown above by overwriting the status binding with a new blob name:

io:p3> cli put status2.html
sha256_32_XDZOUHLABSFPCFT456UMXZKKGODPOS4YKOMIGQZOKLQREYTOHJMQ====
io:p3> cli -p key.public -q key.private binding status last
status
io:p3> curl 'localhost:8000/status'
<center>
<h1>Things are all better now. </h1>

Task 4: Verify bindings

A binding can be verified by using PublicKeyHash property to retrieve the public key from the blob store, and then using that public key to verify the binding's signature.

Do this by creating a new command: verifybinding <name>. Assuming the public key is in the store, as described above, you will not need to pass either key to the blob client at the command line.

io:~/818/projects/p3/solution> cli verifybinding status
Verification of "status" w/ key.public succeeded

More pedantically, verify a binding as follows:

  • retrieve the actual binding blob (not what it resolves to)
  • unmarshal a binding object from the blob
  • grab the object's PublicKeyHash property and retrieve the associated blob
  • write that blob to a file in /tmp
  • verify the signature on the binding JSON w/ the saved public key

Servers and Trust

Most people of servers as trusted, and with good reason. Trusted servers make life easier. However, even if you trust your server provider's intent, no security is unbreakable.

Our server in this project might be categorized as "trust but verify". Anything we get back from the server can be verified or checked for tampering. Simple blobs must match their hashes. Signed JSON recipes or bindings can be verified.

However, we have no way of checking the server is not throwing blobs away, or returning "blob not found" which it is actually there.

In this course we will be discussing a couple papers ("SUNDr", "Spork") that develop completely untrusted servers. But not trusting your server has consequences, both in terms of complexity and performance. In practice, most systems are somewhere in the middle. For example, we could require clients to authenticate with the server, or require servers to validate signed blobs before accepting them.

Video walkthrough

p3 walkthrough

Grading

I will give roughly even credit for each of the tasks.

Submit

Commit to the repository.