ZMQ sockets
---------
(pebbe/zmq4)
"sockets on steroids"

"....yes, zmq sockets are the world-saving superheroes of the
networking world"

"0MQ looks like an embeddable networking library but acts like a
concurrency framework. It gives you sockets that carry atomic messages
across various transports like in-process, inter-process, TCP, and
multicast. You can connect sockets N-to-N with patterns like fan-out,
pub-sub, task distribution, and request-reply. It's fast enough to be
the fabric for clustered products. Its asynchronous I/O model gives
you scalable multicore applications, built as asynchronous
message-processing tasks. It has a score of language APIs and runs on
most operating systems. ØMQ is from iMatix and is LGPLv3 open source."

#### Big things:

- sockets over TCP, in-mem, .....
- N-to-N patterns, fan-out, pub-sub, task distribution...
- fast
- async (I/O in bg threads)
- bindings for all languages
- use protoBufs (or JSON) and all your agents diff languages

#### Small things:

- order 2 ends set up doesn't matter (normal sockets pain in the ass!)
- temp disconnects don't matter
- msgs buffered, not lost (except PUB-SUB)
- msgs sent to non-existant host no prob
- N-to-N!

#### Details:

- bind/conn to port/machine
- has all sorts of crypto, we ignore

## Msg types:

#### REQ-REP

- REQ only send, REP only receive
- either can be connected to multiple endpoints
- REP goes to right place
- must alternate

#### PUB

- best effort send

#### SUB

- must set filter: 	sock.SetSubscribe("")
- might lose msgs before startup

#### INPROC

- fast memory between threads in a proc (but channels)

Caveats:

- only contexts threadsafe, sockets are not

## Project Comm 

RPC's:

- broadcast of new Dirs  (Flusher      to FlushHandler)
- request of prior Dirs  (FlushHandler to Replyer)
- request of chunks      (Fuse         to Replyer)

Goroutines:

- main (there are many, but let's ignore that)
- flusher
- flushhandler
- replyer

Who does who:

     flusher (PUB)      to flushHandler(SUB)
     flushHandler (REQ) to replyer (REP)
     main (REQ)         to replyer(REP)

           flusher  flushHandler
           /        /      \
          /        /        \
     main/--------/  ->->    replyer


Details
---------
all I flush are:

- dirs
- chunks
- "head"

Can (mostly) just send to to other replicas:

    const (
    	MSG_FLUSH = iota
    	MSG_CONSOLE
    	MSG_REQ_BLOCKS
    	MSG_REP_BLOCKS
    	MSG_REQ_DIRS
    	MSG_REP_DIRS
    )
    
    type Message struct {
    	Mtype int
    	From  int
    	Pairs []EncodePair
    }

json
