The code we write in this chapter has real external dependencies that we need to get set up before we can start to build our system.
Be sure to check out the chapter notes at https://github.com/matryer/goblueprints if you get stuck on installing any of the dependencies.
In most cases, services such as mongod
and nsqd
will have to be started before we can run our programs. Since we are writing components of a distributed system, we will have to run each program at the same time, which is as simple as opening many terminal windows.
NSQ is a messaging queue that allows one program to send messages or events to another, or to many other programs running either locally on the same machine, or on different nodes connected by a network. NSQ guarantees the delivery of messages, which means it keeps undelivered messages cached until all interested parties have received them. This means that, even if we stop our counter
program, we won't miss any votes. You can contrast this capability with fire-and-forget message queues where information is deemed out-of-date, and therefore is forgotten if it isn't delivered in time, and where the sender of the messages doesn't care if the consumer received them or not.
A message queue abstraction allows you to have different components of a system running in different places, provided they have network connectivity to the queue. Your programs are decoupled from others; instead, your designs start to care about the ins and outs of specialized micro-services, rather than the flow of data through a monolithic program.
NSQ transfers raw bytes, which means it is up to us how we encode data into those bytes. For example, we could encode the data as JSON or in a binary format depending on our needs. In our case, we are going to send the vote option as a string without any additional encoding, since we are only sharing a single data field.
Open http://nsq.io/deployment/installing.html in a browser (or search install nsq
) and follow the instructions for your environment. You can either download pre-compiled binaries or build your own from the source. If you have homebrew installed, installing NSQ is as simple as typing:
brew install nsq
Once you have installed NSQ, you will need to add the bin
folder to your PATH
environment variable so that the tools are available in a terminal.
To validate that NSQ is properly installed, open a terminal and run nsqlookupd
; if the program successfully starts, you should see some output similar to the following:
nsqlookupd v0.2.27 (built w/go1.3) TCP: listening on [::]:4160 HTTP: listening on [::]:4161
We are going to use the default ports to interact with NSQ so take note of the TCP and HTTP ports listed in the output, as we will be referring to them in our code.
Press Ctrl + C to stop the process for now; we'll start them properly later.
The key tools from the NSQ install that we are going to use are nsqlookupd
and nsqd
. The nsqlookupd
program is a daemon that manages topology information about the distributed NSQ environment; it keeps track of all the nsqd
producers for specific topics and provides interfaces for clients to query such information. The nsqd
program is a daemon that does the heavy lifting for NSQ such as receiving, queuing, and delivering messages from and to interested parties. For more information and background on NSQ, visit http://nsq.io/.
MongoDB is a document database, which basically allows you to store and query JSON documents and the data within them. Each document goes into a collection that can be used to group the documents together without enforcing any schema on the data inside them. Unlike rows in a traditional RDBMS such as Oracle, Microsoft SQL Server, or MySQL, it is perfectly acceptable for documents to have a different shape. For example, a people
collection can contain the following three JSON documents at the same time:
{"name":"Mat","lang":"en","points":57} {"name":"Laurie","position":"Scrum Master"} {"position":"Traditional Manager","exists":false}
This flexibility allows data with varying structure to coexist without impacting performance or wasting space. It is also extremely useful if you expect your software to evolve over time, as we really always should.
MongoDB was designed to scale while also remaining very easy to work with on single-box install such as our development machine. When we host our application for production, we would likely install a more complex multi-sharded, replicated system, which is distributed across many nodes and locations, but for now, just running mongod
will do.
Head over to http://www.mongodb.org/downloads to grab the latest version of MongoDB and install it, making sure to register the bin
folder with your PATH
environment variable as usual.
To validate that MongoDB is successfully installed, run the mongod
command, then hit Ctrl + C to stop it for now.
Gustavo Niemeyer has done a great job in simplifying interactions with MongoDB with his
mgo
(pronounced "mango") package hosted at http://labix.org/mgo, which is go gettable with the following command:
go get gopkg.in/mgo.v2
Now that we have all the pieces we need installed, we need to start our environment. In this section, we are going to:
nsqlookupd
so that our nsqd
instances are discoverablensqd
and tell it which nsqlookupd
to usemongod
for data servicesEach of these daemons should run in their own terminal window, which will make it easy for us to stop them by just hitting Ctrl + C.
In a terminal window, run:
nsqlookupd
Take note of the TCP port, which by default is 4160
, and in another terminal window, run:
nsqd --lookupd-tcp-address=localhost:4160
Make sure the port number in the --lookupd-tcp-address
flag matches the TCP port of the nsqlookupd
instance. Once you start nsqd
, you will notice some output is printed to the terminal from both nsqlookupd
and nsqd
; this indicates that the two processes are talking to each other.
In yet another window or tab, start MongoDB by running:
mongod --dbpath ./db
The dbpath
flag tells MongoDB where to store the data files for our database. You can pick any location you like, but you'll have to make sure the folder exists before mongod
will run.
Now that our environment is running, we are ready to start building our components.