In this chapter, we will build five small programs that we will combine together at the end. The key features of the programs are as follows:
.com
and .net
) to the endFive programs might seem like a lot for one chapter, but don't forget how small entire programs can be in Go.
Our first program augments incoming words with some sugar terms in order to improve the odds of finding available names. Many companies use this approach to keep the core messaging consistent while being able to afford the .com
domain. For example, if we pass in the word chat
, it might pass out chatapp
; alternatively, if we pass in talk
, we may get back talk time
.
Go's math/rand
package allows us to break away from the predictability of computers to give a chance or opportunity to get involved in our program's process and make our solution feel a little more intelligent than it actually is.
To make our Sprinkle program work, we will:
bufio
package to scan input from stdin
and fmt.Println
to write output to stdout
math/rand
package to randomly select which transformation to apply to the word, such as appending "app" or prefixing the term with "get"In the $GOPATH/src
directory, create a new folder called sprinkle
and add a main.go
file containing the following code:
package main import ( "bufio" "fmt" "math/rand" "os" "strings" "time" ) const otherWord = "*" var transforms = []string{ otherWord, otherWord, otherWord, otherWord, otherWord + "app", otherWord + "site", otherWord + "time", "get" + otherWord, "go" + otherWord, "lets " + otherWord, } func main() { rand.Seed(time.Now().UTC().UnixNano()) s := bufio.NewScanner(os.Stdin) for s.Scan() { t := transforms[rand.Intn(len(transforms))] fmt.Println(strings.Replace(t, otherWord, s.Text(), -1)) } }
From now on, it is assumed that you will sort out the appropriate import
statements yourself.
The preceding code represents our complete Sprinkle program. It defines three things: a constant, a variable, and the obligatory main
function, which serves as the entry point to Sprinkle. The otherWord
constant string is a helpful token that allows us to specify where the original word should occur in each of our possible transformations. It lets us write code such as otherWord+"extra"
, which makes it clear that, in this particular case, we want to add the word extra to the end of the original word.
The possible transformations are stored in the transforms
variable that we declare as a slice of strings. In the preceding code, we defined a few different transformations such as adding app
to the end of a word or lets
before it. Feel free to add some more in there; the more creative, the better.
In the main
function, the first thing we do is use the current time as a random seed. Computers can't actually generate random numbers, but changing the seed number for the random algorithms gives the illusion that it can. We use the current time in nanoseconds because it's different each time the program is run (provided the system clock isn't being reset before each run).
We then create a bufio.Scanner
object (called bufio.NewScanner
) and tell it to read input from os.Stdin
, which represents the standard in stream. This will be a common pattern in our five programs since we are always going to read from standard in and write to standard out.
The bufio.Scanner
object actually takes io.Reader
as its input source, so there is a wide range of types that we could use here. If you were writing unit tests for this code, you could specify your own io.Reader
for the scanner to read from, removing the need for you to worry about simulating the standard input stream.
As the default case, the scanner allows us to read, one at a time, blocks of bytes separated by defined delimiters such as a carriage return and linefeed characters. We can specify our own split function for the scanner or use one of the options built in the standard library. For example, there is bufio.ScanWords
that scans individual words by breaking on whitespace rather than linefeeds. Since our design specifies that each line must contain a word (or a short phrase), the default line-by-line setting is ideal.
A call to the Scan
method tells the scanner to read the next block of bytes (the next line) from the input, and returns a bool
value indicating whether it found anything or not. This is how we are able to use it as the condition for the for
loop. While there is content to work on, Scan
returns true
and the body of the for
loop is executed, and when Scan
reaches the end of the input, it returns false
, and the loop is broken. The bytes that have been selected are stored in the Bytes
method of the scanner, and the handy Text
method that we use converts the []byte
slice into a string for us.
Inside the for
loop (so for each line of input), we use rand.Intn
to select a random item from the transforms
slice, and use strings.Replace
to insert the original word where the otherWord
string appears. Finally, we use fmt.Println
to print the output to the default standard output stream.
Let's build our program and play with it:
go build –o sprinkle ./sprinkle
Once the program is running, since we haven't piped any content in, or specified a source for it to read from, we will use the default behavior where it reads the user input from the terminal. Type in chat
and hit return. The scanner in our code notices the linefeed character at the end of the word and runs the code that transforms it, outputting the result. For example, if you type chat
a few times, you might see output like:
chat go chat chat lets chat chat chat app
Sprinkle never exits (meaning the Scan
method never returns false
to break the loop) because the terminal is still running; in normal execution, the in pipe will be closed by whatever program is generating the input. To stop the program, hit Ctrl + C.
Before we move on, let's try running Sprinkle specifying a different input source, we are going to use the echo
command to generate some content, and pipe it into our Sprinkle program using the pipe character:
echo "chat" | ./sprinkle
The program will randomly transform the word, print it out, and exit since the echo
command generates only one line of input before terminating and closing the pipe.
We have successfully completed our first program, which has a very simple but useful function, as we will see.
Some of the words that output from Sprinkle contain spaces and perhaps other characters that are not allowed in domains, so we are going to write a program, called Domainify, that converts a line of text into an acceptable domain segment and add an appropriate Top-level Domain (TLD) to the end. Alongside the sprinkle
folder, create a new one called domainify
, and add a main.go
file with the following code:
package main var tlds = []string{"com", "net"} const allowedChars = "abcdefghijklmnopqrstuvwxyz0123456789_-" func main() { rand.Seed(time.Now().UTC().UnixNano()) s := bufio.NewScanner(os.Stdin) for s.Scan() { text := strings.ToLower(s.Text()) var newText []rune for _, r := range text { if unicode.IsSpace(r) { r = '-' } if !strings.ContainsRune(allowedChars, r) { continue } newText = append(newText, r) } fmt.Println(string(newText) + "." + tlds[rand.Intn(len(tlds))]) } }
You will notice a few similarities between the Domainify and Sprinkle programs: we set the random seed using rand.Seed
, generate a NewScanner
method wrapping the os.Stdin
reader, and scan each line until there is no more input.
We then convert the text to lowercase and build up a new slice of rune
types called newText
. The rune
types consist only of characters that appear in the allowedChars
string, which strings.ContainsRune
lets us know. If rune
is a space that we determine by calling unicode.IsSpace
, we replace it with a hyphen, which is an acceptable practice in domain names.
Ranging over a string returns the index of each character and a rune
type, which is a numerical value (specifically int32
) representing the character itself. For more information about runes, characters, and strings, refer to http://blog.golang.org/strings.
Finally, we convert newText
from a []rune
slice to a string and add either .com
or .net
to the end before printing it out using fmt.Println
.
go build –o domainify ./domainify
Type in some of these options to see how domainify
reacts:
Monkey
Hello Domainify
"What's up?"
One (two) three!
You can see that, for example, One (two) three!
might yield one-two-three.com
.
We are now going to compose Sprinkle and Domainify to see them work together. In your terminal, navigate to the parent folder (probably $GOPATH/src
) of sprinkle
and domainify
, and run the following command:
./sprinkle/sprinkle | ./domainify/domainify
Here we ran the Sprinkle program and piped the output into the Domainify program. By default, sprinkle
uses the terminal as the input and domanify
outputs to the terminal. Try typing in chat
a few times again, and notice the output is similar to what Sprinkle was outputting previously, except now the words are acceptable for domain names. It is this piping between programs that allows us to compose command-line tools together.
Often domain names for common words such as chat
are already taken and a common solution is to play around with the vowels in the words. For example, we might remove the a
leaving cht
(which is actually less likely to be available), or add an a
to produce chaat
. While this clearly has no actual effect on coolness, it has become a popular, albeit slightly dated, way to secure domain names that still sound like the original word.
Our third program, Coolify, will allow us to play with the vowels of words that come in via the input, and write the modified versions to the output.
Create a new folder called coolify
alongside sprinkle
and domainify
, and create the
main.go
code file with the following code:
package main const ( duplicateVowel bool = true removeVowel bool = false ) func randBool() bool { return rand.Intn(2) == 0 } func main() { rand.Seed(time.Now().UTC().UnixNano()) s := bufio.NewScanner(os.Stdin) for s.Scan() { word := []byte(s.Text()) if randBool() { var vI int = -1 for i, char := range word { switch char { case 'a', 'e', 'i', 'o', 'u', 'A', 'E', 'I', 'O', 'U': if randBool() { vI = i } } } if vI >= 0 { switch randBool() { case duplicateVowel: word = append(word[:vI+1], word[vI:]...) case removeVowel: word = append(word[:vI], word[vI+1:]...) } } } fmt.Println(string(word)) } }
While the preceding Coolify code looks very similar to the codes of Sprinkle and Domainify, it is slightly more complicated. At the very top of the code we declare two constants, duplicateVowel
and removeVowel
, that help make Coolify code more readable. The switch
statement decides whether we duplicate or remove a vowel. Also, using these constants, we are able to express our intent very clearly, rather than using just true
or false
.
We then define the randBool
helper function that just randomly returns true
or false
by asking the rand
package to generate a random number, and checking whether if that number comes out as zero. It will be either 0
or 1
, so there's a 50/50 chance of it being true
.
The main
function for Coolify starts the same way as the main
functions for Sprinkle and Domainify—by setting the rand.Seed
method and creating a scanner of the standard input stream before executing the loop body for each line of input. We call randBool
first to decide whether we are even going to mutate a word or not, so Coolify will only affect half of the words passed through it.
We then iterate over each rune in the string and look for a vowel. If our randBool
method returns true
, we keep the index of the vowel character in the vI
variable. If not, we keep looking through the string for another vowel, which allows us to randomly select a vowel from the words rather than always modifying the same one.
Once we have selected a vowel, we then use randBool
again to randomly decide what action to take.
This is where the helpful constants come in; consider the following alternative switch statement:
switch randBool() { case true: word = append(word[:vI+1], word[vI:]...) case false: word = append(word[:vI], word[vI+1:]...) }
In the preceding code snippet, it's difficult to tell what is going on because true
and false
don't express any context. On the other hand, using duplicateVowel
and removeVowel
tells anyone reading the code what we mean by the result of randBool
.
The three dots following the slices cause each item to pass as a separate argument to the append
function. This is an idiomatic way of appending one slice to another. Inside the switch
case, we do some slice manipulation to either duplicate the vowel or remove it altogether. We are reslicing our []byte
slice and using the append
function to build a new one made up of sections of the original word. The following diagram shows which sections of the string we access in our code:
If we take the value blueprints
as an example word, and assume that our code selected the first e
character as the vowel (so that vI
is 3
), we can see what each new slice of word represents in this table:
Code |
Value |
Description |
---|---|---|
|
|
Describes a slice from the beginning of the word slice to the selected vowel. The |
|
|
Describes a slice starting at and including the selected vowel to the end of the slice. |
|
|
Describes a slice from the beginning of the word slice up to, but not including, the selected vowel. |
|
|
Describes a slice from the item following the selected vowel to the end of the slice. |
After we modify the word, we print it out using fmt.Println
.
Let's build Coolify and play with it to see what it can do:
go build –o coolify ./coolify
When Coolify is running, try typing blueprints
to see what sort of modifications it comes up with:
blueprnts bleprints bluepriints blueprnts blueprints bluprints
Let's see how Coolify plays with Sprinkle and Domainify by adding their names to our pipe chain. In the terminal, navigate back (using the cd
command) to the parent folder and run the following commands:
./coolify/coolify | ./sprinkle/sprinkle | ./domainify/domainify
We will first spice up a word with extra pieces and make it cooler by tweaking the vowels before finally transforming it into a valid domain name. Play around by typing in a few words and seeing what suggestions our code makes.
So far, our programs have only modified words, but to really bring our solution to life, we need to be able to integrate a third-party API that provides word synonyms. This allows us to suggest different domain names while retaining the original meaning. Unlike Sprinkle and Domainify, Synonyms will write out more than one response for each word given to it. Our architecture of piping programs together means this is no problem; in fact we do not even have to worry about it since each of the three programs is capable of reading multiple lines from the input source.
The Big Hugh Thesaurus at bighughlabs.com has a very clean and simple API that allows us to make a single HTTP GET
request in order to look up synonyms.
If in the future the API we are using changes or disappears (after all, this is the Internet!), you will find some options at https://github.com/matryer/goblueprints.
Before you can use the Big Hugh Thesaurus, you'll need an API key, which you can get by signing up to the service at http://words.bighugelabs.com/.
Your API key is a sensitive piece of configuration information that you won't want to share with others. We could store it as const
in our code, but that would not only mean we couldn't share our code without sharing our key (not good, especially if you love open source projects), but also, and perhaps more importantly, you would have to recompile your project if the key expires or if you want to use a different one.
A better solution is using an environment variable to store the key, as this will allow you to easily change it if you need to. You could also have different keys for different deployments; perhaps you have one key for development or testing and another for production. This way, you can set a specific key for a particular execution of code, so you can easily switch keys without having to change your system-level settings. Either way, different operating systems deal with environment variables in similar ways, so they are a perfect choice if you are writing cross-platform code.
Create a new environment variable called BHT_APIKEY
and set your API key as its value.
Making a request for http://words.bighugelabs.com/apisample.php?v=2&format=json in a web browser shows us what the structure of JSON response data looks like when finding synonyms for the word love:
{ "noun":{ "syn":[ "passion", "beloved", "dear" ] }, "verb":{ "syn":[ "love", "roll in the hay", "make out" ], "ant":[ "hate" ] } }
The real API returns a lot more actual words than what is printed here, but the structure is the important thing. It represents an object where the keys describe the types of words (verbs, nouns, and so on) and values are objects that contain arrays of strings keyed on syn
or ant
(for synonym and antonym respectively); it is the synonyms we are interested in.
To turn this JSON string data into something we can use in our code, we must decode it into structures of our own using capabilities found in the encoding/json
package. Because we're writing something that could be useful outside the scope of our project, we will consume the API in a reusable package rather than directly in our program code. Create a new folder called thesaurus
alongside your other program folders (in $GOPATH/src
) and insert the following code into a new bighugh.go
file:
package thesaurus import ( "encoding/json" "errors" "net/http" ) type BigHugh struct { APIKey string } type synonyms struct { Noun *words `json:"noun"` Verb *words `json:"verb"` } type words struct { Syn []string `json:"syn"` } func (b *BigHugh) Synonyms(term string) ([]string, error) { var syns []string response, err := http.Get("http://words.bighugelabs.com/api/2/" + b.APIKey + "/" + term + "/json") if err != nil { return syns, errors.New("bighugh: Failed when looking for synonyms for "" + term + """ + err.Error()) } var data synonyms defer response.Body.Close() if err := json.NewDecoder(response.Body).Decode(&data); err != nil { return syns, err } syns = append(syns, data.Noun.Syn...) syns = append(syns, data.Verb.Syn...) return syns, nil }
In the preceding code, the BigHugh
type we define houses the necessary API key and provides the Synonyms
method that will be responsible for doing the work of accessing the endpoint, parsing the response, and returning the results. The most interesting parts of this code are the synonyms
and words
structures. They describe the JSON response format in Go terms, namely an object containing noun and verb objects, which in turn contain a slice of strings in a variable called Syn
. The tags (strings in backticks following each field definition) tell the encoding/json
package which fields to map to which variables; this is required since we have given them different names.
Typically, JSON keys have lowercase names, but we have to use capitalized names in our structures so that the encoding/json
package knows that the fields exist. If we didn't, the package would simply ignore the fields. However, the types themselves (synonyms
and words
) do not need to be exported.
The Synonyms
method takes a term
argument and uses http.Get
to make a web request to the API endpoint in which the URL contains not only the API key value, but also the term
value itself. If the web request fails for some reason, we will make a call to log.Fatalln
, which writes the error out to the standard error stream and exits the program with a non-zero exit code (actually an exit code of 1
)—this indicates that an error has occurred.
If the web request is successful, we pass the response body (another io.Reader
) to the json.NewDecoder
method and ask it to decode the bytes into the data
variable that is of our synonyms
type. We defer the closing of the response body in order to keep memory clean before using Go's built-in append
function to concatenate both noun
and verb
synonyms to the syns
slice that we then return.
Although we have implemented the BigHugh
thesaurus, it isn't the only option out there, and we can express this by adding a Thesaurus
interface to our package. In the thesaurus
folder, create a new file called thesaurus.go
, and add the following interface definition to the file:
package thesaurus type Thesaurus interface { Synonyms(term string) ([]string, error) }
This simple interface just describes a method that takes a term
string and returns either a slice of strings containing the synonyms, or an error (if something goes wrong). Our BigHugh
structure already implements this interface, but now other users could add interchangeable implementations for other services, such as Dictionary.com or the Merriam-Webster Online service.
Next we are going to use this new package in a program. Change directory in terminal by backing up a level to $GOPATH/src
, create a new folder called synonyms
, and insert the following code into a new main.go
file you will place in that folder:
func main() { apiKey := os.Getenv("BHT_APIKEY") thesaurus := &thesaurus.BigHugh{APIKey: apiKey} s := bufio.NewScanner(os.Stdin) for s.Scan() { word := s.Text() syns, err := thesaurus.Synonyms(word) if err != nil { log.Fatalln("Failed when looking for synonyms for ""+word+""", err) } if len(syns) == 0 { log.Fatalln("Couldn't find any synonyms for "" + word + """) } for _, syn := range syns { fmt.Println(syn) } } }
When you manage your imports again, you will have written a complete program capable of looking up synonyms for words by integrating the Big Huge Thesaurus API.
In the preceding code, the first thing our main
function does is get the BHT_APIKEY
environment variable value via the os.Getenv
call. To bullet proof your code, you might consider double-checking to ensure this value is properly set, and report an error if it is not. For now, we will assume that everything is configured properly.
Next, the preceding code starts to look a little familiar since it scans each line of input again from os.Stdin
and calls the Synonyms
method to get a list of replacement words.
Let's build a program and see what kind of synonyms the API comes back with when we input the word chat
:
go build –o synonyms ./synonyms chat confab confabulation schmooze New World chat Old World chat conversation thrush wood warbler chew the fat shoot the breeze chitchat chatter
The results you get will most likely differ from what we have listed here since we're hitting a live API, but the important aspect here is that when we give a word or term as input to the program, it returns a list of synonyms as output, one per line.
By composing the four programs we have built so far in this chapter, we already have a useful tool for suggesting domain names. All we have to do now is run the programs while piping the output into input in the appropriate way. In a terminal, navigate to the parent folder and run the following single line:
./synonyms/synonyms | ./sprinkle/sprinkle | ./coolify/coolify | ./domainify/domainify
Because the synonyms
program is first in our list, it will receive the input from the terminal (whatever the user decides to type in). Similarly, because domainify
is last in the chain, it will print its output to the terminal for the user to see. At each step, the lines of words will be piped through the other programs, giving them each a chance to do their magic.
Type in some words to see some domain suggestions, for example, if you type chat
and hit return, you might see:
getcnfab.com confabulationtim.com getschmoozee.net schmosee.com neew-world-chatsite.net oold-world-chatsite.com conversatin.net new-world-warblersit.com gothrush.net lets-wood-wrbler.com chw-the-fat.com
The number of suggestions you get will actually depend on the number of synonyms, since it is the only program that generates more lines of output than we give it.
We still haven't solved our biggest problem—the fact that we have no idea whether the suggested domain names are actually available or not, so we still have to sit and type each of them into a website. In the next section, we will address this issue.
Our final program, Available, will connect to a WHOIS server to ask for details about domains passed into it—of course, if no details are returned, we can safely assume that the domain is available for purchase. Unfortunately, the WHOIS specification (see http://tools.ietf.org/html/rfc3912) is very small and contains no information about how a WHOIS server should reply when you ask it for details about a domain. This means programmatically parsing the response becomes a messy endeavor. To address this issue for now, we will integrate with only a single WHOIS server that we can be sure will have No match
somewhere in the response when it has no records for the domain.
A more robust solution might be to have a WHOIS interface with well-defined structures for the details, and perhaps an error message for the cases when the domain doesn't exist—with different implementations for different WHOIS servers. As you can imagine, it's quite a project; perfect for an open source effort.
Create a new folder called available
alongside the others in $GOPATH/src
and add a main.go
file in it containing the following function code:
func exists(domain string) (bool, error) { const whoisServer string = "com.whois-servers.net" conn, err := net.Dial("tcp", whoisServer+":43") if err != nil { return false, err } defer conn.Close() conn.Write([]byte(domain + " ")) scanner := bufio.NewScanner(conn) for scanner.Scan() { if strings.Contains(strings.ToLower(scanner.Text()), "no match") { return false, nil } } return true, nil }
The exists
function implements what little there is in the WHOIS specification by opening a connection to port 43
on the specified whoisServer
instance with a call to net.Dial
. We then defer the closing of the connection, which means that however the function exits (successfully or with an error, or even a panic), Close()
will still be called on the connection conn
. Once the connection is open, we simply write the domain followed by
(the carriage return and line feed characters). This is all the specification tells us, so we are on our own from now on.
Essentially, we are looking for some mention of no match in the response, and that is how we will decide whether a domain exists or not (exists
in this case is actually just asking the WHOIS server if it has a record for the domain we specified). We use our favorite bufio.Scanner
method to help us iterate over the lines in the response. Passing the connection into NewScanner
works because net.Conn
is actually an io.Reader
too. We use strings.ToLower
so we don't have to worry about case sensitivity, and strings.Contains
to see if any of the lines contains the no match text. If it does, we return false
(since the domain doesn't exist), otherwise we return true
.
The com.whois-servers.net
WHOIS service supports domain names for .com
and .net
, which is why the Domainify program only adds these types of domains. If you used a server that had WHOIS information for a wider selection of domains, you could add support for additional TLDs.
Let's add a main
function that uses our exists
function to check to see whether the incoming domains are available or not. The check mark and cross mark symbols in the following code are optional—if your terminal doesn't support them you are free to substitute them with simple Yes
and No
strings.
Add the following code to main.go
:
var marks = map[bool]string{true: "✔", false: "×"} func main() { s := bufio.NewScanner(os.Stdin) for s.Scan() { domain := s.Text() fmt.Print(domain, " ") exist, err := exists(domain) if err != nil { log.Fatalln(err) } fmt.Println(marks[!exist]) time.Sleep(1 * time.Second) } }
In the preceding code for the main
function, we simply iterate over each line coming in via os.Stdin
, printing out the domain with fmt.Print
(but not fmt.Println
, as we do not want the linefeed yet), calling our exists
function to see whether the domain exists or not, and printing out the result with fmt.Println
(because we do want a linefeed at the end).
Finally, we use time.Sleep
to tell the process to do nothing for 1
second in order to make sure we take it easy on the WHOIS server.
Most WHOIS servers will be limited in various ways in order to prevent you from taking up too much resources. So slowing things down is a sensible way to make sure we don't make the remote servers angry.
Consider what this also means for unit tests. If a unit test was actually making real requests to a remote WHOIS server, every time your tests run, you will be clocking up stats against your IP address. A much better approach would be to stub the WHOIS server to simulate real responses.
The marks
map at the top of the preceding code is a nice way to map the Boolean response from exists
to human-readable text, allowing us to just print the response in a single line using fmt.Println(marks[!exist])
. We are saying not exist because our program is checking whether the domain is available or not (logically the opposite of whether it exists in the WHOIS server or not).
We can use the check and cross characters in our code happily because all Go code files are UTF-8 compliant—the best way to actually get these characters is to search the Web for them, and use copy and paste to bring them into code; else there are platform-dependent ways to get such special characters.
After fixing the import
statements for the main.go
file, we can try out Available to see whether domain names are available or not:
go build –o available ./available
Once Available is running, type in some domain names:
packtpub.com packtpub.com × google.com google.com × madeupdomain1897238746234.net madeupdomain1897238746234.net ✔
As you can see, for domains that are obviously not available, we get our little cross mark, but when we make up a domain name using random numbers, we see that it is indeed available.