Over the course of the book, you’ve learned a bit of theory about how HBase is designed and how it distributes the load across different servers. Let’s poke around the system a little and get familiar with how these things work in practice. First, we’ll look at what ZooKeeper says about your HBase instance. When running HBase in standalone mode, ZooKeeper runs in the same JVM as the HBase Master. In chapter 9, you learned about fully distributed deployments, where ZooKeeper runs as a separate service. For now, let’s examine the commands and see what ZooKeeper has to tell you about your deployment.
Your main interface to interact directly with ZooKeeper is through the HBase shell. Launch the shell, and issue the zk_dump command.
Let’s look at a zk_dump from a fully distributed HBase instance. We’ve included the output from a running cluster and masked the server names. The portions in bold are analogous to the things we mentioned:
hbase(main):030:0> > zk_dump HBase is rooted at /hbase Master address: 01.mydomain.com:60000 Region server holding ROOT: 06.mydomain.com:60020 Region servers: 06.mydomain.com:60020 04.mydomain.com:60020 02.mydomain.com:60020 05.mydomain.com:60020 03.mydomain.com:60020 Quorum Server Statistics: 03.mydomain.com:2181 Zookeeper version: 3.3.4-cdh3u3--1, built on 01/26/2012 20:09 GMT Clients: ... 02.mydomain.com:2181 Zookeeper version: 3.3.4-cdh3u3--1, built on 01/26/2012 20:09 GMT Clients: ... 01.mydomain.com:2181 Zookeeper version: 3.3.4-cdh3u3--1, built on 01/26/2012 20:09 GMT Clients: ...
If you have access to one, run the command on the shell and see what it tells you about the system. This is useful information when you’re trying to understand the state of the system—what hosts are participating in the cluster, which host is playing what role, and, most important, which host is serving the -ROOT- table. The HBase client needs all this information to perform reads and writes on your application’s behalf. Note that your application code isn’t involved in this process; the client library handles all of this for you.
The client automatically handles communicating with ZooKeeper and finding the relevant RegionServer with which to interact. Let’s still examine the -ROOT- and .META. tables to get a better understanding of what information they contain and how the client uses that information.
Let’s look at our standalone instance we’ve used so far for TwitBase. Here’s -ROOT- for the standalone instance:
hbase(main):030:0> scan '-ROOT-' ROW COLUMN+CELL .META.,, column=info:regioninfo, timestamp=1335465653682, 1 value={NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} .META.,, column=info:server, timestamp=1335465662307, 1 value=localhost:58269 .META.,, column=info:serverstartcode, 1 timestamp=1335465662307, value=1335465653436 .META.,, column=info:v, timestamp=1335465653682, 1 value=x00x00 1 row(s) in 5.4620 seconds
Let’s examine the contents of the -ROOT- table. The -ROOT- table stores information about the .META. table. It’s the first place a client application looks when it needs to locate data stored in HBase. In this example, there is only one row in -ROOT-, and it corresponds to the entire .META. table. Just like a user-defined table, .META. will split once it has more data to hold than can go into a single region. The difference between a user-defined table and .META. is that .META. stores information about the regions in HBase, which means it’s managed entirely by the system. In the current example, a single region in .META. can hold everything. The corresponding entry in -ROOT- has the rowkey defined by the table name and the start key for that region. Because there is only one region, the start key and end key for that region are empty. This indicates the entire key range is hosted by that region.
The previous entry in the -ROOT- table also tells you which server the .META. region is hosted on. In this case, because it’s a standalone instance and everything is on local-host, that column contains the value localhost:port. There is also a column for regioninfo that contains the name of the region, start key, end key, and encoded name. (The encoded name is used internally in the system and isn’t of any consequence to you.) The HBase client library uses all this information to locate the correct region to talk to while performing the operations in your application code. When no other table exists in the system, .META. looks like the following:
hbase(main):030:0> scan '.META.' ROW COLUMN+CELL 0 row(s) in 5.4180 seconds
Notice that there are no entries in the .META. table. This is because there are no user-defined tables in this HBase instance. On instantiating the users table, .META. looks like the following:
hbase(main):030:0> scan '.META.' ROW COLUMN+CELL users,,1335466383956.4a1 column=info:regioninfo, 5eba38d58db711e1c7693581 timestamp=1335466384006, value={NAME => af7f1. 'users,,1335466383956.4a15eba38d58 db711e1c7693581af7f1.', STARTKEY => '', ENDKEY => '', ENCODED => 4a15eba38d58db711e1c7693581af7f1,} users,,1335466383956.4a1 column=info:server, timestamp=1335466384045, 5eba38d58db711e1c7693581 value=localhost:58269 af7f1. users,,1335466383956.4a1 column=info:serverstartcode, 5eba38d58db711e1c7693581 timestamp=1335466384045, value=1335465653436 af7f1. 1 row(s) in 0.4540 seconds
In your current setup, that’s probably what it looks like. If you want to play around a bit, you can disable and delete the users table, examine .META., and re-create and repopulate the users table. Disabling and deletion can be done in the HBase shell just like creating a table.
Similar to what you saw in -ROOT-, .META. contains information about the users table and other tables in your system. Information for all the tables you instantiate goes here. The structure of .META. is similar to that of -ROOT-.
Once you’ve examined the .META. table after instantiating the TwitBase users table, populate some users into the system using the LoadUsers command provided in the sample application code. When the users table outgrows a single region, that region will split. You can also split the table manually for experimentation right now:
hbase(main):030:0> split 'users' 0 row(s) in 6.1650 seconds hbase(main):030:0> scan '.META.' ROW COLUMN+CELL users,,1335466383956.4a15eba column=info:regioninfo, 38d58db711e1c7693581af7f1. timestamp=1335466889942, value={NAME => 'users,,1335466383956.4a15eba38d58db711e 1c7693581af7f1.', STARTKEY => '', ENDKEY => '', ENCODED => 4a15eba38d58db711e1c7693581af7f1, OFFLINE => true, SPLIT => true,} users,,1335466383956.4a15eba column=info:server, 38d58db711e1c7693581af7f1. timestamp=1335466384045, value=localhost:58269 users,,1335466383956.4a15eba column=info:serverstartcode, 38d58db711e1c7693581af7f1. timestamp=1335466384045, value=1335465653436 users,,1335466383956.4a15eba column=info:splitA, 38d58db711e1c7693581af7f1. timestamp=1335466889942, value= {NAME => 'users,,1335466889926.9fd558ed44a63f016 c0a99c4cf141eb5.', STARTKEY => '', ENDKEY => '}7 x8ExC3xD1xE3x0Fx0DxE9xFE'fIKxB7 xD6', ENCODED => 9fd558ed44a63f016c0a99c4cf141eb5,} users,,1335466383956.4a15eba column=info:splitB, 38d58db711e1c7693581af7f1. timestamp=1335466889942, value={NAME => 'users,}7x8ExC3xD1xE3x0F x0DxE9xFE'fIKxB7xD6,1335466889926.a3 c3a9162eeeb8abc0358e9e31b892e6.', STARTKEY => '}7x8E xC3xD1xE3x0Fx0DxE9xFE'fIKxB7xD6' , ENDKEY => '', ENCODED => a3c3a9162eeeb8abc0358 e9e31b892e6,} users,,1335466889926.9fd558e column=info:regioninfo, d44a63f016c0a99c4cf141eb5. timestamp=1335466889968, value={NAME => 'users,,1335466889926.9fd558ed44a63f016c 0a99c4cf141eb5.', STARTKEY => '', ENDKEY => '}7x8ExC3 xD1xE3x0Fx0DxE9xFE'fIKxB7xD6', ENCODED => 9fd558ed44a63f016c0a99c4cf141eb5,} users,,1335466889926.9fd558e column=info:server, d44a63f016c0a99c4cf141eb5. timestamp=1335466889968, value=localhost:58269 users,,1335466889926.9fd558e column=info:serverstartcode, d44a63f016c0a99c4cf141eb5. timestamp=1335466889968, value=1335465653436 users,}7x8ExC3xD1xE3x0F column=info:regioninfo, x0DxE9xFE'fIKxB7xD6,133 timestamp=1335466889966, value={NAME => 5466889926.a3c3a9162eeeb8abc 'users,}7x8ExC3xD1xE3x0Fx0DxE9xF 0358e9e31b892e6. E'fIKxB7xD6,1335466889926.a3c3a9162eee b8abc0358e9e31b892e6.', STARTKEY => '}7x8ExC3xD1xE3x0F x0DxE9xFE'fIKxB7xD6', ENDKEY => '', ENCODED => a3c3a9162eeeb8abc0358e9e31b892e6,} users,}7x8ExC3xD1xE3x0F column=info:server, x0DxE9xFE'fIKxB7xD6,133 timestamp=1335466889966, 5466889926.a3c3a9162eeeb8abc value=localhost:58269 0358e9e31b892e6. users,}7x8ExC3xD1xE3x0F column=info:serverstartcode, x0DxE9xFE'fIKxB7xD6,133 timestamp=1335466889966, 5466889926.a3c3a9162eeeb8abc value=1335465653436 0358e9e31b892e6. |
3 row(s) in 0.5660 seconds |
After you split users, new entries are made in the .META. table for the daughter regions. These daughter regions replace the parent region that was split. The entry for the parent region contains the information about the splits, and requests are no longer served by the parent region. For a brief time, the information for the parent region remains in .META. Once the hosting RegionServer has completed the split and cleaned up the parent region, the entry is deleted from .META.. After the split is complete, .META. looks like this:
hbase(main):030:0> scan '.META.' ROW COLUMN+CELL users,,1335466889926.9fd558e column=info:regioninfo, d44a63f016c0a99c4cf141eb5. timestamp=1335466889968, value={NAME => 'users,,1335466889926.9fd558ed44a63f016c 0a99c4cf141eb5.', STARTKEY => '', ENDKEY => '}7x8ExC3 xD1xE3x0Fx0DxE9xFE'fIKxB7xD6', ENCODED => 9fd558ed44a63f016c0a99c4cf141eb5,} users,,1335466889926.9fd558e column=info:server, d44a63f016c0a99c4cf141eb5. timestamp=1335466889968, value=localhost:58269 users,,1335466889926.9fd558e column=info:serverstartcode, d44a63f016c0a99c4cf141eb5. timestamp=1335466889968, value=1335465653436 users,}7x8ExC3xD1xE3x0F column=info:regioninfo, x0DxE9xFE'fIKxB7xD6,133 timestamp=1335466889966, value={NAME => 5466889926.a3c3a9162eeeb8abc 'users,}7x8ExC3xD1xE3x0Fx0DxE9xF 0358e9e31b892e6. E'fIKxB7xD6,1335466889926.a3c3a9162eee b8abc0358e9e31b892e6.', STARTKEY => '}7x8ExC3xD1xE3 x0Fx0DxE9xFE'fIKxB7xD6', ENDKEY => '', ENCODED => a3c3a9162eeeb8abc0358e9e31b892e6,} users,}7x8ExC3xD1xE3x0F column=info:server, x0DxE9xFE'fIKxB7xD6,133 timestamp=1335466889966, 5466889926.a3c3a9162eeeb8abc value=localhost:58269 0358e9e31b892e6 users,}7x8ExC3xD1xE3x0F column=info:serverstartcode, x0DxE9xFE'fIKxB7xD6,133 timestamp=1335466889966, 5466889926.a3c3a9162eeeb8abc value=1335465653436 0358e9e31b892e6. |
2 row(s) in 0.4890 seconds |
You may wonder why the start and end keys contain such long, funny values that don’t look anything like the entries you put in. This is because the values you see are byte-encoded versions of the strings you entered.
Let’s tie this back to how the client application interacts with an HBase instance. The client application does a get(), put(), or scan(). To perform any of these operations, the HBase client library that you’re using has to find the correct region(s) that will serve these requests. It starts with ZooKeeper, from which it finds the location of -ROOT-. It contacts the server serving -ROOT- and reads the relevant records from the table that point to the .META. region that contains information about the particular region of the table that it has to finally interact with. Once the client library gets the server location and the name of the region, it contacts the RegionServer with the region information and asks it to serve the requests.
These are the various steps at play that allow HBase to distribute data across a cluster of machines and find the relevant portion of the system that serves the requests for any given client.