Chapter 1 – What is Columnar?

“When you go into court you, are putting your fate into the hands of twelve people who weren’t smart enough to get out of jury duty.”

– Norm Crosby

What is Parallel Processing?

"After enlightenment, the laundry"

- Zen Proverb

"After parallel processing the laundry, enlightenment!"

-Matrix Zen Proverb

Two guys were having fun on a Saturday night when one said, "I’ve got to go and do my laundry." The other said, "What!?" The first man explained that if he went to the laundry mat the next morning, he would be lucky to get one machine and be there all day. But if he went on Saturday night, he could get all the machines. Then, he could do all his wash and dry in two hours. Now that's parallel processing mixed in with a little dry humor!

Nothing Happens on Disk

“When you are courting a nice girl, an hour seems like a second. When you sit on a red-hot cinder, a second seems like an hour. That’s relativity.”

– Albert Einstein

Data on disk does absolutely nothing. When data is requested, the computer moves the data one block at a time from disk into memory. Once the data is in memory, it is processed by the CPU at lightning speed. All computers work this way. The "Achilles Heel" of every computer is the slow process of moving data from disk to memory. The real theory of relativity is to find out how to get blocks of data from the disk into memory faster!

Data in Memory is fast as Lightning

“You can observe a lot by watching."

– Yogi Berra

Once the data block is moved off of the disk and into memory, the processing of that block happens as fast as lightning. It is the movement of the block from disk into memory that slows down every computer. Data being processed in memory is so fast that even Yogi Berra couldn't catch it!

Parallel Processing Of Data

"If the facts don't fit the theory, change the facts."

-Albert Einstein

Big Data is all about parallel processing. Parallel processing is all about taking the rows of a table and spreading them among many parallel processing units. Above, we can see a table called Orders. There are 16 rows in the table. Each parallel processor holds four rows. Now they can process the data in parallel and be four times as fast. What Albert Einstein meant to say was, “If the theory doesn't fit the dimension table, change it to a fact."

The Problem with Row-Based Data

Nothing happens on disk. For data to be processed, the block of disk data must be copied and moved into memory. The problem with row-based data is that the entire block must be moved into memory even when the query only needs to analyze a single column. When queries only need a few columns, moving the entire block is a lot of wasted energy.

Columnar Data Can Store Each Column in Their Own Block

Columnar systems can store each column of a table in their own individual block. This is extremely efficient when a query only needs a relatively few columns from the table to satisfy the query. Our query above only needs the Order_Total column to get the average Order_Total. Only one small block on each parallel process is moved into memory. Wow, that was fast!

Why Columnar?

Each data block holds a single column. The row can be rebuilt because everything is aligned perfectly. If someone runs a query that would return the average salary, then only one small data block is moved into memory. The salary block moves into memory where it is processed as fast as lightning. We just cut down on moving large blocks by 80%! Why columnar? Because like our Yiddish Proverb says, "All data is not kneaded on every query, so that is why it costs so much dough."

Row Based Blocks vs. Columnar Based Blocks

Both designs have the same amount of data. Both take up just as much space. In this example, both have 9 rows and five columns. If a query needs to analyze all of the rows or return most of the columns, then the row based design is faster and more efficient. However, if the query only needs to analyze a few rows or merely a few columns, then the columnar design is much lighter because not all of the data is moved into memory. Just one or two columns move. Take the road less traveled.

Visualize the Data – Rows vs. Columns

Both examples above have the same data and the same amount of data. If your applications tend to need to analyze the majority of columns or read the entire table, then a row-based system (top example) can move more data into memory. Columnar tables are advantageous when only a few columns need to be read. This is just one of the reasons why analytics goes with columnar like bread goes with butter. A row-based system must move the entire block into memory even if it only needs to read one row or even a single column. If a user above needed to analyze the Salary, the columnar system would move 80% less block mass.

The Architecture of Vertica

“Be the change that you want to see in the world.”

- Mahatma Gandhi

Vertica is a shared nothing architecture, designed as a collection of Linux cluster nodes connected by a TCP/IP network. Storage can be directly attached to each node, or SAN-based. This technology is relatively inexpensive. It might not "be the change you want to see in the world", but it will help your company "keep the change" because costs are low.

Vertica Architecture Terms

Host - A server with a 64-bit processor, memory, hard disk and TCP/IP network interface. Hosts share neither disk space nor main memory with each other. Vertica is a shared nothing MPP architecture.

Instance - An instance is a node running the Vertica process and disk storage (catalog and data) on a host. Only one instance of HP Vertica can be running on each host. Multiple instances make up a cluster.

Node - A host that is configured to run an instance of Vertica. It is a member of a database cluster, which consists of one or more nodes working together in parallel. HP has recoverability for multiple nodes (minimum 3 – recommended 4) so a database can recover from a node failure.

Cluster - A collection of nodes bound to a database.

Database - A cluster of nodes that perform distributed data storage and SQL statement execution as a single unit.

Above are the key terms for the Vertica architecture.

Vertica has Linear Scalability

"A Journey of a thousand miles begins with a single step."

- Lao Tzu

Vertica was born to be parallel. With each query, a single step is performed in parallel by each Segment. A Vertica system consists of a series of nodes that will work in parallel to store and process your data. This design allows you to start small and grow infinitely. If your Vertica system provides you with an excellent Return on Investment (ROI), then continue to invest by purchasing more nodes (adds additional Segments). Most companies start small, but after seeing what Vertica can do, they continue to grow their ROI from the single step of implementing a Vertica system to millions of dollars in profits. Double the Segments and double the speeds. . . . Forever. Vertica actually provides a journey of a thousand smiles!