Decentralized Networks

In order to work with a typical database, we need a database connection and sufficient rights to update the database. In the simplest cases there’s a single database to contend with, so we need a single IP address to make the connection. The entire system relies on that database being available. If the database were to crash or lose connectivity, or if our rights were revoked, the application stops working. This is what is referred to as a centralized system.

Blockchains were designed to run over decentralized networks. People and companies run nodes in the network, and, in many respects, all nodes are peers. This is possible because every node contains the full history of every transaction that ever occurred on the blockchain. With every node as a self-sufficient, independent entity, there is no center of the network. All nodes validate transactions and propagate them to their peers. On top of this, some nodes also participate in the block creation process, and are financially incentivized to do so via receiving a “block reward” in the blockchain’s native cryptocurrency. Unlike our database example, any node in a decentralized network can join or drop at will. No special permission or rights are needed to read or write to the blockchain as long as the protocol is followed.

Without a single, centralized database to connect to, how do nodes find a connection to the network? Many smart contract developers punt on this issue and use a service that hides this complexity behind a web API. That is a tradeoff between coupling your system to a third-party service and the complexity of running your own Ethereum node. If you choose to run your own node, it will need to pull itself up by its bootstraps and connect to the public Ethereum network. Every major Ethereum node software has a hardcoded list of “bootnodes.” These are well-known, relatively reliable IP addresses that can be used to create a sufficiently large pool of network peer connections. If for some reason those nodes aren’t available or are no longer trustworthy, the node software can be given a custom list of bootnodes to use.

In 2019, the global, public Ethereum network had more nodes than any other blockchain network. With over 12,000 nodes, it’s unlikely that any two nodes will have the exact same set of peers. As a transaction enters the network through a single peer, it spreads quickly around the world to every node. Similarly, when a new block is added to the blockchain, news of this addition spreads quickly. Unfortunately, quickly is not the same as instantaneously, meaning that when two blocks are created at the same time by two different nodes, the respective peers of those diverging nodes will be operating on two different versions of the blockchain. This is called a temporary fork. In general, temporary forks are resolved by granting precedence to whichever fork adds the next block, summarized as “longest chain wins.” These rules are determined by a blockchain’s consensus protocol.

Consensus Protocols

Without a central source of truth, the nodes in a blockchain network need a way to come to consensus on the state of the system. Consensus protocols are how this is accomplished, and are an active area of research. A consensus protocol is simply a system of agreement across a blockchain. There are three protocols that we need to cover, and we’ll start with Proof-of-Work, which Ethereum used when they launched their blockchains.

Transaction Processing

In 2019, a new block of transactions was added to the Ethereum blockchain approximately every 13 seconds, on average. That said, a sample of block times across a 65-hour span (between blocks 8,662,243 and 8,679,755) contained a block that took over 2 minutes to be written to the chain (block 8,674,540) and others that took just 1 second. This wide variation in block times is due to the random nature of Proof-of-Work. As Ethereum transitions to Proof-of-Stake, we expect to see block times and block time variability decrease.

While block times are regulated by the protocol, the actual time that a transaction takes, from the moment it is first broadcast on the network to its execution in a block, can vary from a few seconds to a few hours. This variability is due to the limitations of the current Ethereum protocol combined with its popularity. For example, if Ethereum can process around 25 transactions per second, but there are over 30 transactions entering the network every second, then there will be transactions that will remain unexecuted until network demand slows down to under 25 transactions per second. These pending transactions are held in-memory by each Ethereum node in what is called a “mempool.” The most reliable way to quickly get a transaction out of the mempool and into a block is to pay more fees to the block creator. In Ethereum, these fees are called “gas,” and as a smart contract developer, you’ll have to frequently consider “gas prices.” See more on the topic of gas and gas prices in Figure 1-2.

Figure 1-2. In step 1, just 20 transactions/second are broadcasted on the network. Assuming the network can currently handle 25 transactions/second, all transactions are mined into the next block. In step 2, 40 transactions/second are broadcasted and the network falls behind, creating a mempool of pending transactions. In step 3, few transactions are sent—just 10 per second—and the mempool drains as the network catches up. The fewer transactions in step 3 would have likely been caused by rising gas prices.

If the same account has two transactions in the mempool, the block creators know which transaction to choose first based on its “nonce,” an account-specific counter that is incremented with each transaction. This means that while the mempool is generally a heap of transactions that block creators can choose from, it’s actually a heap of queues in the cases where accounts have sent multiple transactions before seeing them succeed. Figure 1-3 shows how each account’s transactions are sequenced by its nonce.

Figure 1-3. Alice’s current nonce is 1, Bob’s is 0, and Charlie’s is 2

Regardless of whether a transaction is delayed by the nature of the protocol or due to network congestion, as a developer, you need to consider the unpredictable and asynchronous nature of your users’ experiences with your software system. In a world where financial, commerce, and social media software have been tuned to accept transactions in less than a second, systems backed by smart contracts have significant user experience challenges. The next section digs a level deeper into these challenges.

Transaction Finality

When you successfully execute a transaction in a typical database, developers assume that its effects will never be rolled back. In the case of blockchains, though, it’s possible that a user could see a successful transaction included in a block, only to see that block promptly orphaned and replaced with a different block. This block may not have included the previously successful transaction! This lack of finality with blockchain transactions is due to the decentralized nature of the network. It’s possible for two different nodes to create blocks nearly simultaneously, and for their respective peers to be on different “forks” of the blockchain, as mentioned earlier. Whichever chain creates the next block first will beat the other fork, and previously successful transactions can disappear in the process. From the user’s standpoint, it appears that their once successful transaction gets reverted. Figures 1-4 through 1-6 illustrate the issue.

Figure 1-4. Illustrating a temporary fork, step 1: all four nodes have the same block 53 and consensus with each other.

Figure 1-5. Illustrating a temporary fork, step 2: nodes B and C have forked from each other due to mining (or receiving) two different block 54s simultaneously. Node D has received Node C’s block 54, while Node A has received Node B’s block 54. There are now two divergent blockchains.

Figure 1-6. Illustrating a temporary fork, step 3: note that the competing blocks of a temporary fork can contain different transactions. Once this fork is resolved, either TX_w or TX_z will no longer exist on the blockchain.

This problem can be solved by waiting for additional blocks to be added to the chain before considering the transaction settled. Each application will need to have its own threshold for considering a transaction final. Popular cryptocurrency exchanges will wait for up to 50 blocks for a transaction to be treated as “confirmed” on the Ethereum network. When the stakes are lower, it’s less necessary to wait that long. Each application should consider how long it wants to wait to consider transactions final. In some cases, this “confirmation” will be built into the user experience, and in others, it may just be a general warning that not all transactions are final.

Now that we’ve covered temporary forks, let’s talk about their more permanent cousins.

Hard Forks

Each node in a blockchain network must run software that is compatible with the protocol in order to participate in the propagation of blocks and transactions as well as the creation of new blocks. The community of blockchain node developers, protocol developers, block creators, exchanges, and application developers all weigh in on how the protocol should best evolve to suit the needs of the community. Ultimately, this evolution happens through developers changing the blockchain node software, and the people who run the node software (what we call “node runners”) choosing whether to use the new version or not. Sometimes, the changes made to the protocol are so significant that the new version is not compatible with previous versions of the software. This situation is called a “hard fork” because a new blockchain will spawn from the old, creating a fork in the road for node runners, who will have to decide for themselves which fork they will support.

Hard forks can be contentious or non-contentious. Non-contentious hard forks happen when the entire community upgrades, effectively abandoning the old fork for the new one. This is typical when there is a serious defect discovered in the protocol. Contentious hard forks happen when a critical mass of node runners decide to continue using the existing fork and to evolve the old protocol separately from the new version. Ethereum (ETH) had a contentious hard fork in 2016 that resulted in the previous blockchain surviving and calling itself Ethereum Classic (ETC). There could be additional contentious hard forks as the community implements the various components of the Serenity release.

We’ve been focused mainly on general aspects of blockchains; now, let’s focus in on Ethereum specifically.

Ether and Gas

The Ethereum protocol has its own currency, called ether. The fundamental use of this currency is to pay block creators to include transactions in blocks. Much like the US dollar, ether is divisible, though to much smaller fractions than a cent. The smallest unit of ether is called a wei, which is a quintillionth of an ether (for perspective, a quintillion is a billion billions, or 10¹⁸). Due to a wei’s small size in proportion to ether, you’ll often see ether denominated by Gwei, particularly when it comes to gas prices. A Gwei is a billion wei, and a billion Gwei is one ether.

While there is a single Ethereum protocol, there is more than one network running that protocol. It’s possible to create private networks running Ethereum, similar to how a private “internet” is called an “intranet.” These private networks still deal in ether just like the public Ethereum network, but their ether isn’t worth anything on the open market. The ether on the public Ethereum network is known as ETH and it has real-world value. The public Ethereum network is referred to as “mainnet” by developers. There are also public test networks or “testnets” that the community uses as staging environments. These test networks typically have “faucets,” or mechanisms to give developers free ether in order to test their smart contracts. Every network running the Ethereum protocol has ether, but mainnet’s ETH is the one that has actual value. To deploy and execute smart contracts on mainnet, we need an account that holds ETH. To acquire ETH you will need to purchase it via an exchange, receive it from a friend, earn it from a business, or mine it yourself.

Smart contract code, such as Solidity, is compiled to bytecode, which provides a series of opcodes to the EVM. An opcode is an instruction such as PUSH1 or MLOAD that is interpreted by the EVM. Each of these opcodes has an associated “gas cost.”² Let’s take a look at a simple smart contract, and inspect its bytecode and opcodes. Thankfully, as smart contract developers, we don’t need to understand the contents of the bytecode or opcodes since we can understand the Solidity easily enough. That said, as you progress as a smart contract developer, digging into these lower-level concepts will give you a deeper understanding of what’s possible.

pragma solidity ^0.4.25;

contract Incrementer {
    uint256 public count;

    function addOne() public {
        count++;
    }
}

Compiling this contract using solc results in the following bytecode:

6060604052341561000f57600080fd5b60cb8061001d6000396000f300606060405260043610604
9576000357c0100000000000000000000000000000000000000000000000000000000900463ffff
ffff168063a7916fac14604e578063febb0f7e146060575b600080fd5b3415605857600080fd5b6
05e6086565b005b3415606a57600080fd5b60706099565b60405180828152602001915050604051
80910390f35b6000808154809291906001019190505550565b600054815600a165627a7a7230582
08e9afbffafd387e67b7c38d8239aaa70fde96a805cebfb6f30517dd68e8664be0029

And the opcodes as reported by solc look like this:

PUSH1 0x60 PUSH1 0x40 MSTORE CALLVALUE ISZERO PUSH2 0xF JUMPI PUSH1 0x0 DUP1
REVERT JUMPDEST PUSH1 0xCB DUP1 PUSH2 0x1D PUSH1 0x0 CODECOPY PUSH1 0x0 RETURN
STOP PUSH1 0x60 PUSH1 0x40 MSTORE PUSH1 0x4 CALLDATASIZE LT PUSH1 0x49 JUMPI
PUSH1 0x0 CALLDATALOAD PUSH29 0x100000000000000000000000000000000000000000000000
000000000 SWAP1 DIV PUSH4 0xFFFFFFFF AND DUP1 PUSH4 0xA7916FAC EQ PUSH1 0x4E
JUMPI DUP1 PUSH4 0xFEBB0F7E EQ PUSH1 0x60 JUMPI JUMPDEST PUSH1 0x0 DUP1 REVERT
JUMPDEST CALLVALUE ISZERO PUSH1 0x58 JUMPI PUSH1 0x0 DUP1 REVERT JUMPDEST PUSH1
0x5E PUSH1 0x86 JUMP JUMPDEST STOP JUMPDEST CALLVALUE ISZERO PUSH1 0x6A JUMPI
PUSH1 0x0 DUP1 REVERT JUMPDEST PUSH1 0x70 PUSH1 0x99 JUMP JUMPDEST PUSH1 0x40
MLOAD DUP1 DUP3 DUP2 MSTORE PUSH1 0x20 ADD SWAP2 POP POP PUSH1 0x40 MLOAD DUP1
SWAP2 SUB SWAP1 RETURN JUMPDEST PUSH1 0x0 DUP1 DUP2 SLOAD DUP1 SWAP3 SWAP2 SWAP1
PUSH1 0x1 ADD SWAP2 SWAP1 POP SSTORE POP JUMP JUMPDEST PUSH1 0x0 SLOAD DUP2 JUMP
STOP LOG1 PUSH6 0x627A7A723058 KECCAK256 DUP15 SWAP11 CREATE2 SELFDESTRUCT 0xaf
0xd3 DUP8 0xe6 PUSH28 0x7C38D8239AAA70FDE96A805CEBFB6F30517DD68E8664BE0029000000

The concept of “gas” exists to decouple the price of ether from Ethereum’s transaction fees. Without this decoupling, Ethereum transaction fees would be pegged to the price of ether, which would have negative impacts on the ecosystem due to the volatility of ETH’s value. To decouple the exchange rate between gas and ether, every Ethereum transaction sets its own gasprice to determine how many wei a single unit of gas costs. As block creators decide which transactions to include in a block, they are incentivized to include the transactions that will give them the most generous gasprice for their computations. At any given moment there is an implicitly understood market rate for gas. Paying below market rate will mean waiting longer than most other transactions for your transaction to be executed, while paying above market rate will allow you to leave the mempool sooner than most.

Ethereum transaction creators need to consider the costs versus benefits of setting their gas price above or below the current market rate. Some transactions are not time-sensitive and can wait many hours to be executed. In this case, it makes sense to pay relatively little ether for a transaction’s execution. Block creators finally include low-gas-price transactions when the mempool is shallow and there isn’t a critical mass of transactions with a generous gasprice. Inversely, some transactions are incredibly time-sensitive, and are so valuable that it makes sense to pay significantly above the market rate for fast block inclusion. There are even people who run software to observe the mempool in real time and try to “front-run” certain financial trading transactions by seeing a pending trade and jumping ahead of it by setting a higher gasprice.

Each Ethereum transaction has to include gas and gasprice attributes, which when multiplied will set the maximum transaction fee for the transaction, denominated in wei. This gas attribute sets a limit to how many computations the transaction can perform. If that limit is reached, the smart contract execution reverts, but the transaction is still written to the blockchain, and the fees are consumed by the block creator. If the smart contract call completes with leftover gas, the gas is returned to the transaction creator. The sum of the gas used across all transactions in a block cannot exceed the block’s specified gaslimit. This also means that no single transaction’s gas usage can exceed the block’s gaslimit.

The next section will dig into the details of the players within a transaction and how they are identified.

Accounts

The most basic Ethereum transactions are just externally owned addresses (EOAs) sending ether to each other. Additionally, Ethereum transactions can be sent from an EOA to a smart contract. Both EOAs and smart contracts are identified by an Ethereum address like the following:

0x52bc44d5378309ee2abf1539bf71de1b7d7be3b5

An address is represented by a hexadecimal number. Given the magnitude of the numbers involved in generating an address, it is practically impossible to generate two identical addresses. There is no way to distinguish an address for a smart contract from an EOA without inspecting the blockchain. While accounts and contracts have undistinguishable addresses, they have some important differences.

Every transaction in the Ethereum blockchain is initiated by an EOA. Smart contracts can’t spontaneously perform an action. They can call other smart contracts, but every transaction originates from an EOA. When contracts are called, they can emit events, store data, receive ether, send ether to EOAs, or send data or ether to other contracts. Besides initiating transactions, EOAs, on the other hand, can only receive ether. They cannot react to any transactions that involve them the way that a smart contract does.

Contracts

Contracts in Solidity are organized in an object-oriented style similar to the Java programming language. In object-oriented parlance, a contract is really a class, or a collection of state variables and functions. In order to reuse common functionality, object-oriented languages allow a class to inherit from another class (or contracts, in this case). Since Solidity uses function as a keyword, we will refer to as “functions” what most object-oriented languages refer to as “methods.” Solidity functions can be separated into two distinct types: write-only and read-only.

Read-only Solidity functions are denoted with the pure and view keywords. These functions can take input data, read contract data, operate on that data, and return data. Read-only functions cannot change the state of the contract or emit events. Because no updates are needed on-chain, read-only functions are instantaneous—they are very similar to web API calls, particularly GET requests. It’s important to note that being able to skip on-chain updates means that read-only functions can be called without paying any gas costs, and there will be no transaction created.

Write-only functions are the default in Solidity so they require no additional keywords. Despite being “write-only,” they can actually return data, but due to the asynchronous nature of Ethereum, the return data is practically useless, hence “write-only.” These functions are the workhorses of Ethereum, and their data must be sent via transaction and included in a block in order for the function to be executed. An unsuccessful write-only method will revert, either because it has run out of gas or reached an invalid EVM state, or because of an explicit statement in the contract, such as failing a require statement. A successful write-only method doesn’t actually have to change anything, but it typically changes something and often emits one or more events in the process.

The purpose of events in Ethereum is generally twofold: to provide a custom historical log of what has occurred in the contract, and to allow observers to subscribe to real-time updates. Due to the nature of the blockchain, we already have a historical ledger of everything that has ever happened, but events are a convenient way to provide more domain-specific logging and updates so that users don’t have to create their own interpretations to transactions and state transitions.

To better understand the mechanics of Ethereum’s smart contracts, let’s now focus on the vehicles of their modification.

Blocks and Transactions

Only Ethereum block creators determine the attributes of a block, such as which transactions are included. Similarly, only Ethereum users determine the attributes of a transaction, such as which contract to send data to. As smart contract developers, we frequently need to be aware of the state of the block as well as the state of the transaction that is currently executing.

Solidity exposes the following transaction (tx) attributes:

gasprice

The gas price (in wei) set by the EOA that created the transaction.

origin

The address of the EOA that created the transaction. Surprisingly, this address is rarely useful and often insecure.

A transaction can have an arbitrary number of contracts involved in its execution, provided that its execution fits within the constraints of the block’s gaslimit. Solidity exposes a number of other transaction-related attributes, but groups them into a message (msg) abstraction. Messages refer to the communication between contracts and anything that can call them, such as other contracts. For example, a contract function call will always have a msg.sender. That msg.sender could be equal to the tx.origin or the creator of the transaction, or it could be the address of an intermediary contract.

Solidity’s message (msg) attributes are as follows:

data

The raw bytes of data that were sent to the currently executed external or public function. This is also referred to as calldata.

sender

The address of the caller of the currently executed external or public function.

sig

The first four bytes of the calldata determine which function is being called. This is also referred to as the function identifier.

value

The amount of wei that is being sent to this function.

Solidity exposes the following block attributes:

number

Each block increments this number. The genesis block was block 0.

timestamp

The time in seconds since epoch that the block was created. You may also see code that uses its alias, now.

blockhash

In addition to its sequential block number, each block is uniquely identified by its hash. A hash is a hexadecimal number such as 0x88e96d4537bea4d9c05d12549​907b32561d3bf31f45aae734cdc119f13406cb6. The hash of the current block is not available, but by providing a block number, you can get the hash of any block within the past 256 blocks.

difficulty

The mining difficulty level of the current block.

gaslimit

The maximum amount of gas that can be consumed by this block. This is set by the block creator.

coinbase

The address of the block creator.

Next, we will address the quirky aspect of how time works in a smart contract.

What Time Is It?

Most programming languages allow developers to check the time, which typically uses the time reported by the computer the program is running on. For instance, this JavaScript program running on Node.js reports on the current time as it is executing:

for (let i = 0; i < 10; i++) {
    console.log(new Date());
}

The output of this program would be something like this:

2019-10-05T05:08:45.058Z
2019-10-05T05:08:45.059Z
2019-10-05T05:08:45.060Z
2019-10-05T05:08:45.060Z
2019-10-05T05:08:45.060Z
2019-10-05T05:08:45.060Z
2019-10-05T05:08:45.060Z
2019-10-05T05:08:45.060Z
2019-10-05T05:08:45.060Z
2019-10-05T05:08:45.060Z

Not surprisingly, you can see the time changing as the program executes. Here is the equivalent code in Solidity:

pragma solidity ^0.5.0;

contract TimeReporter {
    event TimeLog(uint256 time);

    function reportTime() public {
        for (uint8 i = 0; i < 10; i++) {
            emit TimeLog(block.timestamp);
        }
    }
}

The events emitted by this function would all have the same exact time attribute. That time attribute, set by block.timestamp, is the time that the block was added to the blockchain. For every transaction in that block, the block.timestamp attribute will be identical. While the clock on a computer ticks at least once a millisecond, the “clock” on a blockchain only ticks as often as blocks are added to the chain. Due to a blockchain’s low-resolution clock, we can never expect an exact second to occur. When you write code that checks the time, comparisons should always involve greater or less than, rather than exactly equal.

When designing smart contracts, it is also important to keep in mind that that block creators can manipulate the time a block is created as well as the ordering of the transactions to their advantage. For instance, if a smart contract has a built-in deadline, and it would significantly benefit most block creators to have that deadline missed, then they can choose to delay any transactions that would meet the deadline. Block creators manipulating the ordering of transactions to their benefit can come in the form of “front-running” token trades. For example, someone sends a transaction to buy 5 ABC tokens for 1 ETH, due to some new information about ABC tokens. A block creator with significant hash power could delay that 5 ABC / 1 ETH transaction and add their own transaction to a block for a lesser amount and grab those ABC tokens for themselves. This is only possible if the block creator actually succeeds at creating a block, which is highly competitive. So these concerns aren’t a significant vulnerability of smart contracts, but they are important to keep in mind as you consider your designs and incentives.

Finally, let’s consider some of the “crypto” in cryptocurrency.

Signing Transactions

We need to appreciate the reasons why when we “sign” a transaction, we can be sure that signature was created by a specific private key. This comes down to good old public-key cryptography. A private key is the secret counterpart to a public key. The address of an EOA is a truncation of the hash of its public key.³

Mercifully, in the course of developing smart contracts and decentralized applications (DApps), we don’t often work directly with private keys. We strongly recommend to leave private key management and cryptographic signatures to wallet software such as MetaMask. Knowing an EOA’s private key is synonymous with owning that account because the private key is what is used to sign transactions. Without this cryptographic signature, there is no way to authenticate whether a transaction was actually sent by its specified EOA.

When we send an Ethereum transaction using any of the Web3 libraries, the cryptographic signature happens in the background. The following transaction attributes are concatenated, encoded, and then signed with the configured private key:

nonce

Sequence number of this transaction for this EOA.

gasPrice

Amount of wei that this transaction is paying per unit of gas.

gas

Amount of gas that this transaction is willing to spend.

to

Recipient address of this transaction. It could be an EOA or a contract.

value

Amount of wei (if any) that this transaction is sending to the recipient.

data

In the case of a contract call, this contains the function name and all parameters. In the case of a contract deployment, this contains the contract bytecode. If no contracts are involved, this is generally blank.

chainId

Each public Ethereum network has a chainId. Mainnet is 1, the Kovan testnet is 42, etc.

Once those attributes are signed, the signature itself is included in the transaction so that Ethereum nodes can validate that the sender is legitimate. In order to validate this, nodes use the sender’s address to validate the signature. If someone were to try to send a transaction with a bad signature, the nodes would reject it.