© Karan Singh Garewal 2020

K. S. GarewalPractical Blockchains and Cryptocurrencieshttps://doi.org/10.1007/978-1-4842-5893-4_16

16. Blockchain Maintenance

Karan Singh Garewal¹ 

(1)

Toronto, ON, Canada

 

In this chapter, we will examine Helium blockchain maintenance. The following topics will be discussed:

1. 1.

   How a new Helium node with an empty blockchain acquires blocks from the Helium network and synchronizes itself with the distributed blockchain. This procedure for acquiring blocks also applies if a node goes down or if there is a temporary network partition.

    
2. 2.

   Rebuilding the Chainstate database.

    
3. 3.

   Rebuilding the blk_index database.

    

Blockchain Maintenance

A new node that joins the Helium peer-to-peer network will need to obtain a blockchain. Furthermore, the blockchain held by a node may need synchronization if it goes down or if there is a temporary network partition.

The Helium network interface provides two remote procedure calls to handle these use cases:

get_blockchain_height() -> "integer"

get_block(block_no: "int") ->  "stringified block or an error string"

get_blockchain_height obtains the height of the blockchain held by a remote node. The call get_block obtains a specific block from a remote node. Both of these remote procedure calls were discussed in the previous chapter.

I will leave it to you to develop tests to obtain blocks. In a production setting and in order to improve performance, you will want to enhance the code to provide for the concurrent downloading of blocks from several remote nodes.

Reliability Engineering in Helium

An application that manages financial transactions must have extreme reliability. The cardinal principle is that the application should implement the simplest architectural patterns possible to achieve its objectives.

In the previous chapter, we saw that the distributed peer-to-peer architecture of Helium creates a network architecture that cannot be easily degraded. All of the nodes in the ecosystem are equal in capability, and anyone can instantiate a node and join the Helium network. Furthermore, Bitcoin and Helium have very modest requirements and thus can run on inexpensive commodity hardware. It is entirely feasible to run a Helium or Bitcoin node on a laptop.

Furthermore, we saw in the previous section that a Helium node can always synchronize its blockchain with the distributed blockchain in the event that the node goes down or if there is an unexpected network partition.

Aside from these reliability features, Helium and Bitcoin implement a single point of failure architecture. This point of failure is the block file, which is a simple, encoded text file. A blockchain is the collection of these files.

A Helium node can rebuild all of the data structures and databases that it needs for its operation from the blockchain (the collection of block files). Helium uses two key-value stores, the Chainstate Database and the blk_index database, and ensures by design that these databases are not critical points of failure. In other words, Helium can always rebuild these databases as long as it has all of the block files.

Helium blocks are encoded text files that are read-only. Since a blockchain is an immutable structure, block file updates and deletes are not required or supported. The create and read operations are handled by the underlying operating system. The following code snippet from the hblockchain module shows how block persistence is implemented in Helium:

def add_block(block: "dictionary") -> "bool":

    """

    add_block: adds a block to the blockchain. Receives a block. The block

    attributes are checked for validity and each transaction in the block is

    tested for validity. If there are no errors, the block is written to a file

    as asequence of raw bytes. Then the block is added to the blockchain.

    The chainstate database and the blk_index databases are updated.

    returns True if the block is added to the blockchain and False otherwise

    """

    try:

        # validate the received block parameters

        if validate_block(block) == False:

            raise(ValueError("block validation error"))

        # validate the transactions in the block

        # update the chainstate database

        for trx in block['tx']:

            # first transaction in the block is a coinbase transaction

            if block["height"] == 0 or block['tx'][0] == trx: zero_inputs = True

            else: zero_inputs = False

            if tx.validate_transaction(trx, zero_inputs) == False:

                raise(ValueError("transaction validation error"))

            if hchaindb.transaction_update(trx) == False:

                raise(ValueError("chainstate update transaction error"))

        # serialize the block to a file

        if (serialize_block(block) == False):

                raise(ValueError("serialize block error"))

        # add the block to the blockchain in memory

        blockchain.append(block)

        #  update the blk_index

        for transaction in block['tx']:

            blockindex.put_index(transaction["transactionid"], block["height"])

    except Exception as err:

        print(str(err))

        logging.debug('add_block: exception: ' + str(err))

        return False

    return True

def serialize_block(block: "dictionary") -> "bool":

    """

    serialize_block: serializes a block to a file using pickle.

    Returns True if the block is serialized and False otherwise.

    """

    index = len(blockchain)

    filename = "block_" + str(index) + ".dat"

    # create the block file and serialize the block

    try:

        f = open(filename, 'wb')

        pickle.dump(block, f)

    except Exception as error:

        logging.debug("Exception: %s: %s", "serialize_block", error)

        f.close()

        return False

    f.close()

    return True

Once a block has been validated, it is persisted to a specific directory.¹ You will want the blocks written into the data directory. The function serialize_block writes a block into a directory. As you will notice, it is exceptionally simple. Each block is written to a file named

 "block" + "height of block " + ".dat"

This simple naming scheme indexes the block files by height. The genesis block has height zero.

The collection of all of the blocks in the directory is the blockchain.

Constructing a Simulated Blockchain

The following program code builds a simulated blockchain with a height of 500 blocks. Such a synthetic blockchain can be very useful for a number of use cases:

1. 1.

   Testing blockchains during the development phase

    
2. 2.

   Examining the behavior of the blockchain when its parameters are changed

    
3. 3.

   Examining the resiliency and reliability of the blockchain

    

Copy this code into a Python file called simulated_blockchain.py and save it in the unit_tests directory. After this, execute it in the Helium virtual environment, with

$(virtual) simulated_blockchain.py

You should see a blockchain of 500 blocks in the unit_tests directory. You can build blockchains of arbitrary height by providing the requisite height parameter to the make_blocks function.

A caveat is in order. As transactions occur on the simulated blockchain, the proportion of fragmented transactions with very small outputs will increase. Due to this fragmentation, builds of the blockchain for very high heights will fail. Consolidation of fragmented transaction outputs is typically handled by wallets. You are invited to build a fragmented transaction consolidator for our simulated blockchain. This will provide you with very valuable insight into how blockchains are built and evolve over time.

######################################################

# simulated blockchain constructor

######################################################

import blk_index as blkindex

import hchaindb

import hmining

import hconfig

import hblockchain

import rcrypt

import tx

import json

import logging

import pdb

import os

import secrets

import sys

import time

"""

   log debugging messages to the file debug.log

"""

logging.basicConfig(filename="debug.log",filemode="w",  \

format='client: %(asctime)s:%(levelname)s:%(message)s', level=logging.DEBUG)

def startup():

    """

    start the databases

    """

    # start the Chainstate Database

    ret = hchaindb.open_hchainstate("heliumdb")

    if ret == False: return "error: failed to start Chainstate database"

    else: print("Chainstate Database running")

    # start the LevelDB Database blk_index

    ret = blkindex.open_blk_index("hblk_index")

    if ret == False: return "error: failed to start blk_index"

    else: print("blkindex Database running")

def stop():

    """

    stop the databases

    """

    hchaindb.close_hchainstate()

    blkindex.close_blk_index()

# unspent transaction fragment values [{fragmentid:value}]

unspent_fragments = []

######################################################

# Make A Synthetic Random Transaction For Testing

# receives a block no and a predicate indicating

# whether the transaction is a coinbase transaction

######################################################

def make_random_transaction(blockno, is_coinbase):

    txn = {}

    txn["version"] =  "1"

    txn["transactionid"] = rcrypt.make_uuid()

    if is_coinbase == True: txn["locktime"] = hconfig.conf["COINBASE_LOCKTIME"]

    else: txn["locktime"] = 0

    # the public-private key pair for this transaction

    transaction_keys = rcrypt.make_ecc_keys()

    # previous transaction fragments spent by this transaction

    total_spendable = 0

    #######################

    # Build the vin array

    #######################

    txn["vin"] = []

    # genesis block transactions have no prior inputs.

    # coinbase transactions do not have any inputs

    if (blockno > 0) and (is_coinbase != True):

        max_inputs = secrets.randbelow(hconfig.conf["MAX_INPUTS"])

        if max_inputs == 0: max_inputs = hconfig.conf["MAX_INPUTS"] - 1

        # get some random previous unspent transaction

        # fragments to spend

        ind = 0

        ctr = 0

        while ind < max_inputs:

            # get a random unspent fragment from a previous block

            index = secrets.randbelow(len(unspent_fragments))

            frag_dict = unspent_fragments[index]

            key = [*frag_dict.keys()][0]

            val = [*frag_dict.values()][0]

            if val["blockno"] == blockno:

                ctr += 1

                if ctr == 10000:

                    print("failed to get random unspent fragment")

                    return False

                continue

            unspent_fragments.pop(index)

            total_spendable += val["value"]

            tmp = hchaindb.get_transaction(key)

            if tmp == False:

                print("cannot get fragment from chainstate: " + key)

            assert tmp != False

            assert tmp["spent"] == False

            assert tmp["value"] > 0

            # create a random vin element

            key_array = key.split("_")

            signed = rcrypt.sign_message(val["privkey"], val["pubkey"])

            ScriptSig = []

            ScriptSig.append(signed)

            ScriptSig.append(val["pubkey"])

            txn["vin"].append({

                    "txid": key_array[0],

                    "vout_index": int(key_array[1]),

                    "ScriptSig": ScriptSig

                })

            ctr = 0

            ind += 1

    #####################

    # Build Vout list

    #####################

    txn["vout"] = []

    # genesis block

    if blockno == 0:

        total_spendable = secrets.randbelow(10_000_000) + 50_000

    # we need at least one transaction output for non-coinbase

    # transactions

    if is_coinbase == True: max_outputs = 1

    else:

        max_outputs = secrets.randbelow(hconfig.conf["MAX_OUTPUTS"])

        if max_outputs == 0: max_outputs = 6

    ind = 0

    while ind < max_outputs:

        tmp = rcrypt.make_SHA256_hash(transaction_keys[1])

        tmp = rcrypt.make_RIPEMD160_hash(tmp)

        ScriptPubKey = []

        ScriptPubKey.append("<DUP>")

        ScriptPubKey.append("<HASH-160>")

        ScriptPubKey.append(tmp)

        ScriptPubKey.append("<EQ-VERIFY>")

        ScriptPubKey.append("<CHECK-SIG>")

        if is_coinbase == True:

            value = hmining.mining_reward(blockno)

        else:

            amt = int(total_spendable/max_outputs)

            value = secrets.randbelow(amt)   # helium cents

            if value == 0:

               pdb.set_trace()

               value = int(amt / 10)

            total_spendable -= value

            assert value > 0

            assert total_spendable >= 0

        txn["vout"].append({

                  "value": value,

                  "ScriptPubKey": ScriptPubKey

                })

        # save the transaction fragment

        fragid = txn["transactionid"] + "_" + str(ind)

        fragment = {}

        fragment[fragid] = { "value":value,

                             "privkey":transaction_keys[0],

                             "pubkey":transaction_keys[1],

                             "blockno": blockno

                           }

        unspent_fragments.append(fragment)

        print("added to unspent fragments: " + fragid)

        if total_spendable <= 0: break

        ind += 1

    return txn

########################################################

# Build some random Synthetic Blocks For Testing.

# Makes num_blocks, sequentially linking each

# block to the previous block through the prevblockhash

# attribute.

# Returns an array of synthetic blocks

#########################################################

def make_blocks(num_blocks):

    ctr = 0

    blocks = []

    global unspent_fragments

    unspent_fragments.clear()

    hblockchain.blockchain.clear()

    while ctr < num_blocks:

        block = {

                "prevblockhash": "",

                "version": "1",

                "timestamp": int(time.time()),

                "difficulty_bits": 20,

                "nonce": 0,

                "merkle_root": "",

                "height": ctr,

                "tx": []

            }

        # make a random number of transactions for this block

        # genesis block is ctr == 0

        if ctr == 0: num_transactions = 200

        else:

            num_transactions = secrets.randbelow(50)

            if num_transactions == 0: num_transactions = 40

            num_transactions = 2

        txctr = 0

        while txctr < num_transactions:

            if ctr > 0 and txctr == 0: is_coinbase = True

            else: is_coinbase = False

            trx = make_random_transaction(ctr, is_coinbase)

            assert trx != False

            block["tx"].append(trx)

            txctr += 1

        if ctr > 0:

            block["prevblockhash"] = \

               hblockchain.blockheader_hash(hblockchain.blockchain[ctr - 1])

        ret = hblockchain.merkle_root(block["tx"], True)

        assert ret != False

        block["merkle_root"] = ret

        ret = hblockchain.add_block(block)

        assert ret == True

        blocks.append(block)

        ctr+= 1

    return blocks

###################################################

# make the simulated blockchain

###################################################

startup()

make_blocks(500)

print("finished synthetic blockchain construction")

stop()

Directory File Constraints

A running Helium node will accumulate a large number of block files in its data directory. This raises the issue of the maximum number of files that can be written into a directory. This constraint depends upon the underlying operating system. In Linux with the ext4 filesystem, there is no theoretical limit on the number of files that can be written to a directory. We can nevertheless elect to write blocks into a number of sub-directories using a sharding algorithm, such as the following:

    create n sub-directories of the data directory: 0, 1,2,3,...n-1

    for each block:

        compute the SHA-256 hash of the block.

        convert this hash into an integer: hash_num.

        compute the n-modulus of hash_num: hash_mod = hash_num%n

        write the block into directory hash_mod

Chainstate Maintenance

In a typical cryptocurrency transaction, an entity spends previous transaction values that have been received and not spent. The Chainstate database keeps track of all of the spent and unspent values in the cryptocurrency ecosystem. Each such value is represented as a transaction fragment. At each point in time, the Chainstate provides a snapshot of the spent and unspent fragments in the system. You can refer to Chapter 12 to review the theory of the Chainstate.

If a Helium node has the blockchain, it can always rebuild the Chainstate database . The following algorithm in pseudo-code accomplishes this:

for each block:

        for each transaction in the block:

            let transaction id be trnid.

            let ctr = 0.

            for each vout element in the transaction:

                  fragment_id = trnid + "_" + string(ctr)

                  value = vout["value"]

                  hash = RIPEMD160(SHA256(public_key))

                  chainstate[fragment_id] = {"value": value,

                                             "spent": False,

                                             "tx_chain": "",

                                             "pkhash": hash

                                            }

             ctr += 1

            for each vin element in the transaction:

                  fragment_id = vin["txid] + "_" + vin["vout_index"]

                  chainstate[fragment_id] = {"value": value,

                                             "spent": True,

                                             "tx_chain": some_fragment,

                                             "pkhash": hash

                                            }

The following program code rebuilds the Chainstate database from scratch from the collection of block files. You will notice that this code leverages the hchaindb module and, in particular, uses the update_transaction function in this module to build the Chainstate database.

In order to exercise this code, copy this code into the file build_chainstate.py and save it in the unit_tests directory. Then make a simulated blockchain with a height of 500 blocks in the unit_tests directory. Finally, delete the Chainstate database, helium_db in the unit_tests/helium sub-directory (rm -rf heliumdb), if it exists. The following command will rebuild the Chainstate:

    $(virtual) python build_chainstate.py

###########################################################################

# build_chainstate: Builds the chainstate database from the blockchain files.

###########################################################################

import hblockchain

import hchaindb

import pickle

import os.path

import pdb

def startup():

    """

    start the chainstate database

    """

    # start the Chainstate Database

    ret = hchaindb.open_hchainstate("heliumdb")

    if ret == False:

        print("error: failed to start Chainstate database")

        return

    else: print("Chainstate Database running")

def stop():

    """

    stop the chainstate database

    """

    hchaindb.close_hchainstate()

def build_chainstate():

    '''

    build the Chainstate database from the blockchain files

    '''

    blockno  = 0

    tx_count = 0

    try:

        while True:

            # test whether the block file exists

            block_file = "block" + "_" + str(blockno) + ".dat"

            if os.path.isfile(block_file)== False: break

            # load the block file into a dictionary object

            f = open(block_file, 'rb')

            block = pickle.load(f)

            # process the vout and vin arrays for each block transaction

            for trx in block["tx"]:

                ret = hchaindb.transaction_update(trx)

                tx_count += 1

                if ret == False:

                    raise(ValueError("failed to rebuild chainstate. block no: " + \

                          str(blockno)))

            blockno += 1

    except Exception as err:

        print(str(err))

        return False

    print("transactions processed: " + str(tx_count))

    print("blocks processed: " +  str(blockno))

    return True

# start the chainstate build

print("start build chainstate")

startup()

build_chainstate()

print("chainstate rebuilt")

stop()

The build_chainstate is available in Appendix 9.

blk_index Maintenance

In Chapter 12, we discussed the blk_index database . This is a key-value store that determines the block in which a particular transaction is located. The keys are the transaction IDs and the values are the block heights.

Like the Chainstate database, the blk_index database can be rebuilt from the blockchain files.

The following pseudo-code rebuilds blk_index from the blockchain:

for block in blockchain:

      for transaction in block:

            blk_index[transaction[transactionid]] = block["height"]

The following module, build_blk_index, rebuilds the blk_index store. Copy the following code into the build_blk_index.py file and save it in the unit_tests directory. Delete the hblk_index sub-directory, if it exists.

We can now rebuild the blk_index database:

$(virtual) python build_index.py

#########################################################################

# build_blk_index: rebuilds the blk_index database from the blockchain

#########################################################################

import blk_index

import pickle

import os.path

import pdb

def startup():

    """

    start/create the blk_index database

    """

    # start the blk_index Database

    ret = blk_index.open_blk_index("hblk_index")

    if ret == False:

        print("error: failed to start blk_index database")

        return

    else: print("blk_index Database running")

    return True

def stop():

    """

    stop the blk_index database

    """

    blk_index.close_blk_index()

def build_blk_index():

    '''

    build the blk_index database from the blockchain

    '''

    blockno = 0

    tx_count = 0

    try:

        while True :

            # test whether the block file exists

            block_file = "block" + "_" + str(blockno) + ".dat"

            if os.path.isfile(block_file)== False: break

            # load the block file into a dictionary object

            f = open(block_file, 'rb')

            block = pickle.load(f)

            # process the transactions in the block

            for trx in block["tx"]:

                ret = blk_index.put_index(trx["transactionid"], blockno)

                tx_count += 1

                if ret == False:

                    raise(ValueError("failed to rebuild blk_index. block no: " \

                       + str(blockno)))

            blockno += 1

    except Exception as err:

        print(str(err))

        return False

    print("transactions processed: " + str(tx_count))

    print("blocks processed: " +  str(blockno))

    return True

# start the build_index build

print("start build blk_index")

ret = startup()

if ret == False:

    print("failed to open blk_index")

    os._exit(-1)

build_blk_index()

print("blk_index rebuilt")

stop()

Conclusion

In this chapter, we have considered some of important maintenance aspects of Helium. These have included obtaining a blockchain for a new node and synchronizing the blockchain held by a node if a node goes down or if there is a temporary network partition. We have also exposed code to rebuild the Chainstate database and the blk_index database from the blockchain files.

In the next chapter, we will develop a user wallet for Helium.

Footnotes

1

The block filename can be specified as a full or partial path. Otherwise, we can start the networknode script from a specific directory, such as the data directory. As you may have noticed, our unit tests were executed from the unit_tests directory and the Helium blocks were thus written into this directory.