Dude, where did my block go?
It's official now, you can configure your hived node to get rid of some of the 490+ gigabytes of compressed block log burden. You can even shed all of it, but do you really want to and what if you change your mind?
The time of no choice is over
Since the beginning of Hive blockchain (and its predecessor) there was that huge single file named block_log, mandatory for all nodes. A single file with a size of over 490 gigabytes now, requiring continuous disk space of its size. The block log revolution that comes into force with 1.27.7rc0 tag brings following improvements:
- Multiple one-million-block-each block log files can be used instead of a legacy single monolithic file.
- You can keep all of the files or only a number of most recent ones.
- Complete wipeout of block log is possible too, leaving you with one last block only kept in memory.
The pros and cons
Let's examine the new modes in detail:
- Split mode - keeps each full block from genesis, hence provides full functionality of e.g. block API & allows blockchain replay. At the same time its 1M-blocks part files of block log may be physically distributed to different filesystems using symlinks which allows to e.g. keep only the latest part files on fast storage. Good for API node.
- Pruned mode - a variation of split mode, which keeps only several latest part files of block log. Replay is no longer guaranteed. Provides only partial functionality of block API & others - handles requests for not-yet-pruned blocks, e.g. serves latest several months of blocks through
block_api. Good for transaction broadcaster. - Memory-only mode - the ultimate pruning - no block log files at all, only single latest irreversible block held in memory. Unable to replay obviously. Unable to provide past blocks through block API & similar.
The summary of block log modes
Now we're getting to the question why replay is available sometimes in pruned mode. Full replay (from block #1) requires all blocks to be present in block log, therefore it can be performed as long as block log is not effectively pruned due to combination of block-log-split value in configuration and current head block of the blockchain. After the oldest part file containing initial 1 million blocks is removed, the block log is effectively pruned and full replay is no longer possible.
Block log files of nodes configured with different values of block-log-split option. Note the file size differences.
### Tips & tricks
- There are two ways to obtain split block log files from legacy monolithic one - a) Using
block_log_util's new --split option or b) running hived configured to have split block log with legacy monolithic one provided in its blockchain directory, which triggers built-in auto-split mechanism. The former is recommended as it allows to generate the 490+ GB of split files into output directory other than the source one (possibly on different disk space). - All files of split/pruned block log, except the head one (the latest one, with highest number in filename) can be made read-only as they won't be modified anymore. The head file needs to be writable as it's where the new blocks are applied to.
- Split block log allows to scatter its part files over several disk spaces and symlink them all in hived's
blockchain directory. Not only can smaller disk volumes be used, you can even consider placing older parts (i.e. the ones rarely used by hived) onto slower drives. - The names of split/pruned block log files follow the pattern
block_log_part.???? where ???? stands for consecutive numbers beginning with 0001 followed by 0002, etc. Since each one contains up to a million of blocks, block_log_part.0001 contains blocks numbered 1 to 1 000 000, while block_log_part.0002 contains blocks numbered 1 000 001 to 2 000 000 and so on. Hived recognizes the block log files by their names, so don't change them or it becomes lost.
Links and resources
Your feedback is invaluable and always welcome.
