I've been super busy for a while, so I fell behind a bit on reporting progress (it's been about 4 months since my last report...). Originally I was planning an earlier release of everything, but in the end we decided to push back the releases so that we could add more features that we considered important to completed.
Since so much work has been done since my last report, a report of all the details would be a bit overwhelming, so I decided to write this report in the form of earlier "release notes" for all the software, just to keep it compact.
I'm breaking the report up into two separate posts, this first one covers the backend services (e.g. hived, HAF, API servers, client libraries), and the next report which I'll publish in the next few days will cover work done on the user interfaces (denser, the block explorer UI, clive).
New features and functional improvements
HF28 features:
New configuration options to preventing hived nodes from being flooded by transactions. This included adding several new config options:
rc-flood-level,
rc-flood-surcharge, and
max-mempool-size (the new default settings are probably fine for most hived nodes). RC costs of incoming transactions now start increasing when high activity (flooding) is detected in order to reduce the number of pending transactions that can build up and consume RAM. Once a transaction is finally put into a block, costs are calculated using regular rules, so final RC costs are not affected by this change.
https://gitlab.syncad.com/hive/hive/-/merge_requests/1397https://gitlab.syncad.com/hive/hive/-/merge_requests/1390Better handling of shared_memory.bin allocation errors:
https://gitlab.syncad.com/hive/hive/-/merge_requests/1372 . Hived will produce an error if the shared_memory.bin will lead to an insufficient amount of free space on the filesystem holding shared memory. This change can prevent runtime failures during memory allocation and helps to find a mis-configuration of the shared memory.
Generated config.ini contains generated default option values as comments. This change makes it easier to detect which options have been customized, as such options will have an explicit setting in the file:
https://gitlab.syncad.com/hive/hive/-/merge_requests/1361Eliminated configuration options and plugins previously marked as deprecated:
https://gitlab.syncad.com/hive/hive/-/issues/649Support for pruned block log. Hived configuration options allows storing blocks in local blockchain subdirectory as:
- a single monolithic file (old legacy way),
- split into 1M block parts, which allows you to relocate files across different filesystems and symlink them in the blockchain directory
- pruned to store the latest N 1M block parts (including setting N=0 to avoid storing any blocks)
Block log pruning saves disk space, but also reduces the ability to support block_api usage via hived (although nowadays blocks are better served via HAfAH anyway) and P2P syncing of nodes that need to catch up to the current head block.
Hived can automatically split an existing monolithic block_log when the conf file is changed from monolithic to split storage.
The block-log-split option defaults to 9999 to enable split mode in the block storage. So, if you update an existing hived that is synced with the new version, it will automatically split your block log at startup (this one-time process takes around half an hour).
The block_log_util tool also has been extended to support file splitting.
Bugfixes
- Fixed hived crash that could occur when a block_log file doesn't exist https://gitlab.syncad.com/hive/hive/-/merge_requests/1367
- Fixed concurrency problems between witness and writer threads while evaluating blockchain operations: https://gitlab.syncad.com/hive/hive/-/merge_requests/1268
- Fixes related to the condenser_api and producing the JSON legacy form of operations: https://gitlab.syncad.com/hive/hive/-/merge_requests/1381
New features and functional improvements
- New API calls for HAF apps to allow for more flexible and less error-prone coding of an app's operation processing loop. One of the biggest benefits is that it is easier to make sure a HAF app will always be in a consistent state if it gets interupted and then restarted: https://gitlab.syncad.com/hive/haf/-/merge_requests/504
- Removed block_num, operation type and _timestamp columns to reduce SQL database storage size: https://gitlab.syncad.com/hive/haf/-/merge_requests/492 https://gitlab.syncad.com/hive/haf/-/merge_requests/496
- HAF maintenance actions are performed by pg_cron tool: https://gitlab.syncad.com/hive/haf/-/merge_requests/531
- HAF database uses C collation settings to speedup index search and creation: https://gitlab.syncad.com/hive/haf/-/merge_requests/514
Bugfixes
- Fixed database integrity problem during HAF node restart when reversible data has been collected: https://gitlab.syncad.com/hive/haf/-/merge_requests/532
- Corrected HAF node upgrade issues: https://gitlab.syncad.com/hive/haf/-/merge_requests/529
- Fixed errors in the autodetach procedure for dead apps: https://gitlab.syncad.com/hive/haf/-/merge_requests/525
- Fixed bugs leading to multiple application deadlock during attaching a context: https://gitlab.syncad.com/hive/haf/-/merge_requests/518
- Fixed deadlock during HAF data dumping: https://gitlab.syncad.com/hive/haf/-/merge_requests/505 https://gitlab.syncad.com/hive/haf/-/merge_requests/511
New features and functional improvements
- Switching to PostgREST as HTTP server provider from the current Python based implementation to improve performance and pave the way for a REST-based hivemind API.
- Hivemind APIs additionally return mute reason: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/626
- bridge.get_discussion API returns information related to pinned post: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/715
- bridge.get_account_posts supports observer parameter: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/716
- Community support improvements: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/711
- Sync performance optimizations: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/706 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/718 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/746
- Hivemind uses a new Reputation Tracker HAF application to calculate reputation data in a way matching hived's reputation_plugin algorithm.
- Reduced Hivemind docker image size: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/726
- Hivemind sync processing switched to use the new HAF application main loop: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/701
- Support for a new REST API including generated Swagger documentation. The REST API docs for Hafah are here: https://api.syncad.com/?urls.primaryName=HAfAH
New features and functional improvements
- Support for a new REST API including generated Swagger documentation. The REST API docs for balance tracker are here: https://api.syncad.com/?urls.primaryName=Balance+Tracker
- Support for balance history APIs: https://gitlab.syncad.com/hive/balance_tracker/-/merge_requests/105
- HAF's new app loop used for the operation processing code: https://gitlab.syncad.com/hive/balance_tracker/-/merge_requests/100
- Allow custom schema installation to support separate deployments of balance tracker in the same database (i.e. when balance_tracker is referenced from multiple parent applications or uses some specific configuration).
Created a new HAF application which calculates reputation values matching to Hived reputation plugin.
This application is used by Hivemind and HAF Block Explorer to provide reputation data needed for their APIs.
New features and functional improvements
- Support for a new REST API including generated Swagger documentation. The REST API docs for the block explorer are here: https://api.syncad.com/?urls.primaryName=HAF+Block+Explorer
- API calls accepting block range parameters can take a block number or a timestamp as block range constraints https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/214
- Witness API improvements:
- App sync uses new HAF main loop scheme: https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/196
- Reputation data is read from the reputation_tracker app's schema. Previously reputation calculations were done within the block explorer's sync thread, so syncing the block explorer is now about 2x faster than earlier versions (but we also track many more types of balances, so it is about the same sync speed as the initial release done back last September): https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/177
Initial release candidate of Hive integration library into Typescript (NodeJS/Web browser) environment. Wax allows developers to easily create and process Hive blockchain transactions and operations. It it is also easy to add support new custom API call definitions (e.g. API calls for custom HAF apps) which can then be called as regular functions.
On the implementation side, the Wax library directly shares C++ code from the Hive Protocol to always match the blockchain's behavior. Wax is an object-oriented library and is coded to allow for "intellisense" style support from IDE's such as Visual Code, which greatly simplifies writing client code with the library.
We are also in the process of developing a python version of Wax. The python version is already being used as we develop Clive, a new wallet for Hive, but it is not yet ready for official public integrations.
Beekeeper: light weight process for securely storing cryptographic keys
- New API endpoints:
has_wallet and ability to create temporary (not persisted) wallets. https://gitlab.syncad.com/hive/hive/-/merge_requests/1355 - New API endpoint to allow import of multiple keys: https://gitlab.syncad.com/hive/hive/-/merge_requests/1326
- Preliminary support for python wrappers to enable integrating Beekeeper tool into a python workflow: https://gitlab.syncad.com/hive/helpy/-/merge_requests/55
Bugfixes
- Fixed bug in wallet synchronization between opened multiple network sessions: https://gitlab.syncad.com/hive/hive/-/merge_requests/1344
- Improved session timeout handling: https://gitlab.syncad.com/hive/hive/-/merge_requests/1342
- Shutdown fixes: https://gitlab.syncad.com/hive/hive/-/merge_requests/1311
API Node Benchmarking
We're still making improvements to the backend services, but we have begun initial test of "full api node" syncing. Here's some tests I ran on one of our faster servers (AMD 7950 with 2x4TB T700 nvmes):
- HAF replay time: 15 hours
- Simultaneous replay of hafbe (30 hours), reptracker (35.6 hours), and hivemind (61.8 hours).
- So total time to get a fast node up with all services is 15 (haf) + 61.8 (hivemind, the longest of the three services) = 78.6 hours = 3 days 6.6 hours
I don't expect much improvement in the sync times prior to the release, but these numbers are looking pretty good, especially since the blockchain grows by about 1 million blocks per month and these numbers are still close to times from the release last year despite all the blocks that have been added since then.
What's next?
We're still adding features and working on documentation, but we have started doing real-world deployment testing, and the next step will be to do some production testing with real world traffic. At this point, I'm shooting for a December release, as I think we have too many useful features that should be released as soon as possible, so we shouldn't delay the releases to add many more features.
