‍

Today we’d like to present a major upgrade coming to the DeSo blockchain called Hypersync. Below is a simple FAQ for Hypersync. If you want to get into the details of how it all works you can do so here.

What is Hypersync?

Hypersync is the fastest most scalable way to download a blockchain. 

DeSo engineer Piotr Nojszewski explains “Hypersync is a new approach to node synchronization designed for infinite-state blockchains that is orders of magnitude faster than traditional block synchronization.” 

What Problem Does Hypersync Solve and Why Does it Matter?

One of the key problems that blockchains first solved is trustless data synchronization. This means downloading all the financial transactions, user balances, and importantly, in the case of DeSo, all the social data that is stored on the blockchain. 

It is crucial for this synchronization process to be as efficient as possible so that the blockchain data can be shared quickly across the network. At the same time lowering synchronization time directly translates to better protocol decentralization as the barrier to entry for new blockchain validators decreases. 

The existing state-of-the-art approaches to blockchain synchronization have worked well with finite-state blockchains that store up to 100GB of state. 

However, inifinite-state blockchains like DeSo are intended to store 10s, or 100s of TBs of state data, or even more. We discovered that the existing blockchain synchronization approaches are unable to scale to this level. 

Yet, in order for DeSo to reach its full potential and offer an infrastructure capable of sustaining billions of social media users, the synchronization process needed drastic improvements. As a solution to these pressing scalability issues, Hypersync was born.

In short: Hypersync is the result of months of cryptographic and distributed systems research, yielding a significant improvement to the existing state-of-the-art blockchain state sync approaches.

How Hypersync Works

Traditionally, blockchains are synced by downloading blocks. The node will download all blocks starting from the genesis (first block) up to the tip (latest block), along with all the transactions contained within these blocks. Next, the node will reprocess all these transactions – i.e. process EVERY single transaction that ever happened in the blockchain’s history. Finally, after hours of transaction processing, the node will arrive at the latest blockchain state.

As you can tell, this process seems rather inefficient. These historical transactions can be overlapping many times. For instance, if you sent someone $100, and they sent you back $100 – these two transactions don’t change how much money any of you has, yet these are two separate transactions that would have had to be processed while we were syncing blocks.

A better solution could be syncing state. Imagine that instead of downloading and reprocessing all transactions, you just download the result of these transactions. That is, imagine you just download the resulting “balance sheet,” also referred to as the “snapshot,” of how much money each user has. 

In this approach, it doesn’t matter how many transactions the users made; be it 100, 100k, or 100m, what matters is who and how much money they have at the end. Downloading this snapshot is in turns significantly more efficient than downloading all the blocks. Hypersync works in a similar fashion; however, with a few game-changing improvements, we termed EllipticSum and Ancestral Records.

EllipticSum

First, consider what could go wrong in the above-simplified state sync model. The major problem turns out to be confirming that the snapshot really is the result of applying all these historical transactions. This problem is referred to as verifying state integrity. 

In fact, the mathematics that allowed us to verify state integrity was the slowest part in existing state sync approaches. Our first order of business while making Hypersync was getting rid of this bottleneck.

By using a completely new algorithm we’ve developed, called EllipticSum, based on cutting-edge Elliptic Curve Cryptography, we were able to solve the state integrity problem with great efficiency. 

Moreover, where existing solutions to the state integrity problems (namely Merkle trees) required storing 10s of GB of additional dummy data, EllipticSum only stores a single 33byte number. Pause… yes, that’s a gigabyte vs a byte. And in fact, this storage difference combined with higher performance is what makes EllipticSum the key enabler when it comes to scaling to and beyond TBs of state data.

Ancestral Records

Another key upgrade that we made to the existing state sync approaches is how we compute these snapshots. Let’s return to that big balance sheet that we’ve previously mentioned. In order for blockchain nodes to download this balance sheet, it needs to be standby, ready to be sent. As a result, blockchains make a copy of this balance sheet specifically used for sending snapshot data. This means that given a 100GB state, you might need to copy some 100GB of data just to serve these snapshots, and you’d be doing this periodically. That’s a lot, and completely unacceptable when our goal is scalability.

Ancestral Records reduce this overhead by only requiring the theoretical minimal amount of additional data needed to serve the snapshot. That’s as close to the ideal solution to this problem as we can get. 

In fact, Ancestral Records are inspired by existing efficient solutions to software version control; although these ideas were never applied to blockchains before. If you ever used the Git version control utility for software, you can think of Ancestral Records as a Git for the blockchain.

How Does The Hypersync Upgrade Help The DeSo Ecosystem?

Hypersync makes it faster to sync a node which means it’s much easier to run one which increases decentralization. It also helps with scalability and building applications on the decentralized social blockchain. It’s much easier to build because you can play around more in a local environment with Hypersync.

You can read more about Hypersync here if you want to get into the nitty-gritty details.

‍