Introducing OpenWeaver, a scalable framework to deploy relay networks on demand

We’re excited to release OpenWeaver, a framework to deploy a relay network across a cluster of nodes in only a matter of minutes. We have been using it to benchmark and test communication in different public blockchains for some time to great success. Not only is OpenWeaver scalable across users and connections, it’s also easily extensible to any blockchain. We hope that open-sourcing OpenWeaver will encourage greater decentralization, experimentation and innovative business models to develop in the blockchain space.

Continue reading below to deploy a relay network for your favourite blockchain(s) ;)

Motivation

We’ve had a number of conversations with blockchain teams where we were told that miners of their respective chains congregate in the same region and even more so run on the same cloud service to reduce latencies and circumvent networking problems that the blockchain otherwise faces. This is an unfortunate result of the scalability trilemma wherein decentralization is sacrificed to boost throughput benchmarks. 

However, having a vast majority of nodes running on data centers headquartered in just a few jurisdictions jeopardizes the tamper and censorship resistance properties of decentralized networks while also bringing into question their availability guarantees as well. On the contrary, decentralized systems are expected to be composed of nodes distributed across the globe and controlled by multiple non-cooperating authorities to make consensus around a ban impractical, similar to the fatigue that plagues climate change. 

Relay networks, unlike full nodes, are quick to setup and migrate and provide much of the same advantages of co-locating nodes or using the same cloud services. Moreover, they can be made to run over the public Internet using techniques from the field of SD-WAN. This allows nodes to operate from homes (or any arbitrary location) while mitigating the networking challenges that occur on doing so. In what follows we look into the reasons that make networking bottlenecks a significant concern, prior work in the Bitcoin community in addressing them via relay networks and how OpenWeaver could help you achieve the same.

Dissemination in pure P2P networks

Blockchains are inherently broadcast networks which means that every block and transaction is propagated to all the other nodes in the network. Blocks in PoW chains especially, have to be propagated fast enough so that the other miners mine on top of the latest block instead of mining at the same height as already mined ones. Such competing blocks at the same height lead to multiple branches existing simultaneously and are known as forks. 

Forks can occur due to natural as well as deliberate reasons. The most common natural cause behind the occurrence of forks is the block propagation delay. Blocks hop over a number of flaky nodes with unstable internet connections (contributing to packet losses and consequent retransmissions) and pass through trans-continental submarine links as they make their way to the other nodes spread across the globe. Unsurprisingly, the propagation requires some time leading to the occasional discovery and transfer of a competing block at the same height within the same time.

Some miners, malicious for some while economically rational for others, are also in a position to secretly mine a fork and release it selectively for nefarious reasons. The capacity to quickly propagate their privately mined blocks and transactions across the network in contrast to the organic peer-to-peer but slow propagation of competing blocks significantly enhances the profitability of such attacks (here and here). Interestingly, such intentional forks can be disguised as natural forks to make detection and social outrage (in the form of UASFs) difficult. As a result, slow propagation of blocks amongst honest miners via an unstructured P2P network raises valid security concerns.

FIBRE as Bitcoin’s Spinal Cord

As elucidated above, asymmetric network connectivity and communication capabilities make certain attacks easier and profitable for malicious actors. The occurrence of forks compromise open networks in two ways: 

1. They reduce the security of the main chain as hash power that could have been spent on securing one dominant chain is now dispersed across a couple of orphan branches that are eventually discarded making 51% attacks on the main chain slightly more easier. This could be considered an unfortunate leak off the security budget.
2. They encourage geographical clustering and incentivize miners to centralize. Blocks orphaned in branches that do not make it to the longest chain impose an economic loss on the miner who mined them as the associated coinbase transactions become worthless. Every moment that miners spend mining on top of old blocks while the latest block is being propagated through the network is time lost* with respect to the miner who mined the latest block or received it soon enough and is now building on top of it. This violates the fairness property of PoW chains driving miners with a lower proportion of hashpower out of competition.

These issues aren’t just theoretical, but have been observed in the live Bitcoin network. Greg Maxwell, a Bitcoin core developer once recalled, “A couple years ago as blocks started getting above 250k regularly we saw what looked like a rapid consolidation of miners towards the most popular pools. One of the reasons was smaller pools were experiencing high orphaning.... It seemed like we were rapidly on a trend where only a single pool would exist.”

To address these concerns, Matt Corallo, another Bitcoin core developer and Blockstream co-founder designed and deployed a relay network (homonymous to the protocol, FIBRE) consisting of a few servers strategically located across the globe in a star topology. Miners can connect to the closest node of the relay and use it to send and receive blocks from other nodes in the Bitcoin network. The optimizations (also see BIP 152), improvements and years of sysad experience culminated in over a 90% reduction in block propagation time and fork rate (more interesting charts and analysis here and here respectively).

OpenWeaver as Tech Crypto’s FIBRE

Marlin is blockchain-agnostic. As a result, to benchmark performances for various other blockchains, we required a FIBRE-like protocol and relay network for chains that FIBRE did not support. FIBRE’s implementation is currently closely intertwined with the Bitcoin Core software thus making quick modifications for other blockchains strenuous if not impossible. Consequently, we set out on designing an architecture that would not require significant changes to the core network once deployed to add support for new blockchains.

Some key features of OpenWeaver include:

1. Extensibility: OpenWeaver was built to be cross-platform from the start. Adding support for new blockchains irrespective of the language their client implementations are written in, Rust, Python, C++, Haskell or OCaml, requires minimal if not absolutely no changes to the core OpenWeaver network once deployed. A compatible SDK and full nodes/light clients of the concerned blockchain at the edge (to prevent spam) are all that are required to add support for a new chain (more on this later).
2. User-friendliness: FIBRE is understandably a core component of the Bitcoin ecosystem. The network layer, after all, forms the foundation for P2P applications. OpenWeaver could be a crucial piece of infrastructure for several other blockchains. It is, thus, necessary to keep the barrier to entry for deploying OpenWeaver as low as possible. Thus, as part of the OpenWeaver framework is included a set of deployment scripts to spin up nodes in your favourite cloud provider in a matter of minutes.
3. Monetization: While Matt Corallo runs the FIBRE network for free, we understand that not everyone is altruistic. Furthermore, where does volunteering fit in systems designed for environments with economically rational players and byzantine actors anyway! Thankfully, OpenWeaver comes packed with a simple micropayments implementation that can be optionally activated if operators wish to charge for their services.

And there’s more…

OpenWeaver can help reduce uncle rates, increase profitability of small miners, improve decentralization and enhance security. But, it also enables an entrepreneurial operator to achieve more out of his deployment.

DeFi: Not only does OpenWeaver enable quick transmissions of blocks, it also instantly transmits transactions. With participation in defi growing, so are increasingly the possibilities of arbitrage. Non-negligible block times and competition amongst traders to capture available opportunities often lead to gas wars. Such priority gas auctions and related miner-extractable value strategies were explored by Daian et al in Flash Boys 2.0. Competitive traders run full nodes across the globe to be on top of the current mempool and place competitive counter-bids. An OpenWeaver operator could easily satisfy such demand amongst traders as it comes equipped with low latency mempool sync, transaction relay and streaming support.

Meta-transactions: Wallets, exchange accounts, multiple tokens, gas prices, all make up for a terrible Web 3 experience. The concept of meta transactions allows third parties to pay on behalf of a user, hence providing the user an opportunity to interact with a dapp without holding ether. Building a meta transaction service on top of OpenWeaver is fairly straightforward and could be of interest to some operators.

Full-node incentivization: An OpenWeaver deployment need not be managed by a single operator! Instead a group of full nodes could jointly operate one and share returns. This would be an unique way to incentivize full nodes different from the decentralized Infuras that seek to incentivize state queries.

For the nerds

OpenWeaver can be roughly broken down into five layers.

At the base layer is UDP, the most lightweight networking protocol one can expect to satisfactorily work over the Internet or inside cloud networks today. On top of it, built are: (i) a discovery protocol to discover peers in the network and (ii) a messaging protocol to relay messages. Together they form the pubsub layer which is the backbone of the relay network. 

The discovery protocol is used to discover peers and automatically manage connections based on peer metrics making the network fault-tolerant and self-healing. Channels can be used to further multiplex various message classes in a single blockchain (blocks, votes, transactions, etc) or even multiple blockchains in the same core relay network!

The messaging protocol provides connection semantics, reliable in-order delivery, congestion control and stream multiplexing (no head-of-line blocking anymore!) which are necessary to make UDP work for our ^[1]case. It also provides transport layer encryption using modern Curve25519 based cryptography and Integrated Encryption Schemes which are well suited to p2p networks unlike TLS which requires a PKI and CA infrastructure in practice.

Forward error correction enables the messaging protocol to recover from intermittent packet losses without having to wait for retransmissions, thereby minimizing round trips and making the network more stable.

Cut-through ^[2]routing is automatically enabled for large messages to minimize the queueing delay from buffering. They are concurrently carried on dedicated streams with a witness mechanism to prevent any loops from forming in the forwarding tree. To minimize bandwidth usage, peers can be asked to skip a particular message they are in the process of sending if it's not needed. While cut-through routing can not be implemented in open networks to ensure spam checks can be run, OpenWeaver necessitates spam checks only once at the edge.

To top it all, a compression layer enables block compression and decompression by transparently replacing transactions in blocks with short ids at the sender’s end and reassembling them using previously propagated transactions at the receiver’s end.

Geek++

A few technicalities about the features highlighted in the introduction follow:

Multilingual: The core codebase is written in C++. However, OpenWeaver can also be used through a C wrapper which allows for easy interoperability with languages with FFI support. A Go SDK built using CGo is provided as an example. Feel free to submit pull requests to add support for other languages!

Integrations: OpenWeaver can be easily integrated either directly into full nodes using our SDK or connected to via gateways. If you need a gateway that’s not available out-of-the-box, it’s easy to build one yourself with the TCP bridge made available! All one requires is to run the bridge, make a TCP connection to it from the full node and forward any received messages. The bridge takes care of interfacing with OpenWeaver internally.

If code-level changes to the full node are not desirable, you can also build a shadow peer which integrates with the SDK. Our Ethereum gateway is available as an example which requires no changes to the full node beyond adding them as peers in the config .

Devops: OpenWeaver is easy to deploy and manage. Provisioning scripts using Pulumi are provided to spin up instances on popular cloud platforms. Ansible roles and playbooks are provided to easily deploy the various node types in a few commands. It includes an Elasticsearch instance and built-in dashboards to monitor network activity.

Is OpenWeaver a panacea to blockchain scaling issues?

It might be tempting to think that since OpenWeaver reduces block propagation times, it could be used to reduce block times or alternatively increase gas limits as well. Unfortunately, there are two issues. For one, while block propagation time is a significant scaling bottleneck and has been at the center of block size debates, it isn’t the only bottleneck. The fast increasing state size, initial sync times, disk accesses and validation requirements are of equally great concern. Thankfully, there are researchers working on optimizing those fronts. However, OpenWeaver alone still wouldn’t be a secure scaling solution and that brings us to the second argument.

OpenWeaver helps the ecosystem - it encourages decentralization as nodes need not congregate in the same geographical location and also not be significantly disadvantaged, economically speaking, for doing so. But, OpenWeaver deployments would still be run by a select few entities. The networking layer forms the basis of blockchains as for any P2P application. Having centralized entities control such a crucial piece of the stack undoes the advances in decentralization brought about by blockchains. So, while OpenWeaver is an effective add-on to current blockchains, relying on it directly to scale the latter is very insecure as in that case, its role no longer remains optional. Shameless plug: we, at Marlin, are working on designs that attempt to address these security issues, some of which use a modified version of OpenWeaver in the architecture. 

I’m hungry for more

Read through the article, browsed the codebase and wondering what to do next? Trying to figure the best places to run nodes in terms of cost and performance? Or perhaps curious about how to cross the Great Firewall. Join our discord channel or forum to share and learn from each other’s experiences on discovering optimal paths, cheapest VPS alternatives or just shitbpost about the latest networking buzz. The codebase is under constant development and will keep maturing with time. Do help us by raising issues, pull requests or contributing to documentation.

[1] QUIC

[2] Falcon network

*It was once estimated by Corallo that a 10 second delay in receiving newly mined blocks resulted in a 1.5% loss in mining revenue. These numbers might have changed since then. Moreover, the loss isn’t as stark in Ethereum where orphan blocks are also rewarded.

Make sure you’re always up to date by following our official channels:

Twitter | Telegram Announcements | Telegram Chat | Discord | Website