The Evolution of Web3 Data Access: Overview of Indexers and Related Projects

The core of blockchain technology is data. It is the foundation for developing decentralized applications ( dApp ). While most discussions currently focus on data availability ( DA ), data accessibility is equally important yet often overlooked.

In the era of modular blockchains, DA solutions have become indispensable. They ensure that all participants can access transaction data, enabling real-time verification and maintaining network integrity. However, the functionality of the DA layer resembles a billboard rather than a database. This means that data is not stored indefinitely but will be deleted over time.

In contrast, data accessibility focuses on the ability to retrieve historical data, which is crucial for developing dApps and conducting blockchain analysis. Although it is less frequently discussed, it is equally important as data availability. Both play different but complementary roles in the blockchain ecosystem, and a comprehensive data management approach must address both issues simultaneously to support robust and efficient blockchain applications.

Traditional blockchain data retrieval methods

Since its inception, blockchain has fundamentally changed the infrastructure, driving the creation of dApps in areas such as gaming, finance, and social networking. However, building these dApps requires access to a large amount of blockchain data, which is both difficult and costly.

For dApp developers, one option is to host and run their own archival RPC nodes. These nodes store all historical blockchain data from the beginning, allowing full access to the data. However, maintaining archival nodes is costly, and their query capabilities are limited, making it impossible to query the data in the format required by developers. While running cheaper nodes is an option, these nodes have limited data retrieval capabilities, which may hinder the operation of dApps.

Another approach is to use commercial RPC node providers. These providers handle the costs and management of the nodes and provide data through RPC endpoints. Public RPC endpoints, while free, have rate limits that may affect the user experience of dApps. Private RPC endpoints offer better performance by reducing congestion, but even simple data retrieval requires substantial communication. This makes them resource-intensive and inefficient for complex data queries. Additionally, private RPC endpoints are often difficult to scale and lack cross-network compatibility.

Better Solutions: Blockchain Indexer

Blockchain indexers play a critical role in organizing chain data and sending it to databases for convenient querying, hence they are often referred to as "the Google of blockchain." They index blockchain data and make it available using a SQL-like query language ( such as GraphQL API ). By providing a unified data query interface, indexers allow developers to quickly and accurately retrieve the information they need using standardized query languages, significantly simplifying the process.

Different types of indexers optimize data retrieval in various ways:

1. Full Node Indexer: Runs a complete blockchain node and extracts data directly, ensuring data completeness and accuracy, but requires significant storage and processing power.

2. Lightweight Indexer: Relies on full nodes to obtain specific data on demand, reducing storage requirements but potentially increasing query time.

3. Dedicated Indexer: Optimized for specific types of data or specific blockchains, such as NFT data or DeFi transactions.

4. Aggregated Indexer: Extracts data from multiple blockchains and sources, including off-chain information, providing a unified query interface, which is particularly useful for multi-chain dApps.

Ethereum alone requires 3TB of storage space, which continues to increase as the blockchain grows. The indexer protocol deploys multiple indexers, enabling efficient indexing and high-speed querying of large amounts of data, which cannot be achieved by RPC.

Indexers also allow for complex queries, easy data filtering, and post-analysis extraction. Some indexers can aggregate multi-source data, avoiding the need for multi-chain dApp deployment of multiple APIs. By being distributed across multiple nodes, indexers provide enhanced security and performance, whereas RPC providers may experience interruptions and downtimes due to centralized characteristics.

Overall, compared to RPC node providers, indexers improve data retrieval efficiency and reliability while reducing the deployment costs of a single node. This makes blockchain indexer protocols the preferred choice for dApp developers.

Indexer Application Scenarios

Building a dApp requires retrieving and reading blockchain data to operate services. This includes various types of dApps, such as DeFi, NFT platforms, games, and even social networks, as these platforms need to read data first to execute other transactions.

DeFi

DeFi protocols require different information to provide users with specific prices, rates, fees, etc. The automated market maker (AMM) needs the price and liquidity information from the liquidity pool to calculate the swap rate, while lending protocols need utilization to determine borrowing rates and liquidation debt ratios. It is essential to input the information into the dApp before calculating the execution rate for users.

Game

GameFi requires quick indexing and access to data to ensure a smooth gaming experience for users. Only through rapid data retrieval and execution can Web3 games match the performance of Web2 games, thus attracting more users. These games need data such as land ownership, in-game token balances, and in-game operations. By using indexers, they can better ensure stable data flow and uptime, guaranteeing a perfect gaming experience.

NFT

NFT markets and lending platforms need to index data to access various information, such as NFT metadata, ownership and transfer data, royalty information, etc. Quickly indexing such data can avoid browsing through each NFT one by one to find ownership or attribute data.

Whether it's a DeFi AMM that requires price and liquidity information, or a SocialFi application that needs to update new user posts, quick data retrieval is crucial for the normal operation of dApps. With the help of an indexer, they can efficiently and accurately fetch data, providing a smooth user experience.

Analysis

The indexer provides a method to extract specific data from the raw blockchain data (, including smart contract events in each block ). This creates opportunities for more specific data analysis, thereby providing comprehensive insights.

For example, perpetual trading protocols can identify which tokens have high trading volumes and which tokens generate fees, thereby deciding whether to list them as perpetual contracts on the platform. DEX developers can create dashboards for their products to gain insights into which liquidity pools have the highest returns or the strongest liquidity. They can also create public dashboards that allow developers to flexibly query any type of data to be displayed on charts.

As there are multiple blockchain indexers available, identifying the differences between indexing protocols is essential to ensure that developers choose the indexer that best meets their needs.

Overview of Blockchain Indexer ###

The Graph

The Graph is the first indexing protocol launched on Ethereum, which allows for easy querying of transaction data that was previously difficult to access. It uses subgraph definitions and filters to collect subsets of data from the blockchain, such as all transactions related to a specific platform's USDC/ETH pool.

Using index proof, indexers stake the native token GRT for indexing and querying services, and delegators can choose to stake their tokens here. Curators can access high-quality subgraphs, helping indexers determine which subgraphs to compile data for to earn the best query fees. During the transition to greater decentralization, The Graph will ultimately stop its hosting services, requiring subgraphs to upgrade to its network while providing upgraded indexers.

Its infrastructure allows the average cost per million queries to reach $40, far lower than self-hosted nodes. By using file data sources, it also supports parallel indexing of both on-chain and off-chain data, enabling efficient data retrieval.

The rewards for The Graph's indexers have steadily increased over the past few quarters. This is partly due to the increase in query volume and also attributed to the rise in token prices, as they plan to integrate AI-assisted queries in the future.

Subsquid

Subsquid is a peer-to-peer, horizontally scalable decentralized data lake that efficiently aggregates large amounts of on-chain and off-chain data, protected by zero-knowledge proofs. As a decentralized worker network, each node is responsible for storing a specific subset of block data, speeding up the data retrieval process by quickly identifying the nodes that hold the required data.

Subsquid supports real-time indexing, allowing indexing before block finalization. It also supports storing data in formats chosen by developers, making it easier to analyze using tools like BigQuery, Parquet, or CSV. Additionally, subgraphs can be deployed on the Subsquid network without the need to migrate to the Squid SDK, enabling no-code deployment.

Although still in the testnet phase, Subsquid has achieved impressive statistics, with over 80,000 testnet users, deployed over 60,000 Squid indexers, and more than 20,000 verified developers on the network. Recently, Subsquid launched the mainnet for its data lake.

In addition to indexing, the Subsquid Network data lake can also replace RPC in use cases such as analytics, ZK/TEE co-processors, AI agents, and Oracles.

SubQuery

SubQuery is a decentralized middleware infrastructure network that provides RPC and indexing data services. It originally supported the Polkadot and Substrate networks and has now expanded to include over 200 chains. Its operation is similar to The Graph, which uses indexing proofs; indexers index data and provide query requests, while delegators stake shares to indexers. However, it introduces consumers to submit purchase orders, indicating that the indexer's revenue is guaranteed, rather than that of a manager.

It will introduce SubQuery data nodes that support sharding, preventing constant synchronization of new data between each node, optimizing query efficiency, and moving towards greater decentralization. Users can choose to pay approximately 1 SQT token as a computation fee for every 1000 requests, or set custom fees for the indexer through the protocol.

Although SubQuery only launched its token earlier this year, the issuance rewards for nodes and delegators have also increased month-on-month in USD value, representing a continuous increase in the number of query services offered on its platform. Since the TGE, the total amount of staked SQT has increased from 6 million to 125 million, highlighting the growth in network participation.

Covalent

Covalent is a decentralized indexing network, created by block sample producers (BSP) network nodes that create copies of blockchain data through batch exports and publish proofs on the Covalent L1 blockchain. This data is then refined by block result producers (BRP) nodes according to established rules, filtering out the data that meets the requirements.

Through a unified API, developers can easily extract relevant blockchain data in a consistent request and response format without having to write custom complex queries to access the data. CQT tokens, which can be settled on a certain platform, can be used as a means of payment to extract these pre-configured datasets from network operators.

Covalent's rewards seem to exhibit an overall growth trend from the first quarter of 2023 to the first quarter of 2024, partly due to the increase in the price of Covalent token CQT.

Considerations for Choosing an Indexer

Data Customizability

Some indexers ( like Covalent ) are general-purpose indexers that provide standard pre-configured datasets exclusively through APIs. While they may be fast, they do not offer the flexibility needed for developers who require custom datasets. Using an indexer framework allows for more customized data processing to meet specific application needs.

Security

Index data must be secure; otherwise, dApps built on these indexers are also vulnerable to attacks. For example, if transactions and wallet balances can be manipulated, the dApp may lose liquidity, affecting its users. Although all indexers adopt some form of security through staking tokens, other indexer solutions may use proofs to further enhance security.

Subsquid provides options for optimistic and zero-knowledge proofs, while Covalent has also released proofs that include block hash values. The Graph offers a dispute challenge period for indexer queries in an optimistic challenge window manner, whereas SubQuery generates Merkle Mountain proofs for each block, calculating the block hash values for all data stored in its database.

Speed and Scalability

As blockchain continues to grow, transaction volumes increase accordingly, making it more cumbersome to index large amounts of data due to the need for more processing power and storage space. As the blockchain network grows, maintaining efficiency becomes more difficult, but indexing protocols have introduced solutions to meet these increasing demands.