How to Build a Solana Indexer: Streams, Backfill, Decoding and Reconciliation

Short answer: a Solana indexer consumes the chain's account and transaction streams and transforms them into a queryable database specific to your application. Token holders, pool activity, NFT transfers, and protocol positions: whatever your product needs, in order to answer faster than an RPC call can. The indexer architecture consists of a stream consumer (Geyser gRPC for live), a backfill source (archive RPC for history), a decoder for every program, an idempotent database writer, and a gap/reorg handling routine. This post describes every part, the decisions that make the indexer robust under load, and the underlying infrastructure.

Key Takeaways

• Index via Geyser gRPC stream, not via polling. Filters on the server side provide exactly the accounts and programs of interest, at node-processing latency.
• Backfill history via an archive once, stay live afterwards. Combining them without a cutover point usually results in holes in the index.
• Every write must be idempotent and keyed by the slot and the signature, due to stream replays and forks.
• Decode using the program's own layout and version the decoder; program upgrades cause silent corruption most often.

What an indexer is for

An indexer is a tool to precompute answers to your questions and serve them from a database. Solana's RPC provides the answers to such questions as "this account now" or "that transaction happened". But it does not answer aggregate and historical questions efficiently: who holds a token, the hourly volume of a pool, the ownership history of an NFT. The indexer computes the answers by consuming the stream of account writes and transactions and storing them into the database of your choice.

The indexer output serves your product needs, whether it's a relational database for analytics, a hot-path key-value database, or a search engine for discovery. While the output varies between products, the input is always the same: the account writes and transactions, filtered to the programs you are interested in, and ordered by slot.

Architecture

The live consumer

Subscribe to the stream using Yellowstone gRPC filters for the accounts (by owner program with memcmp filters on discriminators in case multiple account types exist) and the transactions (including your programs). If the indexer does not handle rollbacks, request confirmed commitment; processed commitment is earlier, but can be rolled back.

Track the slot number on every message. Your consumer's current slot number vs the chain's current slot number is the lag metric that shows how well your indexer keeps up with the chain, and a hole in the series of slots is a trigger for a gap reprocess. Yellowstone gRPC explained describes the stream subscription model; WebSockets vs gRPC explains why the stream uses gRPC at this scale.

Backfill

The history prior to the beginning of the live feed comes from archive RPC. The usual approach:

1. Find the slot number at which your live consumer starts.
2. Walk getSignaturesForAddress for your programs backwards starting from this slot, paging until you find the first slot that needs to be indexed.
3. Fetch each transaction via getTransaction (or a block via getBlock if the density of the transactions is high enough) and pass it through the same decoder and writer as in the live path.
4. Stop at the cutover slot number. Transactions newer than this one are live, and older are from backfill. They should not overlap without idempotent writes.

Backfill is where the request budgets go to die. Batch, page carefully, and use the plan whose archive access and rate limits match the job; historical data is this tier in OrbitFlare, and Solana RPC pricing explained explains why archival calls are expensive under credit pricing.

Decoding

The account data are bytes arranged according to the layout defined by the owning program; the instruction data as well. Decode using the IDL or layout of the program, and treat the decoder as versioned code: in case of a program upgrade and a layout change, the old accounts may contain the old layout until rewritten. A decoder that relies on a single layout silently corrupts the index.

Three practices:

• Key decoders by the program ID and discriminator, and handle the unknown discriminators by logging and skipping, not guessing.
• Calculate token balance differences using pre and post token balances from the transaction metadata. They are independent from the instruction decoding and thus work regardless of any unknown layout.
• Store the raw bytes in addition to decoded records for a retention period to be able to replay the decoding fix without fetching everything again.

Writing idempotently

The streams replay after disconnection, backfill overlaps the live feed at the cutover point, and forks at the processed commitment may deliver the same slot twice with different content. The writer must be ready for all three: every record is keyed by the slot and the signature (and the instruction index where relevant), and writing the same key twice is either no operation or an overwrite, never a duplicate.

Two patterns:

• Upsert by the key for current-state tables (holder balances, pool state).
• Append-only events are keyed by the signature and the index for history, with a unique constraint that excludes duplications.

Commit in slot order and track the last completely written slot as a checkpoint; on restart, the indexing resumes from this point.

Reconciliation

Gaps and reorgs happen and should be expected, not an exception.

• Gaps: if your consumer skips some slots in its sequence, refetch the corresponding blocks from the archive RPC and process them with the pipeline.
• Reorgs: at processed commitment, a slot can be superseded. Either index at confirmed (easier option) or maintain the window of unconfirmed writes that can be rolled back when a slot is superseded.
• Drift checks: regularly compare the sample of indexed state against live getAccountInfo requests; the discrepancy means some problem with the decoder or writing.

Infrastructure

• Streaming: a gRPC endpoint in the region where the indexer is running, with the throughput generated by your filters.
• Archive: an endpoint with the required history depth and a request budget that is capable of surviving the backfill.
• Dedicated capacity when your filter volume is high or backfill is continuous; a dedicated node removes shared limits from both paths.
• Database: sized for write throughput in slot order, with indexes optimized for your queries rather than for ingest.

Conclusion

A Solana indexer is a live gRPC consumer with an archive backfill and a cutover point, with a decoder per program, idempotent database writer keyed by slot and signature, and a reconciler that handles gaps and reorgs. Start by getting the keys and commitment level right, since they define whether the indexer will stay correct under load or not. Then choose the streaming and archive endpoints that match your filters and backfill needs, and implement the drift checks to avoid undetected corruption.

OrbitFlare's gRPC service covers the live path, historical data the backfill, and dedicated nodes the volume.