> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/graphprotocol/graph-node/llms.txt
> Use this file to discover all available pages before exploring further.

# Database Sharding

> Scale Graph Node horizontally with PostgreSQL sharding strategies

When a `graph-node` installation grows beyond what a single Postgres instance can handle, you can scale the system horizontally by adding more Postgres instances. This is called **sharding**, and each Postgres instance is called a **shard**.

## Overview

The resulting `graph-node` system uses all Postgres instances together, essentially forming a distributed database. Sharding relies heavily on the fact that in almost all cases:

* Traffic for a single subgraph can be handled by a single Postgres instance
* Load can be distributed by storing different subgraphs in different shards

<Note>
  Sharding requires using a [configuration file](/advanced/config-file) rather than environment variables. It cannot be configured through CLI options alone.
</Note>

## The Primary Shard

In a sharded setup, one shard is special and is called the **primary**. The primary is used to store:

* System-wide metadata
* Mapping of subgraph names to IPFS hashes
* Directory of all subgraphs and the shards in which each is stored
* List of configured chains
* Metadata that rarely changes

<Note>
  Frequently changing metadata (such as subgraph head pointers) is stored in the individual shards, not the primary.
</Note>

## Setting Up Sharding

### Configuration File Setup

Add additional `[store.<name>]` entries to your configuration file:

```toml theme={null}
[store]
[store.primary]
connection = "postgresql://graph:password@primary-host:5432/graph"
pool_size = 10

[store.shard1]
connection = "postgresql://graph:password@shard1-host:5432/graph"
pool_size = 10

[store.shard2]
connection = "postgresql://graph:password@shard2-host:5432/graph"
pool_size = 10
```

See [Multiple Databases](/advanced/multiple-databases) for detailed configuration options.

### Prerequisites: Inter-Shard Communication

Sharding uses the [`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html) extension for inter-shard communication. Graph Node automatically sets up foreign servers and foreign tables.

<Warning>
  **Before setting up sharding, ensure:**

  1. All shards can communicate with each other over the network
  2. Firewall rules allow traffic from each shard to every other shard
  3. Authentication (`pg_hba.conf`) allows connections from all other shards
  4. Connection strings are in the format `postgres://USER:PASSWORD@HOST[:PORT]/DB` (Graph Node must parse these components)
</Warning>

### Initialization

When a new shard is added to the configuration file:

1. Graph Node initializes the database schema during startup
2. Foreign data wrappers are automatically configured
3. Metadata synchronization begins

### Verifying Inter-Shard Connectivity

After schema initialization, manually verify connectivity:

```sql theme={null}
-- Run on any shard to verify connection to primary
SELECT count(*) FROM primary_public.chains;

-- Run on primary to verify connection to a shard
SELECT count(*) FROM shard_shard1_subgraphs.subgraph;
```

<Note>
  The query result doesn't matter - success means connectivity works. Failures indicate network or authentication issues.
</Note>

### Metadata Synchronization

With multiple shards, Graph Node periodically copies metadata from the primary to all other shards:

* Copied metadata is needed to respond to queries
* Each query needs the primary to find which shard stores the subgraph's data
* Metadata copies enable queries when the primary is down

## Sharding Strategies

There are many ways to split data across shards. One recommended setup:

<CardGroup cols={2}>
  <Card title="Small Primary" icon="database">
    Minimal primary shard storing only system metadata
  </Card>

  <Card title="Low-Traffic Shards" icon="chart-line">
    Multiple shards for low-traffic subgraphs with many subgraphs per shard
  </Card>

  <Card title="High-Traffic Shards" icon="gauge-high">
    One or few shards for high-traffic subgraphs with few subgraphs per shard
  </Card>

  <Card title="Block Cache Shards" icon="cube">
    Dedicated shards storing only blockchain block caches
  </Card>
</CardGroup>

### Strategy: By Traffic Level

<CodeGroup>
  ```toml Configuration theme={null}
  [deployment]
  # High-traffic VIP subgraphs
  [[deployment.rule]]
  match = { name = "(vip|production)/.*" }
  shard = "vip"
  indexers = [ "index_node_vip_0" ]

  # Medium-traffic subgraphs
  [[deployment.rule]]
  match = { name = "medium/.*" }
  shard = "shard1"
  indexers = [ "index_node_1" ]

  # Low-traffic community subgraphs
  [[deployment.rule]]
  shards = [ "shard2", "shard3" ]  # Auto-distribute
  indexers = [ "index_node_community_0", "index_node_community_1" ]
  ```

  ```toml Store Config theme={null}
  [store]
  [store.primary]
  connection = "postgresql://graph:${PGPASSWORD}@primary:5432/graph"
  pool_size = 10

  [store.vip]
  connection = "postgresql://graph:${PGPASSWORD}@vip-db:5432/graph"
  pool_size = 50  # High capacity for VIP

  [store.shard1]
  connection = "postgresql://graph:${PGPASSWORD}@shard1:5432/graph"
  pool_size = 25

  [store.shard2]
  connection = "postgresql://graph:${PGPASSWORD}@shard2:5432/graph"
  pool_size = 15

  [store.shard3]
  connection = "postgresql://graph:${PGPASSWORD}@shard3:5432/graph"
  pool_size = 15
  ```
</CodeGroup>

### Strategy: By Blockchain Network

Store different networks in different shards:

```toml theme={null}
[chains.mainnet]
shard = "ethereum_shard"
provider = [ { label = "mainnet", url = "http://eth:8545", features = [] } ]

[chains.polygon]
shard = "polygon_shard"
provider = [ { label = "polygon", url = "http://polygon:8545", features = [] } ]

[chains.arbitrum]
shard = "arbitrum_shard"
provider = [ { label = "arbitrum", url = "http://arbitrum:8545", features = [] } ]

[deployment]
[[deployment.rule]]
match = { network = "mainnet" }
shard = "ethereum_shard"
indexers = [ "index_node_eth_0" ]

[[deployment.rule]]
match = { network = "polygon" }
shard = "polygon_shard"
indexers = [ "index_node_polygon_0" ]

[[deployment.rule]]
match = { network = "arbitrum" }
shard = "arbitrum_shard"
indexers = [ "index_node_arbitrum_0" ]
```

### Strategy: Dedicated Block Cache Shards

Separate block cache storage from subgraph data:

```toml theme={null}
[store.primary]
connection = "postgresql://graph:${PGPASSWORD}@primary:5432/graph"
pool_size = 10

[store.blocks]
connection = "postgresql://graph:${PGPASSWORD}@blocks-db:5432/graph"
pool_size = 20

[store.subgraphs]
connection = "postgresql://graph:${PGPASSWORD}@subgraphs-db:5432/graph"
pool_size = 30

[chains.mainnet]
shard = "blocks"  # Block cache in dedicated shard
provider = [ { label = "mainnet", url = "http://eth:8545", features = [] } ]

[deployment]
[[deployment.rule]]
shard = "subgraphs"  # Subgraph data in separate shard
indexers = [ "index_node_0" ]
```

## Copying and Moving Between Shards

Besides deployment rules for new subgraphs, you can copy and move existing subgraphs between shards.

### Creating a Copy

Start copying a subgraph from one shard to another:

```bash theme={null}
graphman copy create <source-deployment> <dest-shard>
```

<Note>
  A deployment is identified by its IPFS hash. Multiple copies of the same deployment can exist across shards, but only one copy per shard. Only one copy is marked as `active` for query responses.
</Note>

### Copy Behavior

By default, `graphman copy create`:

1. Copies source subgraph data up to the copy initiation point
2. Starts indexing independently from the source
3. Both source and destination continue indexing

### Copy with Activation

Automatically activate the copy when it catches up:

```bash theme={null}
graphman copy create --activate <source-deployment> <dest-shard>
```

This:

1. Copies data from source
2. Indexes independently until caught up to chain head
3. Marks copy as `active` when synced
4. Routes queries to the new copy

### Copy with Replacement

Replace the source with the copy:

```bash theme={null}
graphman copy create --replace <source-deployment> <dest-shard>
```

This:

1. Copies data from source
2. Indexes independently until caught up
3. Marks copy as `active`
4. Marks source as `unused`
5. Source is deleted \~8 hours after copy syncs (default reaper configuration)

<Warning>
  Copying large deployments can take a very long time (hours to days depending on size). Monitor progress regularly.
</Warning>

### Monitoring Copy Progress

Check copy operation status:

```bash theme={null}
graphman copy stats <deployment-id>
```

This shows:

* Copy phase (copying data, building indexes, counting entities)
* Progress percentage
* Estimated completion time

### Listing Copy Operations

```bash theme={null}
graphman copy list
```

Shows all active and pending copy operations.

<Note>
  The number of active copy operations is limited to 5 per source/destination shard pair to limit load on the shards.
</Note>

### Copy Process Details

During copying:

1. **Namespace Creation**: Creates temporary `sgdNNN` namespace in destination matching source's identifier
2. **Table Mapping**: Maps all tables from source into destination shard via foreign data wrapper
3. **Data Transfer**: Copies data in batches
4. **Index Building**: Rebuilds indexes on destination (can be slow)
5. **Entity Counting**: Counts all entities to update `entity_count` (can be very slow with minimal output)
6. **Cleanup**: Automatically deletes temporary namespace when complete

### Deleting Inactive Copies

Make non-active copies eligible for deletion:

1. Unassign the copy from all indexing nodes
2. The unused deployment reaper will delete it automatically (\~8 hours by default)

## Namespaces

Sharding creates Postgres schemas (namespaces) for inter-shard data access:

* **`primary_public`**: Maps important tables from primary into each shard
* **`shard_<name>_subgraphs`**: Maps important tables from each shard into every other shard

These enable queries and operations across shards.

### Rebuilding Mappings

If foreign data wrapper mappings become corrupted or out of sync:

```bash theme={null}
graphman database remap
```

This recreates all foreign server definitions and table mappings.

<Note>
  The mapping setup code is in `ForeignServer::map_primary` and `ForeignServer::map_metadata` in the [connection\_pool.rs](https://github.com/graphprotocol/graph-node/blob/master/store/postgres/src/connection_pool.rs) file.
</Note>

## Best Practices

<AccordionGroup>
  <Accordion title="Start with Single Database">
    Begin with a single database. Add sharding only when you hit resource limits. The original database becomes the primary shard, and existing data can remain there.
  </Accordion>

  <Accordion title="Plan Network Topology">
    Ensure all shards can communicate before configuring sharding. Test network connectivity and authentication between all shard pairs.
  </Accordion>

  <Accordion title="Use Deployment Rules">
    Set up deployment rules to automatically route new subgraphs to appropriate shards. This is simpler than manually moving subgraphs later.
  </Accordion>

  <Accordion title="Monitor Copy Operations">
    When copying subgraphs between shards, actively monitor progress. Copy operations can fail silently or take much longer than expected.
  </Accordion>

  <Accordion title="Separate Block Caches">
    Consider storing blockchain block caches in dedicated shards separate from subgraph data. This isolates block ingestion load.
  </Accordion>

  <Accordion title="Balance Shard Load">
    Use the `shards` array (instead of single `shard`) in deployment rules to automatically distribute subgraphs to the least-loaded shard.
  </Accordion>

  <Accordion title="Keep Primary Small">
    The primary shard should primarily store metadata. Route most subgraph data to other shards.
  </Accordion>

  <Accordion title="Test Before Production">
    Test sharding setup in a non-production environment first. Verify inter-shard connectivity and deployment placement.
  </Accordion>
</AccordionGroup>

## Removing a Shard

When a shard is no longer needed:

1. **Ensure no references**: Verify no deployment is stored in the shard and no chain is stored in it
2. **Remove from config**: Delete the shard declaration from the configuration file
3. **Restart nodes**: Restart all Graph Node instances with the updated configuration

<Note>
  Removing a shard leaves behind:

  * Foreign tables in `shard_<name>_subgraphs` on other shards
  * User mapping and foreign server definitions

  These don't hamper operation but can be manually removed via `DROP` commands in `psql` if desired.
</Note>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Copy Operation Stuck">
    If a copy operation appears stuck:

    ```bash theme={null}
    # Check copy status
    graphman copy stats <deployment-id>

    # Check graph-node logs for errors
    tail -f /var/log/graph-node.log | grep -i copy

    # Check database activity
    # Connect to destination shard and check active queries
    SELECT pid, state, query FROM pg_stat_activity WHERE query LIKE '%sgd%';
    ```

    Long pauses during entity counting or index building are normal.
  </Accordion>

  <Accordion title="Foreign Data Wrapper Errors">
    If you see FDW-related errors:

    ```bash theme={null}
    # Rebuild all foreign data wrapper mappings
    graphman database remap
    ```

    Check network connectivity:

    ```bash theme={null}
    # From primary shard
    psql "postgresql://graph:password@shard1-host:5432/graph" -c "SELECT 1"
    ```
  </Accordion>

  <Accordion title="Shard Connection Issues">
    Verify `pg_hba.conf` allows connections:

    ```
    # Add to pg_hba.conf on each shard
    host    graph    graph    <other-shard-ip>/32    md5
    ```

    Check firewall rules:

    ```bash theme={null}
    # Test TCP connection from one shard to another
    nc -zv <shard-host> 5432
    ```
  </Accordion>

  <Accordion title="Deployment Placed in Wrong Shard">
    Test deployment placement:

    ```bash theme={null}
    graphman --config config.toml config place <subgraph-name> <network>
    ```

    Review your deployment rules - rules are evaluated in order, first match wins.
  </Accordion>
</AccordionGroup>

## Complete Example

<Accordion title="Production Sharding Configuration">
  ```toml theme={null}
  # Store configuration
  [store]
  # Primary: metadata only
  [store.primary]
  connection = "postgresql://graph:${PGPASSWORD}@primary-db.internal:5432/graph"
  pool_size = 10

  # VIP shard: high-traffic production subgraphs
  [store.vip]
  connection = "postgresql://graph:${PGPASSWORD}@vip-db.internal:5432/graph"
  pool_size = 50

  [store.vip.replicas.repl1]
  connection = "postgresql://graph:${PGPASSWORD}@vip-repl1.internal:5432/graph"
  weight = 1

  # Community shard 1: many low-traffic subgraphs
  [store.community1]
  connection = "postgresql://graph:${PGPASSWORD}@community1-db.internal:5432/graph"
  pool_size = 25

  # Community shard 2: many low-traffic subgraphs
  [store.community2]
  connection = "postgresql://graph:${PGPASSWORD}@community2-db.internal:5432/graph"
  pool_size = 25

  # Blocks shard: blockchain block caches
  [store.blocks]
  connection = "postgresql://graph:${PGPASSWORD}@blocks-db.internal:5432/graph"
  pool_size = 20

  # Chain configuration
  [chains]
  ingestor = "block_ingestor_node"

  [chains.mainnet]
  shard = "blocks"  # Block cache in dedicated shard
  amp = "ethereum-mainnet"
  provider = [
    { label = "mainnet-archive", url = "http://eth-archive:8545", features = ["archive", "traces"] },
    { label = "mainnet-regular", url = "http://eth-regular:8545", features = [] }
  ]

  [chains.polygon]
  shard = "blocks"
  amp = "polygon-mainnet"
  provider = [
    { label = "polygon", url = "http://polygon:8545", features = [] }
  ]

  [chains.arbitrum]
  shard = "blocks"
  amp = "arbitrum-one"
  provider = [
    { label = "arbitrum", url = "http://arbitrum:8545", features = [] }
  ]

  # Deployment rules
  [deployment]

  # High-traffic VIP subgraphs -> VIP shard
  [[deployment.rule]]
  match = { name = "(acme-corp|vip|production)/.*" }
  shard = "vip"
  indexers = [ "index_node_vip_0", "index_node_vip_1" ]

  # All other subgraphs -> community shards (auto-distribute)
  [[deployment.rule]]
  shards = [ "community1", "community2" ]  # System picks least-loaded
  indexers = [
    "index_node_community_0",
    "index_node_community_1",
    "index_node_community_2",
    "index_node_community_3"
  ]
  ```

  **Migration Steps:**

  ```bash theme={null}
  # 1. Start with configuration above
  # 2. Restart all graph-node instances
  # 3. New subgraphs automatically route to correct shards

  # 4. Optionally move existing high-traffic subgraph to VIP shard
  graphman copy create --replace <deployment-ipfs-hash> vip

  # 5. Monitor copy progress
  graphman copy stats <deployment-ipfs-hash>

  # 6. Verify deployment location
  graphman info <deployment-ipfs-hash>
  ```
</Accordion>