> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/graphprotocol/graph-node/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring and Metrics

> Monitor Graph Node health and performance with Prometheus metrics

Graph Node exposes comprehensive metrics through a Prometheus endpoint, allowing you to monitor deployment health, indexing performance, query execution, and infrastructure utilization.

## Metrics Endpoint

By default, Graph Node exposes metrics on port **8040**:

```
http://localhost:8040/metrics
```

You can configure this port in your Graph Node configuration file.

## Deployment Metrics

These metrics track the health and performance of individual subgraph deployments.

### Block Processing

<AccordionGroup>
  <Accordion title="deployment_block_processing_duration">
    **Type:** Histogram

    Measures the duration of block processing for a subgraph deployment.

    **Labels:**

    * `deployment` - The deployment hash (IPFS CID)
    * `network` - The blockchain network
    * `shard` - The database shard

    **Use cases:**

    * Identify slow-processing deployments
    * Detect performance degradation over time
    * Compare processing speed across deployments

    **Example query:**

    ```promql theme={null}
    rate(deployment_block_processing_duration_sum[5m]) / rate(deployment_block_processing_duration_count[5m])
    ```
  </Accordion>

  <Accordion title="deployment_trigger_processing_duration">
    **Type:** Histogram

    Measures the duration of trigger processing (event handlers, call handlers, block handlers) for a deployment.

    **Labels:**

    * `deployment`
    * `network`
    * `shard`

    **Use cases:**

    * Identify bottlenecks in handler execution
    * Monitor handler performance after code changes
    * Optimize trigger processing logic
  </Accordion>

  <Accordion title="deployment_transact_block_operations_duration">
    **Type:** Histogram

    Measures the duration of committing entity operations for a block and updating the subgraph pointer.

    **Labels:**

    * `deployment`
    * `network`
    * `shard`

    **Use cases:**

    * Monitor database write performance
    * Identify deployments with high entity churn
    * Detect database bottlenecks
  </Accordion>
</AccordionGroup>

### Trigger and Handler Metrics

<AccordionGroup>
  <Accordion title="deployment_block_trigger_count">
    **Type:** Counter

    Counts the number of triggers in each block for a subgraph deployment.

    **Labels:**

    * `deployment`
    * `network`
    * `shard`

    **Use cases:**

    * Understand trigger density per block
    * Correlate trigger count with processing time
    * Identify blocks with unusual trigger activity
  </Accordion>

  <Accordion title="deployment_handler_execution_time">
    **Type:** Histogram

    Measures the execution time for individual handlers.

    **Labels:**

    * `deployment`
    * `handler` - The handler name
    * `network`
    * `shard`

    **Use cases:**

    * Identify slow handlers
    * Optimize specific handler logic
    * Track performance impact of handler changes
  </Accordion>

  <Accordion title="deployment_host_fn_execution_time">
    **Type:** Histogram

    Measures the execution time for host functions called by the WASM runtime.

    **Labels:**

    * `deployment`
    * `host_fn` - The host function name
    * `network`
    * `shard`

    **Use cases:**

    * Identify expensive host function calls
    * Monitor store operations from handlers
    * Detect excessive entity loads or stores
  </Accordion>
</AccordionGroup>

### Deployment Health

<AccordionGroup>
  <Accordion title="deployment_head">
    **Type:** Gauge

    Tracks the head block number for a deployment.

    **Example:**

    ```protobuf theme={null}
    deployment_head{deployment="QmaeWFYbPwmXEk7UuACmkqgPq2Pba5t2RYdJtEyvAUmrxg",network="mumbai",shard="primary"} 19509077
    ```

    **Labels:**

    * `deployment`
    * `network`
    * `shard`

    **Use cases:**

    * Monitor sync progress
    * Calculate blocks behind chain head
    * Alert on stalled indexing

    **Alert example:**

    ```promql theme={null}
    (ethereum_chain_head_number{network="mainnet"} - deployment_head{network="mainnet"}) > 100
    ```
  </Accordion>

  <Accordion title="deployment_failed">
    **Type:** Gauge (Boolean)

    Indicates whether a deployment has failed (1 = failed, 0 = healthy).

    **Labels:**

    * `deployment`
    * `network`
    * `shard`

    **Use cases:**

    * Alert on deployment failures
    * Track deployment health over time
    * Trigger automatic remediation

    **Alert example:**

    ```promql theme={null}
    deployment_failed{deployment="QmXYZ..."} == 1
    ```
  </Accordion>

  <Accordion title="deployment_reverted_blocks">
    **Type:** Counter

    Tracks the last reverted block for a subgraph deployment.

    **Labels:**

    * `deployment`
    * `network`
    * `shard`

    **Use cases:**

    * Monitor chain reorganizations
    * Identify frequently reorged deployments
    * Correlate reverts with network issues
  </Accordion>

  <Accordion title="deployment_sync_secs">
    **Type:** Counter

    Total time spent syncing a deployment.

    **Labels:**

    * `deployment`
    * `network`
    * `shard`

    **Use cases:**

    * Calculate average sync time
    * Monitor sync efficiency
    * Estimate time to full sync
  </Accordion>

  <Accordion title="deployment_count">
    **Type:** Gauge

    Counts the number of deployments currently being indexed by the graph-node.

    **Use cases:**

    * Monitor node capacity
    * Track deployment growth
    * Plan infrastructure scaling
  </Accordion>
</AccordionGroup>

## Blockchain Metrics

### Ethereum Chain Head

<Accordion title="ethereum_chain_head_number">
  **Type:** Gauge

  Block number of the most recent block synced from Ethereum.

  **Example:**

  ```protobuf theme={null}
  ethereum_chain_head_number{network="mumbai"} 20045294
  ```

  **Labels:**

  * `network` - The blockchain network name

  **Use cases:**

  * Verify RPC connectivity
  * Calculate deployment lag
  * Monitor chain sync status

  **Example query (blocks behind):**

  ```promql theme={null}
  ethereum_chain_head_number{network="mainnet"} - deployment_head{network="mainnet"}
  ```
</Accordion>

### RPC Performance

<AccordionGroup>
  <Accordion title="deployment_eth_rpc_request_duration">
    **Type:** Histogram

    Measures Ethereum RPC request duration for a subgraph deployment.

    **Labels:**

    * `deployment`
    * `network`
    * `method` - The RPC method called

    **Use cases:**

    * Monitor RPC provider performance
    * Identify slow RPC methods
    * Detect RPC provider issues
  </Accordion>

  <Accordion title="deployment_eth_rpc_errors">
    **Type:** Counter

    Counts Ethereum RPC request errors for a subgraph deployment.

    **Labels:**

    * `deployment`
    * `network`
    * `method`

    **Use cases:**

    * Alert on RPC failures
    * Monitor RPC provider reliability
    * Trigger provider failover
  </Accordion>

  <Accordion title="eth_rpc_request_duration">
    **Type:** Histogram

    Global Ethereum RPC request duration across all deployments.

    **Labels:**

    * `network`
    * `method`

    **Use cases:**

    * Monitor overall RPC health
    * Compare RPC performance across networks
    * Benchmark RPC providers
  </Accordion>

  <Accordion title="eth_rpc_errors">
    **Type:** Counter

    Global count of Ethereum RPC request errors.

    **Labels:**

    * `network`
    * `method`

    **Use cases:**

    * Track RPC reliability
    * Alert on widespread RPC issues
    * Monitor error rates by method
  </Accordion>
</AccordionGroup>

## Query Metrics

These metrics help monitor GraphQL query performance and caching efficiency.

### Query Execution

<AccordionGroup>
  <Accordion title="query_execution_time">
    **Type:** Histogram

    Execution time for successful GraphQL queries.

    **Labels:**

    * `deployment`
    * `query_id` - Hash of the query

    **Use cases:**

    * Identify slow queries
    * Monitor query performance trends
    * Optimize query execution
  </Accordion>

  <Accordion title="query_effort_ms">
    **Type:** Gauge

    Moving average of time spent running queries.

    **Use cases:**

    * Monitor overall query load
    * Detect query performance degradation
    * Trigger scaling decisions
  </Accordion>

  <Accordion title="query_blocks_behind">
    **Type:** Histogram

    Tracks how many blocks behind the subgraph head queries are being made.

    **Use cases:**

    * Inform pruning decisions
    * Understand query patterns
    * Optimize history retention
  </Accordion>
</AccordionGroup>

### Query Results

<AccordionGroup>
  <Accordion title="query_result_size">
    **Type:** Histogram

    The size of successful GraphQL query results (in CacheWeight).

    **Labels:**

    * `deployment`

    **Use cases:**

    * Monitor result size distribution
    * Identify queries returning large datasets
    * Optimize pagination strategies
  </Accordion>

  <Accordion title="query_result_max">
    **Type:** Gauge

    The maximum size of a query result (in CacheWeight).

    **Labels:**

    * `deployment`

    **Use cases:**

    * Track largest queries
    * Set result size limits
    * Prevent resource exhaustion
  </Accordion>
</AccordionGroup>

### Caching

<AccordionGroup>
  <Accordion title="query_cache_status_count">
    **Type:** Counter

    Counts top-level GraphQL fields executed and their cache status.

    **Labels:**

    * `deployment`
    * `field` - The GraphQL field name
    * `status` - `hit` or `miss`

    **Use cases:**

    * Monitor cache hit rates
    * Identify frequently queried fields
    * Optimize caching strategy

    **Cache hit rate:**

    ```promql theme={null}
    rate(query_cache_status_count{status="hit"}[5m]) / rate(query_cache_status_count[5m])
    ```
  </Accordion>
</AccordionGroup>

### Load Management

<AccordionGroup>
  <Accordion title="query_kill_rate">
    **Type:** Gauge

    The rate at which the load manager kills queries.

    **Use cases:**

    * Monitor query overload
    * Adjust query timeout settings
    * Alert on excessive query cancellations
  </Accordion>

  <Accordion title="query_semaphore_wait_ms">
    **Type:** Gauge

    Moving average of time spent waiting for the Postgres query semaphore.

    **Use cases:**

    * Monitor database connection contention
    * Adjust connection pool size
    * Identify query queueing issues
  </Accordion>
</AccordionGroup>

## Store Metrics

Metrics related to PostgreSQL database operations.

<AccordionGroup>
  <Accordion title="store_connection_checkout_count">
    **Type:** Gauge

    The number of Postgres connections currently checked out.

    **Labels:**

    * `pool` - The connection pool name
    * `shard` - The database shard

    **Use cases:**

    * Monitor connection pool utilization
    * Detect connection leaks
    * Size connection pools appropriately
  </Accordion>

  <Accordion title="store_connection_wait_time_ms">
    **Type:** Histogram

    Average connection wait time from the pool.

    **Labels:**

    * `pool`
    * `shard`

    **Use cases:**

    * Identify connection pool bottlenecks
    * Optimize pool configuration
    * Alert on connection exhaustion
  </Accordion>

  <Accordion title="store_connection_error_count">
    **Type:** Counter

    The number of Postgres connection errors.

    **Labels:**

    * `pool`
    * `shard`

    **Use cases:**

    * Monitor database connectivity
    * Alert on connection failures
    * Detect database issues
  </Accordion>
</AccordionGroup>

## System Metrics

<AccordionGroup>
  <Accordion title="registered_metrics">
    **Type:** Gauge

    Tracks the number of registered metrics on the node.

    **Use cases:**

    * Monitor metric system health
    * Detect metric registration leaks
    * Track metric growth over time
  </Accordion>

  <Accordion title="metrics_register_errors">
    **Type:** Counter

    Counts Prometheus metrics register errors.

    **Use cases:**

    * Detect metric registration issues
    * Alert on metrics system problems
  </Accordion>

  <Accordion title="metrics_unregister_errors">
    **Type:** Counter

    Counts Prometheus metrics unregister errors.

    **Use cases:**

    * Monitor metric cleanup issues
    * Detect metric lifecycle problems
  </Accordion>
</AccordionGroup>

## Monitoring Setup

### Prometheus Configuration

Add Graph Node to your `prometheus.yml`:

```yaml theme={null}
scrape_configs:
  - job_name: 'graph-node'
    static_configs:
      - targets: ['localhost:8040']
    scrape_interval: 15s
    scrape_timeout: 10s
```

### Grafana Dashboard Setup

<Steps>
  <Step title="Add Prometheus data source">
    Configure Grafana to connect to your Prometheus instance.
  </Step>

  <Step title="Create deployment health panel">
    ```promql theme={null}
    # Blocks behind chain head
    ethereum_chain_head_number{network="mainnet"} - deployment_head{network="mainnet"}
    ```
  </Step>

  <Step title="Create indexing speed panel">
    ```promql theme={null}
    # Blocks per minute
    rate(deployment_head[5m]) * 60
    ```
  </Step>

  <Step title="Create query performance panel">
    ```promql theme={null}
    # Average query time (ms)
    rate(query_execution_time_sum[5m]) / rate(query_execution_time_count[5m]) * 1000
    ```
  </Step>
</Steps>

## Alert Examples

### Deployment Health Alerts

<AccordionGroup>
  <Accordion title="Deployment is falling behind">
    ```yaml theme={null}
    - alert: DeploymentBehind
      expr: (ethereum_chain_head_number - deployment_head) > 100
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Deployment {{ $labels.deployment }} is falling behind"
        description: "Deployment is {{ $value }} blocks behind chain head"
    ```
  </Accordion>

  <Accordion title="Deployment has failed">
    ```yaml theme={null}
    - alert: DeploymentFailed
      expr: deployment_failed == 1
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "Deployment {{ $labels.deployment }} has failed"
        description: "Deployment is in failed state on {{ $labels.shard }}"
    ```
  </Accordion>

  <Accordion title="High RPC error rate">
    ```yaml theme={null}
    - alert: HighRPCErrorRate
      expr: rate(deployment_eth_rpc_errors[5m]) > 0.1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High RPC error rate for {{ $labels.deployment }}"
        description: "RPC error rate is {{ $value }} errors/sec"
    ```
  </Accordion>
</AccordionGroup>

### Query Performance Alerts

<AccordionGroup>
  <Accordion title="Slow query performance">
    ```yaml theme={null}
    - alert: SlowQueries
      expr: (rate(query_execution_time_sum[5m]) / rate(query_execution_time_count[5m])) > 1
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Slow queries on {{ $labels.deployment }}"
        description: "Average query time is {{ $value }}s"
    ```
  </Accordion>

  <Accordion title="High query kill rate">
    ```yaml theme={null}
    - alert: HighQueryKillRate
      expr: query_kill_rate > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High query kill rate"
        description: "{{ $value }} queries/sec are being killed"
    ```
  </Accordion>
</AccordionGroup>

### Database Alerts

<AccordionGroup>
  <Accordion title="Connection pool exhaustion">
    ```yaml theme={null}
    - alert: ConnectionPoolExhaustion
      expr: store_connection_checkout_count / 100 > 0.9
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Connection pool nearly exhausted for {{ $labels.pool }}"
        description: "Pool utilization is {{ $value | humanizePercentage }}"
    ```
  </Accordion>

  <Accordion title="High connection wait time">
    ```yaml theme={null}
    - alert: HighConnectionWait
      expr: store_connection_wait_time_ms > 100
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High connection wait time for {{ $labels.pool }}"
        description: "Average wait time is {{ $value }}ms"
    ```
  </Accordion>
</AccordionGroup>

## Best Practices

<AccordionGroup>
  <Accordion title="Monitor deployment lag consistently">
    Always track `ethereum_chain_head_number - deployment_head` to ensure deployments stay synchronized with the blockchain.
  </Accordion>

  <Accordion title="Set up alerts for critical metrics">
    Configure alerts for deployment failures, high error rates, and significant lag to catch issues early.
  </Accordion>

  <Accordion title="Use recording rules for complex queries">
    Pre-compute expensive queries like cache hit rates and average query times to reduce dashboard load.
  </Accordion>

  <Accordion title="Track query patterns">
    Monitor `query_cache_status_count` and `query_blocks_behind` to optimize caching and pruning strategies.
  </Accordion>

  <Accordion title="Correlate metrics across layers">
    When investigating issues, look at RPC metrics, handler metrics, and store metrics together to identify the root cause.
  </Accordion>

  <Accordion title="Monitor resource utilization">
    Track connection pool usage and query semaphore wait times to prevent resource exhaustion.
  </Accordion>

  <Accordion title="Establish baselines">
    Record normal metric ranges for your deployments to quickly identify anomalies.
  </Accordion>
</AccordionGroup>

## Troubleshooting with Metrics

### Slow Indexing

<Steps>
  <Step title="Check block processing duration">
    ```promql theme={null}
    rate(deployment_block_processing_duration_sum[5m]) / rate(deployment_block_processing_duration_count[5m])
    ```
  </Step>

  <Step title="Identify slow handlers">
    ```promql theme={null}
    topk(5, deployment_handler_execution_time)
    ```
  </Step>

  <Step title="Check RPC performance">
    ```promql theme={null}
    rate(deployment_eth_rpc_request_duration_sum[5m]) / rate(deployment_eth_rpc_request_duration_count[5m])
    ```
  </Step>

  <Step title="Monitor database operations">
    ```promql theme={null}
    rate(deployment_transact_block_operations_duration_sum[5m]) / rate(deployment_transact_block_operations_duration_count[5m])
    ```
  </Step>
</Steps>

### Poor Query Performance

<Steps>
  <Step title="Check cache hit rate">
    ```promql theme={null}
    rate(query_cache_status_count{status="hit"}[5m]) / rate(query_cache_status_count[5m])
    ```
  </Step>

  <Step title="Identify large queries">
    ```promql theme={null}
    query_result_max
    ```
  </Step>

  <Step title="Check connection contention">
    ```promql theme={null}
    query_semaphore_wait_ms
    ```
  </Step>

  <Step title="Monitor query load">
    ```promql theme={null}
    query_effort_ms
    ```
  </Step>
</Steps>

## Additional Resources

* [Graphman CLI](/operations/graphman) - Management commands
* [Maintenance Operations](/operations/maintenance) - Deployment management
* [Pruning Guide](/operations/pruning) - Optimize storage
* [Configuration](/configuration/config-file) - Graph Node configuration