> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/graphprotocol/graph-node/llms.txt
> Use this file to discover all available pages before exploring further.

# Offchain Data Sources

> Implementation details for supporting offchain data sources in Graph Node subgraphs

## Overview

Graph Node supports syncing offchain data sources in subgraphs, such as IPFS files. The implementation provides reusable components and data structures that simplify adding new kinds of offchain data sources.

<Note>
  For subgraph developer documentation on using offchain data sources, refer to the official subgraph documentation. This page focuses on implementation details for Graph Node developers.
</Note>

## Implementation Architecture

### Core Components

The offchain data source implementation consists of several reusable components designed to make adding new data source kinds straightforward.

<Accordion title="Component Locations">
  * **Data Structures**: `graph` crate, `data_source/offchain.rs`
  * **Monitoring Logic**: `OffchainMonitor` in `subgraph/context.rs`
  * **Polling Helper**: `PollingMonitor` generic component
  * **IPFS Implementation**: `IpfsService` (reference implementation)
</Accordion>

### Data Source Representation

Offchain data sources are represented by data structures in the `graph` crate at `data_source/offchain.rs`. These structures handle:

* Parsing from the subgraph manifest
* Creation as dynamic data sources
* Source type enumeration via `enum Source`
* Kind registration in `const OFFCHAIN_KINDS`

<Note>
  Adding a new file-based data source kind typically only requires:

  1. A new `enum Source` variant
  2. Adding the kind to `const OFFCHAIN_KINDS`
</Note>

## OffchainMonitor

The `OffchainMonitor` is responsible for tracking and fetching offchain data. Currently located in `subgraph/context.rs`.

### Key Operations

<Accordion title="fn add_source">
  Called when an offchain data source is created from a template. This function registers the source for monitoring.

  ```rust theme={null}
  fn add_source(/* parameters */)
  ```

  Expectation: A background task will monitor the source for relevant events (e.g., file becoming available).
</Accordion>

<Accordion title="fn ready_offchain_events">
  Called periodically by the subgraph runner to process events from monitored sources.

  ```rust theme={null}
  fn ready_offchain_events(/* parameters */)
  ```

  For file data sources, the event is the file content becoming available.
</Accordion>

## Adding New Data Source Kinds

### File-Based Data Sources

For file-based data sources, most existing code can be reused:

<CodeGroup>
  ```rust Data Structure theme={null}
  // In graph/data_source/offchain.rs
  enum Source {
      Ipfs(IpfsSource),
      YourNewKind(YourSource), // Add new variant
  }

  const OFFCHAIN_KINDS: &[&str] = &[
      "file/ipfs",
      "file/yournewkind", // Add new kind
  ];
  ```

  ```rust Polling Service theme={null}
  // Implement as a tower service
  impl Service<Request> for YourPollingService {
      type Response = FileContent;
      type Error = Error;
      type Future = /* ... */;

      fn poll_ready(&mut self, cx: &mut Context) -> Poll<Result<(), Self::Error>> {
          // Polling logic
      }

      fn call(&mut self, req: Request) -> Self::Future {
          // Fetch logic
      }
  }
  ```
</CodeGroup>

### Using PollingMonitor

For data sources that rely on polling to check availability, use the generic `PollingMonitor` component:

<Accordion title="PollingMonitor Implementation">
  1. Implement the polling logic as a `tower` service
  2. The `IpfsService` serves as a reference implementation
  3. Focus only on the polling and fetching logic
  4. The `PollingMonitor` handles the monitoring infrastructure
</Accordion>

## Testing

### Integration Testing

Automated testing for offchain data sources can be tricky and should be discussed case-by-case.

<Note>
  The `file_data_sources` test in `runner_tests.rs` serves as a starting point for writing integration tests with offchain data sources.
</Note>

<CodeGroup>
  ```rust Test Example theme={null}
  // In runner_tests.rs
  #[tokio::test]
  async fn file_data_sources() {
      // Test setup
      // 1. Create subgraph with file data source template
      // 2. Upload file to IPFS/storage
      // 3. Trigger data source creation
      // 4. Verify data was processed correctly
  }
  ```

  ```yaml Test Considerations theme={null}
  Requirements:
    - Mock or real offchain storage (IPFS, etc.)
    - File upload capability
    - Event trigger mechanism
    - State verification

  Challenges:
    - Async behavior
    - Timing dependencies
    - Storage availability
    - Cleanup between tests
  ```
</CodeGroup>

## Current Limitations

<Accordion title="Dynamic Only">
  Offchain data sources currently can only exist as dynamic data sources, instantiated from templates. They cannot be configured as static data sources in the manifest.

  **Impact**: All offchain data sources must be created at runtime from a template.
</Accordion>

<Accordion title="One-Shot Assumption">
  Some parts of the implementation assume offchain data sources are 'one shot' - only a single trigger is handled per data source instance.

  **Works Well For**: Files (file is found, handled, done)

  **Consideration**: More complex offchain data sources (e.g., continuous streams) will require additional planning and architectural changes.
</Accordion>

<Accordion title="Proof of Indexing (PoI)">
  Entities from offchain data sources do not currently influence the PoI. Causality region IDs are not deterministic.

  **Impact**:

  * Offchain data cannot be verified through PoI
  * May affect dispute resolution
  * Limits trustless verification guarantees
</Accordion>

## Reference Implementation: IPFS

The initially supported data source kind is `file/ipfs`, which serves as a reference implementation.

<CodeGroup>
  ```rust IPFS Service theme={null}
  // Example structure (simplified)
  struct IpfsService {
      client: IpfsClient,
      // ... other fields
  }

  impl Service<IpfsRequest> for IpfsService {
      type Response = FileContent;
      // Implementation details
  }
  ```

  ```yaml IPFS Flow theme={null}
  Sequence:
    1. Template instantiation:
       - Subgraph creates data source from template
       - Specifies IPFS hash to monitor
    
    2. Monitoring:
       - OffchainMonitor tracks the hash
       - IpfsService polls for availability
    
    3. Processing:
       - File becomes available
       - Event emitted with content
       - Subgraph handler processes data
    
    4. Completion:
       - One-shot: Data source done
       - No further events expected
  ```
</CodeGroup>

## Best Practices

### For New Data Source Implementations

1. **Reuse existing components**: Start with `PollingMonitor` for polling-based sources
2. **Study IPFS implementation**: Use `IpfsService` as a template
3. **Consider timing**: Plan for async availability and delays
4. **Test thoroughly**: Write integration tests early in development
5. **Document limitations**: Be clear about one-shot vs. continuous behavior

### Architecture Considerations

<Note>
  When adding support for non-file data sources (e.g., APIs, message queues), consider:

  * Event multiplicity (multiple triggers vs. one-shot)
  * Determinism requirements for PoI
  * Resource management and cleanup
  * Error handling and retry logic
</Note>