Maintenance Operations

Common maintenance tasks for managing Graph Node deployments

This guide covers essential maintenance operations for keeping your Graph Node infrastructure running efficiently using graphman.

Prerequisites

The graphman command is included in official Graph Node containers. You can access it via:

docker exec -it <graph-node-container> graphman --config config.toml <command>

Ensure you have a configuration file set up. See the basic setup guide if you need to create one.

Logging Configuration

Graphman respects the GRAPH_LOG environment variable. To reduce log noise:

unset GRAPH_LOG

Verify Setup

Confirm graphman is configured correctly:

graphman --config config.toml info some/subgraph

This should display basic deployment information, including the Postgres namespace.

Removing Unused Deployments

When you deploy a new version of a subgraph, the previous version’s data remains in the database even after it’s no longer served for queries. These unused deployments consume storage and should be periodically cleaned up.

Understanding Unused Deployments

A deployment is considered unused when it meets all of these criteria:

1. Not assigned to any node
2. Either not marked as active, or is neither the current nor pending version of a subgraph
3. Not the source of a currently running copy operation

Cleanup Process

1. Record unused deployments

   Scan all shards and identify unused deployments:

   graphman --config config.toml unused record
   

   This compiles a list and stores it in the unused_deployments table in the primary shard.

   No data is deleted at this stage. This command only identifies and marks deployments for removal.

2. Review the list

   Inspect which deployments are marked for removal:

   graphman --config config.toml unused list -e
   

   Verify that the listed deployments should indeed be removed.

3. Remove unused deployments

   Delete the data for marked deployments:

   graphman --config config.toml unused remove
   

   This operation is irreversible. All indexed data for these deployments will be permanently deleted.

   Deployments are removed in descending order by entity count (smaller deployments first).

Removal Options

The unused remove command supports several options for fine-grained control:

Remove all unused deployments

graphman --config config.toml unused remove

Removes all deployments previously marked with unused record.

Remove deployments older than a specific time

graphman --config config.toml unused remove --older 720

Only removes deployments recorded at least 720 minutes (12 hours) ago. This provides a safety buffer.

Remove a specific deployment

graphman --config config.toml unused remove --deployment QmfWRZCjT8pri4Amey3e3mb2Bga75Vuh2fPYyNVnmPYL66

Targets a single deployment by its IPFS hash.

Limit the number of removals

graphman --config config.toml unused remove --count 5

Removes only the specified number of deployments, useful for gradual cleanup.

Removing a Subgraph

To stop serving a specific subgraph and free its name for reuse:

1. Remove the subgraph name mapping

   graphman --config config.toml remove author/subgraph-name
   

   This removes the association between the subgraph name and its deployment.

2. Mark the deployment as unused

   If no other subgraph name uses this deployment:

   graphman --config config.toml unused record
   
3. Remove the deployment data

   After confirming it’s safe to delete:

   graphman --config config.toml unused remove
   

Removing a subgraph name does not immediately delete indexed data. The deployment becomes eligible for cleanup through the unused deployment process.

Modifying Assignments

Each deployment is assigned to a specific Graph Node instance for indexing. You can modify these assignments to control which node indexes which subgraphs.

Reassigning Deployments

Move a deployment to a different node:

graphman --config config.toml reassign <deployment> <new-node-id>

Arguments:

• <deployment> - The deployment identifier (name, IPFS hash, or namespace)
• <new-node-id> - The target node ID from your Graph Node configuration

This is useful for:

• Load balancing across multiple nodes
• Moving deployments off a node before maintenance
• Isolating problematic subgraphs

Unassigning Deployments

Permanently stop indexing a deployment:

graphman --config config.toml unassign <deployment>

The deployment remains in the database but stops receiving updates.

To resume indexing, reassign the deployment to an active node. No data is lost during unassignment.

Pausing Indexing

To temporarily pause a deployment without losing its assignment:

graphman --config config.toml reassign <deployment> paused_<original-node-id>

To resume:

graphman --config config.toml reassign <deployment> <original-node-id>

Currently, graphman does not support creating assignments for completely unassigned deployments. You can only reassign or pause existing assignments.

Complete Deployment Removal

To fully remove a deployment in a single operation:

graphman --config config.toml drop <deployment>

This command combines multiple operations:

1. Retrieve deployment info

   Identifies the deployment and its associated data

2. Unassign from node

   Stops indexing activity

3. Remove subgraph name

   Frees the name for reuse

4. Mark as unused

   Registers the deployment for deletion

5. Delete all data

   Permanently removes indexed data

The drop command is irreversible. Use with caution and consider backing up critical data first.

Drop Command Options

Drop by subgraph name

graphman --config config.toml drop author/subgraph-name

Drop by IPFS hash

graphman --config config.toml drop QmfWRZCjT8pri4Amey3e3mb2Bga75Vuh2fPYyNVnmPYL66

Drop only current versions

graphman --config config.toml drop <deployment> --current

Drop only pending versions

graphman --config config.toml drop <deployment> --pending

Skip confirmation prompt

graphman --config config.toml drop <deployment> --force

Use in automation scripts only.

Maintenance Workflows

Regular Cleanup Schedule

Establish a routine maintenance schedule:

1. Weekly: Identify unused deployments

   graphman --config config.toml unused record
   
2. Weekly: Review and remove

   Wait 24-48 hours, then:

   graphman --config config.toml unused remove --older 1440
   

   This ensures deployments have been truly unused for at least a day.

3. Monthly: Audit active deployments

   Review all deployments to identify candidates for archival:

   graphman --config config.toml info <deployment> --status
   

Pre-Deployment Cleanup

Before deploying a new version:

1. Check current deployment info

   graphman --config config.toml info author/subgraph-name --status
   

   Note the current deployment hash and version.

2. Deploy new version

   Deploy your updated subgraph using the Graph CLI or your deployment tool.

3. Wait for sync completion

   Monitor the new deployment until it’s fully synced.

4. Clean up old version

   graphman --config config.toml unused record
   graphman --config config.toml unused remove --older 60
   

   Wait at least 1 hour to ensure the old version is no longer needed.

Node Maintenance

When performing maintenance on a Graph Node instance:

1. List deployments on the node

   Identify which deployments are assigned to the node.

2. Reassign critical deployments

   graphman --config config.toml reassign <deployment> <backup-node-id>
   

   Move important subgraphs to other nodes.

3. Pause remaining deployments

   graphman --config config.toml reassign <deployment> paused_<node-id>
   
4. Perform maintenance

   Update, restart, or repair the node.

5. Restore assignments

   graphman --config config.toml reassign <deployment> <node-id>
   

   Return deployments to the maintained node.

Best Practices

Verify before deleting

Always use graphman info and unused list to confirm which deployments will be affected before running destructive operations.

Use time buffers

Wait at least 1 hour (preferably 24 hours) between marking deployments as unused and removing them using the --older flag.

Monitor disk usage

Track database size trends to determine optimal cleanup frequency. Unused deployments can consume significant storage.

Document assignments

Maintain a record of which deployments are assigned to which nodes, especially in multi-node setups.

Test in staging

If possible, test maintenance operations in a staging environment before running them in production.

Automate routine tasks

Create scripts for regular maintenance tasks, but always include safety checks and confirmation steps.

Keep deployment history

Before removing old deployments, consider exporting critical data or keeping deployment metadata for audit purposes.

Troubleshooting Common Issues

Deployment won’t unassign

Symptom: Unassign command fails or deployment continues indexing.

Solutions:

• Verify the deployment identifier is correct
• Check that no queries are actively using the deployment
• Ensure the node is not restarting or in an error state

Unused remove doesn’t delete deployment

Symptom: Deployment remains in database after unused remove.

Causes and solutions:

• Not marked as unused: Run graphman unused record first
• Still assigned: Unassign the deployment before marking as unused
• Recent deployment: Use --older 0 to remove immediately (not recommended)
• Active queries: Wait for queries to complete

Can’t reassign deployment

Symptom: Reassign command fails or assignment doesn’t change.

Solutions:

• Verify the target node ID exists in your configuration
• Check that the target node is running and connected to the database
• Ensure the deployment isn’t in a failed state
• Try unassigning first, then creating a new assignment

Database space not freed after removal

Symptom: Disk usage remains high after removing deployments.

Solutions:

• Run PostgreSQL’s VACUUM FULL on affected schemas (requires downtime)
• Consider using VACUUM regularly to mark space as reusable
• Check for other large tables or indexes
• Monitor the primary shard’s unused_deployments table size

Automation Examples

Daily Cleanup Script

#!/bin/bash
set -e

CONFIG="/path/to/config.toml"

# Unset logging for cleaner output
unset GRAPH_LOG

# Record unused deployments
echo "Recording unused deployments..."
graphman --config "$CONFIG" unused record

# Wait 24 hours, then remove deployments unused for at least 48 hours
echo "Removing deployments unused for 48+ hours..."
graphman --config "$CONFIG" unused remove --older 2880

echo "Cleanup complete"

Deployment Migration Script

#!/bin/bash
set -e

OLD_DEPLOYMENT="$1"
NEW_NODE="$2"
CONFIG="/path/to/config.toml"

if [ -z "$OLD_DEPLOYMENT" ] || [ -z "$NEW_NODE" ]; then
  echo "Usage: $0 <deployment> <new-node-id>"
  exit 1
fi

# Get current info
echo "Current deployment info:"
graphman --config "$CONFIG" info "$OLD_DEPLOYMENT" --status

# Reassign
echo "Reassigning to $NEW_NODE..."
graphman --config "$CONFIG" reassign "$OLD_DEPLOYMENT" "$NEW_NODE"

# Verify
echo "New deployment info:"
graphman --config "$CONFIG" info "$OLD_DEPLOYMENT" --status

echo "Migration complete"

Additional Resources

• Graphman CLI Reference - Complete command documentation
• Configuration File Guide - Setup and configuration
• Monitoring Guide - Track deployment health and performance
• Pruning Guide - Optimize storage with data pruning