VictoriaMetrics Plugin

Monitor VictoriaMetrics time-series database with comprehensive metrics covering storage statistics, query performance, ingestion rates, and resource usage.

Overview

The VictoriaMetrics plugin collects detailed metrics from VictoriaMetrics API including:

• Storage Statistics - Active series, total samples, storage size
• Ingestion Metrics - Rows ingested, ingestion rate, data points per second
• Query Performance - Query rate, query duration, slow queries
• Cache Statistics - Index cache, data cache, merge statistics
• Resource Usage - Memory usage, disk usage, CPU time
• Retention Metrics - Data retention period, old data cleanup

Requirements

VictoriaMetrics Version

• Minimum: VictoriaMetrics 1.80
• Recommended: VictoriaMetrics 1.95 or later
• Tested with: VictoriaMetrics 1.85, 1.90, 1.95, 1.96

Python Dependencies

pip install requests>=2.28.0

Auto-installed when using PLUGINS=victoriametrics during agent installation.

VictoriaMetrics Access

The agent needs HTTP access to VictoriaMetrics API (default port 8428).

Configuration

Basic Configuration

plugins:
  victoriametrics:
    enabled: true
    host: localhost
    port: 8428

With Authentication

plugins:
  victoriametrics:
    enabled: true
    host: localhost
    port: 8428
    username: admin
    password: password

With HTTPS

plugins:
  victoriametrics:
    enabled: true
    host: victoria.example.com
    port: 8428
    use_ssl: true
    verify_ssl: true

All Configuration Options

plugins:
  victoriametrics:
    enabled: true                # Enable/disable plugin
    host: localhost              # VictoriaMetrics host
    port: 8428                   # VictoriaMetrics port
    username: ""                 # HTTP auth username (optional)
    password: ""                 # HTTP auth password (optional)
    use_ssl: false               # Use HTTPS
    verify_ssl: true             # Verify SSL certificates
    timeout: 10                  # Request timeout (seconds)

Environment Variables

Configuration can be overridden with environment variables:

export VICTORIAMETRICS_HOST="localhost"
export VICTORIAMETRICS_PORT="8428"

VictoriaMetrics Setup

Installation

Binary installation:

# Download latest release
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.95.1/victoria-metrics-linux-amd64-v1.95.1.tar.gz

# Extract
tar -xzf victoria-metrics-linux-amd64-v1.95.1.tar.gz

# Run VictoriaMetrics
./victoria-metrics-prod

Docker:

docker run -d \
  --name victoriametrics \
  -p 8428:8428 \
  -v /var/lib/victoria-metrics:/victoria-metrics-data \
  victoriametrics/victoria-metrics:latest

As systemd service:

sudo nano /etc/systemd/system/victoriametrics.service

[Unit]
Description=VictoriaMetrics
After=network.target

[Service]
Type=simple
User=victoriametrics
WorkingDirectory=/var/lib/victoriametrics
ExecStart=/usr/local/bin/victoria-metrics-prod \
  -storageDataPath=/var/lib/victoriametrics \
  -httpListenAddr=:8428 \
  -retentionPeriod=12
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start victoriametrics
sudo systemctl enable victoriametrics

Collected Metrics

Ingestion Metrics

Storage Metrics

Resource Metrics

HTTP Metrics

Dashboard Metrics

The StatusRadar dashboard displays:

Overview Card

• Active Series - Current time series count
• Storage Size - Total data size on disk
• Ingestion Rate - Rows per second
• Query Rate - Queries per second

Storage Usage Chart

• Data size over time
• Index size
• Total storage growth

Ingestion Rate Chart

• Rows inserted per second
• Active time series count

Query Performance Chart

• Query rate
• Average query duration
• Slow queries

Cache Performance Chart

• Cache hit rate
• Cache size
• Cache entries

Memory Usage Chart

• Memory usage over time
• Memory growth rate

Installation

Quick Install

PLUGINS='victoriametrics' \
TOKEN='your-agent-token' \
bash -c "$(curl -sL https://statusradar.dev/install-agent.sh)"

Install on Existing Agent

1. Configure agent:

   sudo nano /opt/statusradar/config/agent.yaml

   Add:

   plugins:
     victoriametrics:
       enabled: true
       host: localhost
       port: 8428

2. Restart agent:

   sudo systemctl restart statusradar-agent

3. Verify:

   sudo journalctl -u statusradar-agent -n 50 --no-pager | grep victoriametrics

   Expected:

   INFO: Plugin victoriametrics: Metrics collected successfully
   INFO: Plugin victoriametrics: 12345 active series, 1.2 GB storage

Testing

Manual Plugin Test

cd /opt/statusradar
python3 plugins/victoriametrics_plugin.py

Expected Output:

Plugin: victoriametrics
Enabled: True
Available: True

Collecting metrics...
{
  "vm_rows": 123456789,
  "vm_active_time_series": 12345,
  "vm_data_size_bytes": 1288490188,
  "vm_rows_per_second": 1250.5,
  "vm_requests_total": 45678,
  "vm_cache_hits_total": 89012,
  "vm_cache_misses_total": 3456,
  "vm_memory_bytes": 536870912
}

Test VictoriaMetrics API

# Metrics endpoint
curl http://localhost:8428/metrics

# Query API
curl 'http://localhost:8428/api/v1/query?query=up'

# Label values
curl 'http://localhost:8428/api/v1/label/__name__/values'

Troubleshooting

Plugin Not Collecting Metrics

Check 1: Is VictoriaMetrics running?

sudo systemctl status victoriametrics
# Or check process:
ps aux | grep victoria-metrics

Check 2: Can agent access VictoriaMetrics?

curl http://localhost:8428/metrics

Check 3: Check agent logs

sudo journalctl -u statusradar-agent -n 100 --no-pager | grep victoriametrics

Common Errors

"Connection refused"

Error:

ERROR: Plugin victoriametrics: Connection refused

Causes:

1. VictoriaMetrics not running
2. Wrong host/port
3. Firewall blocking connection

Solution:

# Check VictoriaMetrics is running
ps aux | grep victoria-metrics

# Check port
sudo netstat -tlnp | grep 8428

# Test connection
curl http://localhost:8428/metrics

"Authentication required"

Error:

ERROR: Plugin victoriametrics: 401 Unauthorized

Solution:

Add authentication to config:

plugins:
  victoriametrics:
    username: admin
    password: your-password

Performance Impact

On VictoriaMetrics

Minimal impact:

• Metrics endpoint returns pre-calculated statistics
• No query execution overhead
• Response time: < 50ms

Benchmark:

• Overhead: < 0.1% CPU
• No measurable performance degradation

On Agent

Resource usage:

• Memory: +10 MB
• CPU: +2% during collection
• Network: +2 KB per collection

Collection time: < 0.3 seconds

Use Cases

1. Storage Growth Monitoring

Monitor:

• Data size growth rate
• Index size
• Active time series count

Alert on:

• Rapid storage growth
• Storage approaching disk limit
• Excessive time series cardinality

2. Ingestion Rate Monitoring

Monitor:

• Rows per second
• Ingestion spikes
• Slow inserts

Alert on:

• Ingestion rate drop (data loss)
• Ingestion spike (DDoS/bug)

3. Query Performance

Monitor:

• Query rate
• Query duration
• Slow queries

Optimize:

• Query patterns
• Index usage
• Cache configuration

4. Resource Usage

Monitor:

• Memory usage
• Disk usage
• CPU usage

Alert on:

• Memory > 80%
• Disk > 90%
• High CPU sustained

Best Practices

1. Set Retention Period

# Retain data for 12 months
./victoria-metrics-prod -retentionPeriod=12

Calculate retention:

Disk needed = (Ingestion rate × Average sample size × Retention seconds)

Example:
10k samples/sec × 1 byte × 31536000 sec (1 year) = 315 GB

2. Monitor Cardinality

High cardinality = memory issues

Check active series:

curl 'http://localhost:8428/api/v1/query?query=vm_active_time_series'

Reduce cardinality:

• Avoid high-cardinality labels (IDs, timestamps)
• Use relabeling to drop unnecessary labels
• Aggregate at source

3. Optimize Storage

Enable deduplication:

./victoria-metrics-prod -dedup.minScrapeInterval=60s

Set memory limit:

./victoria-metrics-prod -memory.allowedPercent=80

4. Configure Cache

# Index cache
./victoria-metrics-prod -storage.cacheSizeIndexDBDataBlocks=8GB

# Storage cache
./victoria-metrics-prod -storage.cacheSizeStorageTSID=2GB

5. Monitor Merge Performance

# Check merge metrics
curl http://localhost:8428/metrics | grep vm_merge

Slow merges indicate:

• Insufficient disk I/O
• Too many small parts
• Memory pressure

VictoriaMetrics Performance Tuning

Memory Optimization

# Limit memory usage
./victoria-metrics-prod \
  -memory.allowedPercent=80 \
  -search.maxMemoryPerQuery=2GB

Storage Optimization

# Storage path
./victoria-metrics-prod \
  -storageDataPath=/mnt/fast-ssd/victoria-metrics

Use SSD for better performance.

Query Optimization

# Limit concurrent queries
./victoria-metrics-prod \
  -search.maxConcurrentRequests=16 \
  -search.maxQueueDuration=30s

Advanced Configuration

Cluster Monitoring

Monitor multiple VictoriaMetrics instances:

plugins:
  victoria_node1:
    enabled: true
    host: vm-node1.internal
    port: 8428

  victoria_node2:
    enabled: true
    host: vm-node2.internal
    port: 8428

Docker Container

Monitor VictoriaMetrics in Docker:

plugins:
  victoriametrics:
    enabled: true
    host: victoriametrics-container
    port: 8428

Example Configurations

Basic Production

plugins:
  victoriametrics:
    enabled: true
    host: localhost
    port: 8428

With Authentication

plugins:
  victoriametrics:
    enabled: true
    host: victoria.internal
    port: 8428
    username: admin
    password: ${VICTORIA_PASSWORD}

High-Availability Cluster

plugins:
  victoria_storage1:
    enabled: true
    host: vm-storage1.internal
    port: 8428

  victoria_storage2:
    enabled: true
    host: vm-storage2.internal
    port: 8428

Limitations

Current Limitations

1. No per-metric cardinality - Total series count only
2. No slow query details - Query log not analyzed
3. No backup status - Backup metrics not collected

Scalability

Tested with:

• 10M+ active time series
• 1M+ samples per second
• TB-sized storage

Performance:

• Metrics endpoint response constant regardless of data size

Monitoring Checklist

Critical:

1. Storage approaching disk limit
2. High cardinality explosion
3. Ingestion rate drop
4. Memory > 90%

Important: 5. Slow queries increasing 6. Merge backlog 7. Cache hit rate low

Alert Thresholds

vm_data_size_percent: > 85
vm_active_time_series: > 1000000
vm_rows_per_second: < baseline × 0.5
vm_memory_percent: > 85
cache_hit_rate: < 80

Troubleshooting Performance

High Memory Usage

Symptom: Memory > 80%

Solutions:

1. Reduce cache sizes
2. Lower -memory.allowedPercent
3. Reduce query concurrency
4. Add more RAM

Slow Queries

Symptom: High query duration

Solutions:

1. Optimize query patterns
2. Reduce time range
3. Increase cache size
4. Add more CPU cores

Storage Growth

Symptom: Rapid disk usage growth

Solutions:

1. Reduce retention period
2. Enable deduplication
3. Reduce cardinality
4. Aggregate at source

Next Steps

• View all plugins
• System Requirements
• Agent Configuration Guide
• Troubleshooting Guide