E2E Testing Guide

This document describes the end-to-end testing strategy and implementation for VMProber.

Table of Contents

• Overview
• Test Structure
• Running Tests
• Test Configuration
• Docker Compose Services
• Verified Metrics
• Issues Found & Fixed
• Known Limitations

Overview

E2E tests validate the complete flow of metrics from VMProber to VictoriaMetrics, ensuring that:

1. VMProber correctly executes probes
2. Metrics are formatted in Prometheus text format
3. Metrics are successfully pushed to VictoriaMetrics
4. Metrics can be queried from VictoriaMetrics
5. Grafana dashboard displays data correctly

Test Structure

File Location

vmprober/
└── tests/
    └── e2e/
        ├── configs_test.go           # Test configurations for different scenarios
        ├── dns_probes_test.go        # DNS probe tests
        ├── failure_scenarios_test.go # Failure and high-load tests
        ├── grafana_test.go           # Grafana integration tests
        ├── helpers_test.go           # Common helpers and utilities
        ├── http_probes_test.go       # HTTP/HTTPS probe tests
        ├── metrics_flow_test.go      # Metrics flow tests
        └── tcp_probes_test.go        # TCP probe tests

Test Categories

TCP Probe Tests (tcp_probes_test.go)

HTTP Probe Tests (http_probes_test.go)

DNS Probe Tests (dns_probes_test.go)

Failure Scenario Tests (failure_scenarios_test.go)

Metrics Flow Tests (metrics_flow_test.go)

Grafana Tests (grafana_test.go)

Metrics Flow Diagram

flowchart LR
    VMProber["VMProber"] --> vminsert["vminsert"]
    vminsert --> vmstorage["vmstorage"]
    vmstorage --> vmselect["vmselect"]
    vmselect --> Test["Test"]

Running Tests

Prerequisites

• Go 1.24+
• Docker and Docker Compose (or docker compose)
• Built VMProber binary (bin/vmprober)
• Available ports: 8480, 8481, 8482, 3000

Commands

# Build VMProber binary first
make build

# Run all E2E tests
go test -v -timeout 10m ./tests/e2e/...

# Run specific test file
go test -v -timeout 10m ./tests/e2e/tcp_probes_test.go ./tests/e2e/helpers_test.go ./tests/e2e/configs_test.go

# Run specific test
go test -v -timeout 10m -run TestE2E_MetricsFlow ./tests/e2e/...

# Run with verbose output
go test -v -timeout 10m -count=1 ./tests/e2e/...

# Skip long-running tests
go test -v -timeout 5m -short ./tests/e2e/...

# Using make (if configured)
make e2e-test

Troubleshooting Test Failures

Test Configuration

Tests use configuration templates defined in configs_test.go:

Example Configuration

# HTTP Server
listen:
  port: 8429
  host: "0.0.0.0"

# Push mode for VictoriaMetrics
push:
  enabled: true
  endpoints:
    - url: "http://localhost:8480/insert/0/prometheus/api/v1/import"

# Test targets
targets:
  static:
    - host: "127.0.0.1"
      port: 22
      proto: "tcp"
      interval: 5s
      timeout: 2s
      labels:
        test: "e2e"

# Metrics configuration with job label
metrics:
  namespace: "vmprober"
  custom_labels:
    job: "blackbox/vmprober"

Configuration Highlights

Docker Compose Services

The E2E tests use the existing docker-compose.yml which provides:

Service Health Checks

Tests verify service readiness before running:

// Check vmselect is ready
resp, err := http.Get("http://localhost:8481/health")
if err != nil || resp.StatusCode != 200 {
    // Wait and retry...
}

Verified Metrics

The E2E tests verify the following metrics exist in VictoriaMetrics:

Probe Metrics

Job Metrics

Verification Queries

# Check if probe metrics exist
vmprober_probe_attempts_total

# Get probe success rate
vmprober_probe_success_total / vmprober_probe_attempts_total

# Check RTT histogram percentile
histogram_quantile(0.99, vmprober_probe_rtt_seconds_bucket)

# Filter by job label (used by Grafana dashboard)
vmprober_probe_attempts_total{job="blackbox/vmprober"}

Issues Found & Fixed

During E2E testing implementation, several issues were identified and resolved:

1. Missing Job Label in Metrics

2. Incorrect Data Format for vminsert

3. Wrong Content-Type Header

4. Missing Namespace Filtering

Known Limitations

1. Histogram Metric Parsing

Histogram metrics are parsed as separate metrics (_bucket, _sum, _count), which may result in incomplete information during export.

2. Network Dependencies

Tests require network connectivity to:

• 127.0.0.1:22 (SSH) - may fail if SSH is not running
• 8.8.8.8:53 (Google DNS) - requires internet access

3. Resource Requirements

Docker Compose services require:

• ~500MB RAM total
• ~1GB disk space for vmstorage data

See Also

• Setup Guide - Development environment setup
• Configuration - Configuration reference
• Troubleshooting - Troubleshooting guide
• Metrics Reference - Metrics documentation