CPOW Pipeline Documentation

This directory contains detailed technical documentation for the CPOW (Continuous Point-on-Wave) data pipeline, covering high-performance data acquisition, shared memory operations, and real-time processing.

Pipeline Overview

The CPOW data pipeline consists of two main processes that work together to capture, process, and store high-frequency power measurement data at 32kHz across 7 channels.

Acquire Pipeline

Data Flow

Hardware Layer: ADE9000 sensor captures 4-channel power data at 32kHz
Acquire Process: Reads sensor data via TCP and writes to shared memory
Shared Memory IPC: High-performance lock-free circular buffer for inter-process communication
Process Process: Reads from shared memory, processes with SIMD optimizations, outputs to Parquet files
Storage: Compressed Parquet files with overflow handling

Key Features

Performance:

Lock-free shared memory with atomic operations
SIMD-optimized processing (ARM NEON)
High-priority threads for real-time performance
Cache-aligned data structures
Zero-copy design where possible

Robustness:

Graceful shutdown handling
Connection retry logic
Buffer overflow detection and recovery
Data integrity verification

synapse-cpow-process Architecture

The synapse-cpow-process component reads raw sensor data from shared memory, transforms it through SIMD-optimized processing, and outputs structured Parquet files.

Process Pipeline

Core Data Flow

SensorDataRow (21 bytes)
    ↓ [SharedMemoryReader]
WaveBlockData batches
    ↓ [WaveSink buffering]
Bounded channel stream
    ↓ [24→32 bit conversion]
DataBuffer (i32 circular buffer)
    ↓ [SIMD scaling operations]
Scaled float32 data
    ↓ [Arrow serialization]
Parquet files (ZSTD compressed)

Channel Mapping

Channel 0: IA (Current A)   - iscale factor
Channel 1: VA (Voltage A)   - vscale factor
Channel 2: IB (Current B)   - iscale factor
Channel 3: VB (Voltage B)   - vscale factor
Channel 4: IC (Current C)   - iscale factor
Channel 5: VC (Voltage C)   - vscale factor
Channel 6: IN (Neutral)     - iscale factor

Detailed Documentation

CPOW Data Processing Pipeline

cpow_process_analysis.md

Level: Advanced/Expert

Complete architectural deep dive into the CPOW process binary (synapse-cpow-process):

Data pipeline from sensor acquisition through Parquet file output
Shared memory reader/writer implementation
Arrow IPC integration for columnar data storage
NEON optimization for ARM platforms
Command-line interface and configuration options
Error handling and recovery mechanisms

Key sections:

Section 2: Data Pipeline Architecture
Section 7: Processing Pipeline Detailed Walkthrough
Section 10: Performance Considerations

Shared Memory Operations (eq-memops)

eq_memops_analysis.md

Level: Advanced/Expert | Recommended starting point

The eq-memops crate provides foundational shared memory abstractions:

Generic SharedMemoryAccess trait and implementations
ArrowSharedMemory lock-free multi-reader architecture
SharedMemoryReader patterns and builder API
Multi-reader support (up to 16 concurrent readers)
Memory layout, alignment, and atomic synchronization
NEON optimization for ARM platforms

Key sections:

Section 2: Arrow Shared Memory Implementation
Section 3: Reader/Writer Patterns
Section 7: Memory Layout & Synchronization

eq_memops_visuals.md

Level: Intermediate to Advanced | Companion to eq_memops_analysis.md

Visual ASCII diagrams covering:

Data flow architecture
Memory layout diagrams
Writer-reader synchronization flowcharts
State machines and error transitions

SUMMARY.txt

Plain text alternative to eq_memops_analysis.md with linear narrative flow.

Configuration & Deployment

MEMORY_CONFIG_REFERENCE.md

Level: Intermediate | Audience: DevOps, deployment engineers

Memory-related configuration parameters:

DuckDB pool and query configuration
Shared memory buffer sizing
Connection limits and WebSocket configuration
Memory budget analysis (current vs edge-optimized)
Edge device configuration overrides

WebSocket Integration

WEBSOCKET_BATCH_AGGREGATION.md

WebSocket batch aggregation for real-time data streaming.

Reading Paths

New to the Codebase

eq_memops_visuals.md - Visual overview
eq_memops_analysis.md sections 1-4 - Foundational concepts
cpow_process_analysis.md section 2 - Data flow understanding
MEMORY_CONFIG_REFERENCE.md - Deployment considerations

Performance Optimization

cpow_process_analysis.md section 10 - Performance Considerations
eq_memops_analysis.md section 13 - Performance Characteristics
MEMORY_CONFIG_REFERENCE.md - Configuration tuning

Debugging Issues

cpow_process_analysis.md section 9 - Error Handling & Recovery
eq_memops_analysis.md section 8 - Error Handling Hierarchy
eq_memops_visuals.md section 8 - Error State Transitions

Quick Reference

Key Concepts

WaveBlockData: 256-row blocks of 7-channel sensor data (8ms @ 32kHz)
SensorSharedMemory: Raw 24-bit sensor data circular buffer
ArrowSharedMemory: Lock-free multi-reader Arrow IPC shared memory
DataBuffer: Conversion layer from 24-bit to 32-bit with batch views
NEON optimization: ARM SIMD acceleration for data conversion

Critical Constants

N_ROWS_PER_BLOCK: 256 rows
SAMPLE_HZ: 32,000 Hz
N_CHANNELS: 7 (ia, va, ib, vb, ic, vc, i_n)
ROWS_PER_BATCH: 320,000 rows (10 seconds)
MAX_READERS: 16 concurrent readers

Key Source Files

eq-memops/src/ipc/arrow.rs: Arrow shared memory implementation
eq-memops/src/ipc/reader.rs: Generic shared memory reader
synapse-cpow/process/src/bin/default.rs: CPOW process binary entry point
eq-cpow/sensor/src/lib.rs: Sensor shared memory interface

PMON Pipeline - Power monitoring pipeline
Unified Backend API - Backend API and server architecture
Library Crates - Overview of Rust crates and dependencies
Traits and Generics - Rust patterns used in the codebase
Individual crate READMEs in server/crates/ directory

Last Reviewed: 2026-02-02

© 2026 EQ Systems Inc.