Memory Configuration Reference for Edge Device Deployment

This document catalogs all memory-related configurations in the Synapse RAG system for edge device optimization.

NOTE: Some of these values must be reduced prior to running on i.mx8M Plus gateway

Critical Memory Allocations

1. DuckDB Database Configuration (`eq-db/src/duckdb/config.rs`)

Pool Configuration

#![allow(unused)]
fn main() {
pub struct DuckDBPoolConfig {
    memory_limit: String,           // DEFAULT: "6GB" ⚠️ HIGH MEMORY
    threads: usize,                 // DEFAULT: 6
    max_connections: usize,         // DEFAULT: 10
    min_connections: usize,         // DEFAULT: 2
    max_concurrent_queries: usize,  // DEFAULT: 100
}
}

Query Configuration

#![allow(unused)]
fn main() {
pub struct DuckDBQueryConfig {
    batch_size: usize,           // DEFAULT: 10,000 rows
    max_result_size: usize,      // DEFAULT: 100MB ⚠️ HIGH MEMORY
    statement_timeout: Duration, // DEFAULT: 5 minutes
}
}

Data Configuration

#![allow(unused)]
fn main() {
pub struct DuckDBDataConfig {
    metadata_cache_size: usize,  // DEFAULT: 256MB ⚠️ MEMORY USAGE
    max_file_handles: usize,     // DEFAULT: 1,000
}
}

EDGE DEVICE RECOMMENDATIONS:

memory_limit: Reduce to “1GB” or “2GB” max
max_connections: Reduce to 2-4
max_result_size: Reduce to 10-50MB
metadata_cache_size: Reduce to 64MB or 128MB

2. RAG Core Configuration (`rag-core/src/config.rs`)

Context Store Configuration

#![allow(unused)]
fn main() {
pub struct ContextStoreConfig {
    max_chunk_tokens: usize,     // DEFAULT: 1,024
    max_total_tokens: usize,     // DEFAULT: 8,192
    compression_threshold: f32,   // DEFAULT: 0.8
    max_context_age_seconds: u64, // DEFAULT: 86,400 (24 hours)
}
}

Tool Configuration

#![allow(unused)]
fn main() {
pub struct ToolConfig {
    max_memory_mb: usize,        // DEFAULT: 512MB (DuckDB), 256MB (Plot) ⚠️
    timeout_seconds: u64,        // DEFAULT: 30-60s
}

pub struct ToolRegistryConfig {
    max_output_size_bytes: usize, // DEFAULT: 10MB ⚠️
}
}

Database Backend

#![allow(unused)]
fn main() {
pub struct DatabaseBackendConfig {
    max_connections: usize,      // DEFAULT: 10
    connection_string: Option<String>, // DEFAULT: ":memory:" ⚠️
}
}

EDGE DEVICE RECOMMENDATIONS:

max_total_tokens: Reduce to 4,096
max_memory_mb: Reduce DuckDB to 128MB, Plot to 64MB
max_output_size_bytes: Reduce to 1-5MB
max_connections: Reduce to 2-4

3. Embedding Configuration (`eq-semantic/src/embedding/config.rs`)

Model Memory Estimation

#![allow(unused)]
fn main() {
impl EmbeddingConfig {
    fn estimated_memory_mb(&self) -> usize {
        match self.model_name.as_str() {
            "all-MiniLM-L6-v2" => 90,   // 90MB model ✓ EDGE FRIENDLY
            "all-mpnet-base-v2" => 420, // 420MB model ⚠️ HIGH MEMORY
            _ => 200, // Default estimate
        }
    }
}
}

Batch Processing

#![allow(unused)]
fn main() {
pub struct EmbeddingConfig {
    batch_size: usize,              // DEFAULT: 32, MAX: 1000 ⚠️
    max_sequence_length: usize,     // DEFAULT: 256, MAX: 512
    normalize_embeddings: bool,     // DEFAULT: true
}

pub struct ConcurrentConfig {
    batch_size: usize,              // DEFAULT: 32
    channel_buffer_size: usize,     // DEFAULT: 4 batches
    parquet_chunk_size: usize,      // DEFAULT: 1,000 records ⚠️
}
}

Memory Optimization Methods

#![allow(unused)]
fn main() {
// Low memory configuration
pub fn low_memory() -> ConcurrentConfig {
    batch_size: 16,           // Reduced from 32
    channel_buffer_size: 2,   // Reduced from 4
    parquet_chunk_size: 500,  // Reduced from 1000
}

// Memory per batch estimation
fn optimal_batch_size(&self, available_memory_mb: usize) -> usize {
    let memory_per_item = match self.embedding_dimension() {
        384 => 2,  // ~2MB per batch item (MiniLM)
        768 => 4,  // ~4MB per batch item (MPNet) ⚠️
        _ => 3,
    };
}
}

EDGE DEVICE RECOMMENDATIONS:

Use all-MiniLM-L6-v2 model (90MB vs 420MB)
batch_size: 8-16 (vs default 32)
channel_buffer_size: 2 (vs default 4)
parquet_chunk_size: 500 (vs default 1000)

4. RAG Engine Model Loading (`synapse-rag/engine/src/model.rs`)

Model Files and Caching

#![allow(unused)]
fn main() {
// Local model paths - potential high disk usage
let local_base = PathBuf::from("/mnt/code/dev/models/qwen2.5-3b"); 

// Cache directory
fn get_cache_dir(model_name: &str) -> Result<PathBuf> {
    dirs::cache_dir()
        .join("synapse-rag")
        .join("models")
        .join(model_name.replace("/", "_")); // ⚠️ DISK USAGE
}
}

Model Configuration

#![allow(unused)]
fn main() {
pub struct ModelConfig {
    vocab_size: usize,              // DEFAULT: 151,936
    hidden_size: usize,             // DEFAULT: 2,048
    intermediate_size: usize,       // DEFAULT: 11,008 ⚠️
    num_hidden_layers: usize,       // DEFAULT: 36 ⚠️
    max_position_embeddings: usize, // DEFAULT: 32,768 ⚠️
}
}

EDGE DEVICE RECOMMENDATIONS:

CRITICAL: Switch to smaller model (1B params vs 3B)
Clear model cache regularly
Use quantized models if available
Consider using Ollama backend instead of direct model loading

5. Shared Memory Configuration (`synapse-config/src/schema/config-template.yml`)

CPOW Raw Data Buffer

shared_memory:
  raw:
    size: 13520100        # 20s of raw sensor data (~13MB) ⚠️
    name: "cpow_raw"
  # processed:
  #   size: 18874368      # 18MB for processed data ⚠️ (DISABLED)

Connection Limits

connection_limit:
  max_connections_per_ip: 20              # ⚠️ HIGH
  max_incoming_data_per_minute: 1048576   # 1MB/min ⚠️
  max_message_size: 1048576               # 1MB ⚠️

ws:
  max_connections: 10                     # WebSocket connections ⚠️

EDGE DEVICE RECOMMENDATIONS:

shared_memory.raw.size: Reduce to 6760050 (10s vs 20s)
max_connections_per_ip: Reduce to 5-10
ws.max_connections: Reduce to 2-5

Memory Budget Summary for Edge Device

Current Default Configuration (Estimated)

DuckDB: 6GB + 256MB cache = 6.25GB ⚠️
Embedding Model: 420MB (MPNet) ⚠️
RAG Model: 3B params ≈ 6-12GB ⚠️
Shared Memory: 13MB + 18MB = 31MB
Tool Memory: 512MB + 256MB = 768MB
Connection Buffers: ~100MB
TOTAL: ~13-19GB ⚠️ EXCESSIVE FOR EDGE

Recommended Edge Configuration

DuckDB: 1GB + 64MB cache = 1.06GB ✓
Embedding Model: 90MB (MiniLM) ✓
RAG via Ollama: 0MB (external process) ✓
Shared Memory: 6.5MB + 0MB = 6.5MB ✓
Tool Memory: 128MB + 64MB = 192MB ✓
Connection Buffers: ~50MB ✓
TOTAL: ~1.4GB ✓ EDGE SUITABLE

Configuration Override Files

Edge Device Config Override (`edge-config.yml`)

# Override for edge deployment
duckdb:
  memory_limit: "1GB"
  max_connections: 4
  max_result_size: 50MB
  metadata_cache_size: 64MB

embedding:
  model_name: "sentence-transformers/all-MiniLM-L6-v2"
  batch_size: 16
  channel_buffer_size: 2

rag:
  use_ollama: true  # External process, no direct model loading
  ollama_endpoint: "http://localhost:11434"

tools:
  duckdb:
    max_memory_mb: 128
  plot:
    max_memory_mb: 64

shared_memory:
  raw:
    size: 6760050  # 10s instead of 20s

connection_limit:
  max_connections_per_ip: 5
  ws:
    max_connections: 3

Monitoring and Alerts

Memory Usage Monitoring Points

DuckDB Pool: Track active connections and memory usage
Embedding Model: Monitor batch processing memory spikes
Shared Memory: Monitor buffer fill rates
Tool Execution: Track peak memory per tool execution
Connection Pools: Monitor connection count and buffer sizes

Critical Memory Thresholds for Edge

Total System: Stay under 2GB total usage
Single Process: No process should exceed 512MB
DuckDB: Monitor query result sizes
Embedding: Watch for batch size memory spikes
Shared Memory: Alert on buffer overruns

Next Steps

Immediate: Switch to Ollama backend to eliminate direct model loading
Configuration: Apply edge-optimized configuration overrides
Testing: Validate memory usage under load with monitoring
Optimization: Profile actual usage patterns and adjust accordingly

Keyboard shortcuts

Energy Quotient™ Developer Documentation