Why I Chose MongoDB for High-Throughput Media Processing
Database selection is one of those decisions that reverberates through every layer of your application. For the infinitegif platform -- a media processing system that ingests, transforms, and serves thousands of media items daily -- the database choice shaped everything from our data models to our deployment strategy.
The Requirements
Before evaluating databases, I defined what the system needed:
- Flexible schemas: Media metadata varies wildly between sources.
- High write throughput: Thousands of documents updated per hour during peak ingestion.
- Rich querying: Complex aggregations for trending calculations and recommendations.
- Document-level atomicity: Each media item is a self-contained document with nested metadata.
- Reasonable operational complexity: This runs on a single server, not a distributed cluster.
The Contenders
PostgreSQL with JSONB
PostgreSQL is my go-to for most applications. Its JSONB support is excellent. However, the relational model added friction for our highly variable metadata. Every time a new source introduced different fields, we were either adding columns or stuffing everything into JSONB -- at which point, why use a relational database?
Redis
Redis is fantastic for caching and real-time data, but it is not a primary data store for 180,000+ documents with complex querying needs. We do use Redis for caching hot data and rate limiting.
MongoDB
MongoDB's document model maps naturally to our data. Each media item is a document containing everything about that item -- metadata, processing status, derivatives, engagement metrics. No joins, no schema migrations, no impedance mismatch.
Performance in Practice
After six months:
- Average read latency: 12ms
- Average write latency: 8ms
- Aggregation pipeline (trending feed): 180ms
- Full-text search queries: 45ms
- Database size: 4.2GB for 180,000+ documents
Choose your database based on how your data actually looks and flows, not on what is popular. The best database is the one that makes your application code simpler.
Indexing Strategy
The single most important optimization was our indexing strategy. We use compound indexes aligned with our most common query patterns: {status: 1, source: 1, createdAt: -1} for pipeline queries, {type: 1, score: -1} for feed generation, and a text index on title and description for search.
One pitfall to avoid: over-indexing. Each index adds write overhead. We initially had 12 indexes per collection and trimmed it to 6 after analyzing actual query patterns. Write performance improved by 30%.