Why MongoDB Is Hard to Optimize

Getting started with MongoDB is genuinely easy. You insert a document, you query it, it works. The problems emerge at scale — when collections have tens of millions of documents and queries that returned in 1ms now take 30 seconds.

The root cause is almost always one of three things: schema design that ignores access patterns, missing or wrong indexes, or aggregation pipelines that do unnecessary work early. This post covers all three.

Part 1: Document Schema Design

The Core Question: Embed or Reference?

MongoDB's primary schema design decision is whether to embed related data within a document or reference it from a separate collection — the equivalent of denormalization vs normalization in SQL.

// Embedding: related data lives inside the parent document
{
  _id: ObjectId("..."),
  title: "MongoDB Deep Dive",
  author: {
    id: 42,
    name: "Alice Smith",
    email: "alice@example.com"  // ← embedded
  },
  tags: ["database", "nosql", "mongodb"],
  comments: [                   // ← embedded array
    { user: "Bob", body: "Great post!", date: ISODate("2025-03-15") },
    { user: "Carol", body: "Very helpful", date: ISODate("2025-03-16") }
  ]
}

// Referencing: related data lives in a separate collection
{
  _id: ObjectId("..."),
  title: "MongoDB Deep Dive",
  author_id: 42,               // ← reference (foreign key equivalent)
  tag_ids: [ObjectId("..."), ObjectId("...")],  // ← array of references
  comment_ids: [ObjectId("..."), ObjectId("...")]
}

The embedding vs referencing decision:

Embed when:
  ✅ Data is always accessed together (post + its tags)
  ✅ One-to-few relationship (post has 0–50 comments)
  ✅ Data does not change independently (address inside order)
  ✅ You need atomicity across the relationship

Reference when:
  ✅ One-to-many with unbounded growth (post could have 10,000 comments)
  ✅ Data is accessed independently (users are queried separately from posts)
  ✅ Data is shared across multiple parents (product referenced by many orders)
  ✅ Embedded data would exceed 16 MB document limit

The 16 MB document limit: MongoDB's maximum document size is 16 MB. An embedded array that grows without bound (all comments for a viral post) will eventually hit this limit. Unbounded arrays must be referenced.

The Bucket Pattern for Time-Series

When storing time-series data (events, metrics, IoT readings), naive one-document-per-event creates millions of tiny documents — expensive to index and query. The bucket pattern groups events into time buckets:

// Naive: one document per event (bad for time-series)
{ sensor_id: "temp-01", timestamp: ISODate("2025-03-15T10:00:00"), value: 23.5 }
{ sensor_id: "temp-01", timestamp: ISODate("2025-03-15T10:01:00"), value: 23.7 }
// ... 1440 documents per sensor per day

// Bucket pattern: one document per hour per sensor (good)
{
  sensor_id: "temp-01",
  hour: ISODate("2025-03-15T10:00:00"),
  count: 60,
  sum: 1413.6,
  min: 23.1,
  max: 24.2,
  readings: [23.5, 23.7, 23.6, 23.8, ...]  // 60 values embedded
}
// 24 documents per sensor per day instead of 1440
// Aggregations (avg, min, max) pre-computed — no pipeline needed

The Outlier Pattern

When most documents follow one pattern but a few outliers differ dramatically:

// Normal post: embedded comments (works for 99% of posts)
{
  _id: ObjectId("..."),
  title: "Normal Post",
  comments: [/* up to ~100 comments */],
  has_extra_comments: false
}

// Viral post: comments overflow into overflow documents
{
  _id: ObjectId("..."),
  title: "Viral Post",
  comments: [/* first 100 comments */],
  has_extra_comments: true  // ← flag indicating overflow
}

// Overflow collection
{
  post_id: ObjectId("..."),
  comments: [/* next 100 comments */]
}

Part 2: The Aggregation Pipeline

The aggregation pipeline is MongoDB's equivalent of SQL's GROUP BY, JOIN, and window functions. Data flows through a sequence of stages, each transforming the input documents.

// Basic pipeline structure
db.collection.aggregate([
  { $stage1: { ... } },
  { $stage2: { ... } },
  { $stage3: { ... } }
])

Core Stages

$match — Filter documents (like SQL WHERE)

// Put $match as early as possible to reduce documents processed downstream
{ $match: {
  status: "completed",
  created_at: { $gte: ISODate("2025-01-01"), $lt: ISODate("2026-01-01") }
} }

$group — Aggregate (like SQL GROUP BY)

{ $group: {
  _id: "$customer_id",              // group by field
  total_spent: { $sum: "$amount" },
  order_count: { $sum: 1 },
  avg_order:   { $avg: "$amount" },
  first_order: { $min: "$created_at" },
  last_order:  { $max: "$created_at" }
} }

$project — Shape the output (like SQL SELECT)

{ $project: {
  customer_id: "$_id",
  total_spent: 1,
  order_count: 1,
  avg_order: { $round: ["$avg_order", 2] },
  _id: 0                           // exclude _id from output
} }

$sort — Sort results (like SQL ORDER BY)

{ $sort: { total_spent: -1 } }    // -1 = descending, 1 = ascending

$limit and $skip — Paginate

{ $limit: 10 },
{ $skip: 20 }

$lookup — Join collections (like SQL JOIN)

{ $lookup: {
  from: "customers",              // the collection to join
  localField: "customer_id",      // field in current documents
  foreignField: "_id",            // field in joined collection
  as: "customer"                  // output array field name
} },
{ $unwind: "$customer" }         // flatten the array (1:1 join)

$unwind — Flatten array fields

// Input: { order_id: 1, items: [{product: "A"}, {product: "B"}] }
{ $unwind: "$items" }
// Output: two documents:
//   { order_id: 1, items: {product: "A"} }
//   { order_id: 1, items: {product: "B"} }

$addFields — Add computed fields

{ $addFields: {
  revenue_category: {
    $switch: {
      branches: [
        { case: { $gte: ["$amount", 1000] }, then: "high" },
        { case: { $gte: ["$amount", 100] },  then: "medium" }
      ],
      default: "low"
    }
  }
} }

A Complete Aggregation Example

Monthly revenue by product category, top 5 categories:

db.orders.aggregate([
  // Stage 1: Filter to completed orders in 2025
  { $match: {
    status: "completed",
    created_at: { $gte: ISODate("2025-01-01") }
  }},

  // Stage 2: Unwind order items to get one doc per item
  { $unwind: "$items" },

  // Stage 3: Join with products collection to get category
  { $lookup: {
    from: "products",
    localField: "items.product_id",
    foreignField: "_id",
    as: "product"
  }},
  { $unwind: "$product" },

  // Stage 4: Group by month + category
  { $group: {
    _id: {
      month: { $dateToString: { format: "%Y-%m", date: "$created_at" } },
      category: "$product.category"
    },
    revenue: { $sum: { $multiply: ["$items.price", "$items.qty"] } },
    orders:  { $sum: 1 }
  }},

  // Stage 5: Sort by revenue descending
  { $sort: { revenue: -1 } },

  // Stage 6: Group by month, collect top categories
  { $group: {
    _id: "$_id.month",
    categories: { $push: { category: "$_id.category", revenue: "$revenue" } }
  }},

  // Stage 7: Take top 5 categories per month
  { $project: {
    month: "$_id",
    top_categories: { $slice: ["$categories", 5] },
    _id: 0
  }},

  { $sort: { month: 1 } }
])

Pipeline Performance Rules

Rule 1 — $match first: Place $match as the first stage whenever possible. It reduces the document count before expensive stages like $lookup and $group.

Rule 2 — $match before $unwind: Filtering before unwinding is dramatically cheaper — unwinding multiplies document count by the array length.

Rule 3 — $project to reduce document size: Remove fields you do not need before $group or $sort to reduce memory usage.

Rule 4 — $sort + $limit pushdown: MongoDB can push $sort + $limit down to an index scan when they appear early in the pipeline — avoiding a full collection sort.

Rule 5 — Use explain() to verify index usage:

db.orders.explain("executionStats").aggregate([
  { $match: { status: "completed" } },
  { $group: { _id: "$customer_id", total: { $sum: "$amount" } } }
])
// Look for: IXSCAN (index used) vs COLLSCAN (full collection scan)
// totalDocsExamined vs nReturned — high ratio means index is needed

Part 3: Index Strategies

Single Field Index

db.orders.createIndex({ customer_id: 1 })   // 1 = ascending, -1 = descending
db.orders.createIndex({ created_at: -1 })   // descending for "latest first" queries

Compound Index

Multiple fields in a single index — follows the same leftmost prefix rule as PostgreSQL B-Tree indexes.

// Supports: {status}, {status, created_at}, {status, created_at, customer_id}
// Does NOT support: {created_at}, {customer_id}
db.orders.createIndex({ status: 1, created_at: -1, customer_id: 1 })

// ESR Rule: Equality → Sort → Range (optimal compound index order)
// 1. Fields filtered by equality come first
// 2. Sort fields come next
// 3. Range fields come last
db.orders.createIndex({
  status: 1,        // Equality: WHERE status = 'completed'
  created_at: -1,   // Sort: ORDER BY created_at DESC
  amount: 1         // Range: WHERE amount > 100
})

Multikey Index (Indexing Arrays)

MongoDB automatically creates a multikey index when indexing a field that contains an array. Each array element becomes an index entry.

// Document: { tags: ["database", "nosql", "mongodb"] }
db.posts.createIndex({ tags: 1 })
// Creates 3 index entries: "database", "nosql", "mongodb"
// Query: db.posts.find({ tags: "nosql" }) → uses index

// Limitation: cannot create a compound index on two array fields
db.posts.createIndex({ tags: 1, authors: 1 })  // ERROR if both are arrays

Text Index (Full-Text Search)

db.articles.createIndex({ title: "text", body: "text" })
// Tokenizes text, removes stop words, stems words

db.articles.find({ $text: { $search: "mongodb aggregation" } })
// Returns documents containing "mongodb" or "aggregation"
// Sort by relevance score:
db.articles.find(
  { $text: { $search: "mongodb aggregation" } },
  { score: { $meta: "textScore" } }
).sort({ score: { $meta: "textScore" } })

Partial Index

Index only documents matching a filter condition:

// Only index active orders — completed orders are rarely queried by status
db.orders.createIndex(
  { customer_id: 1, created_at: -1 },
  { partialFilterExpression: { status: { $in: ["pending", "processing"] } } }
)
// Smaller index, only covers the query pattern that matters

TTL Index (Auto-Expiry)

Automatically delete documents after a specified time:

// Delete session documents 24 hours after created_at
db.sessions.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 86400 }
)
// MongoDB runs a background job every 60 seconds to delete expired documents

Index Hints and explain()

// Force a specific index
db.orders.find({ status: "completed" }).hint({ status: 1, created_at: -1 })

// Full explain output
db.orders.find({ customer_id: 42 }).explain("executionStats")
// Key fields to check:
//   executionStats.totalDocsExamined  — should be close to nReturned
//   executionStats.executionTimeMillis — total query time
//   winningPlan.inputStage.stage       — IXSCAN good, COLLSCAN bad

The 5 Most Common MongoDB Anti-Patterns

Anti-pattern 1 — Unbounded arrays. Embedding an array that grows without limit eventually hits the 16 MB document limit and degrades write performance. Reference instead of embed when cardinality is high.

Anti-pattern 2 — Too many indexes. Every index slows down writes and consumes RAM. Profile actual queries before adding indexes. Remove unused indexes regularly.

// Find unused indexes
db.orders.aggregate([{ $indexStats: {} }])
// Look for: accesses.ops = 0 (never used)

Anti-pattern 3 — $where or JavaScript in queries. Runs JavaScript server-side, bypasses indexes, extremely slow at scale. Always use native query operators.

Anti-pattern 4 — Large $in arrays. $in: [id1, id2, ..., id10000] performs 10,000 index lookups. Better: use a reference query or reshape data so the lookup is on an indexed field.

Anti-pattern 5 — $lookup without an index on foreignField. Every $lookup does a collection scan on the joined collection if foreignField is not indexed. Always index the foreign field.

// Before running $lookup on customers.customer_id:
db.customers.createIndex({ _id: 1 })  // usually already exists (_id is always indexed)
// But for non-_id foreign fields:
db.orders.createIndex({ customer_id: 1 })  // ensure the join field is indexed

🧭 What's Next

• Post 21: Redis Deep Dive — Redis is not just a cache; its five core data structures solve problems that would require complex application code, and its persistence and Pub/Sub features make it a production-grade platform