Scaling Data Retrieval: Implementing Efficient Filtering and Pagination in ecommerce-fullstack

Building an e-commerce platform like ecommerce-fullstack brings a common challenge: as your product catalog grows, fetching everything at once becomes a performance bottleneck. I recently tackled this by implementing robust server-side filtering and pagination, ensuring the application remains snappy even as the data scales.

The Problem: The 'Fetch Everything' Trap

Initially, our product endpoints were naive. They simply queried the database for all available items. When we had ten products, it was fine. When we reached hundreds, payload sizes ballooned, and database latency spiked. We needed a way to request only what the user actually sees.

The Technical Solution

To solve this, I introduced a tiered request validation and database query pattern using Zod for schema validation and Prisma for database access.

Validating Inputs

First, I defined a schema to ensure that filters (like category) and pagination parameters (like page limit) are type-safe:

import { z } from 'zod';

const ProductQuerySchema = z.object({
  category: z.string().optional(),
  limit: z.coerce.number().min(1).max(100).default(10),
  offset: z.coerce.number().min(0).default(0),
});

Efficient Querying

With valid parameters, the integration with Prisma becomes straightforward. We translate these inputs directly into database constraints, which is far more efficient than filtering data in memory:

async function getProducts(query: unknown) {
  const { category, limit, offset } = ProductQuerySchema.parse(query);
  
  return await prisma.product.findMany({
    where: category ? { category: { name: category } } : {},
    take: limit,
    skip: offset,
  });
}

The Middleware Pattern

To keep the controller logic clean, I offloaded the validation to an Express middleware. By validating the request before it even reaches the route handler, we ensure that only sanitized, predictable data touches our database layer.

Key Takeaways

1. Shift Left on Validation: Use Zod to define input requirements early. If the user provides a negative limit or an invalid format, catch it before it consumes compute resources.
2. Database Over Application: Always push filtering and pagination logic down to the database engine. SQL and Prisma are optimized for these operations.
3. Sanity Limits: Always enforce a maximum limit per request. This prevents malicious actors or bugs from triggering 'select all' style queries that could crash your API.

By treating pagination not just as a UI requirement but as a core system constraint, we've successfully laid the foundation for a much larger product catalog without sacrificing performance.

Generated with Gitvlg.com