When should I use streaming instead of batch processing?

When individual events matter and freshness is part of the product — alerts, live tracking, routing decisions. If output that's minutes late is merely late, batch is simpler and fine. If it's worthless or embarrassing, stream. Ask what staleness costs; the architecture picks itself.

What are the common geofencing pitfalls?

GPS jitter firing enter/exit repeatedly for a stationary device (fix with dwell debouncing), urban-canyon teleports (sanity-check speed between pings), overlapping fences triggering together (define priority rules with product), and unreliable device clocks (order by server receipt time). The map math is trivial; the signal quality is the work.

How does backpressure work in a streaming system?

Producers write to the stream at their pace; consumers read at theirs. A traffic spike grows the queue instead of crashing workers, and consumer lag — how far behind real time you are — becomes a single observable, alertable number. Batch systems lack this: falling behind doubles the next run until memory gives out.

100K Geo Events a Day: Streams, Not Cron Jobs

It started, as these things do, as a cron job. Every five minutes: wake up, pull new location pings, check them against geofences, fire alerts. Perfectly reasonable at 5K events a day. By month two we were at ten times that, the batch took longer than the interval, runs began overlapping, and alerts started arriving about places drivers had already left.

The lesson wasn't 'crons bad.' It was learning to ask one question early: does staleness cost anything? Nightly invoice rollups can be five minutes late forever and nobody cares. A 'your driver has arrived' notification that's five minutes late is not a feature — it's an apology. When individual events matter and freshness is the product, you want a stream.

The shape: pings flow, consumers chew

The architecture that replaced the cron was almost boring: devices push pings into a stream, consumer groups chew through them continuously, each consumer owns a slice of the traffic. No five-minute heartbeat, no overlap window, no thundering batch at :00. Just a steady river with workers standing in it.

Geofencing is where GPS lies to you

On the map, geofencing is a circle and a point. In production, GPS is a drunk narrator:

Jitter bounces a parked vehicle in and out of a fence twelve times in a minute — every crossing 'true', every alert garbage. Fix: dwell debouncing (inside for N seconds before 'entered').
Urban canyons teleport pings across the street and back. Fix: sanity-check speed between consecutive pings; discard physics violations.
Overlapping fences fire together at boundaries. Fix: priority rules decided with product, not by whichever consumer ran first.
Device clocks lie. Some pings arrive from the past, a few from the future. Fix: trust server receipt time for ordering, log device time as advisory.

Backpressure or death

The cron's failure mode is the deadly one: fall behind, next run doubles, memory blows, 2am page. A stream falls behind *gracefully* — consumer lag is a number you can watch, alert on, and scale against. The queue absorbs the spike; the workers grind at their own pace; nothing melts. Lag became our favorite metric: one integer that says exactly how far behind reality the system is.

consumer.ts

// the whole streaming religion in one loop:
// read a batch, process, ack, repeat — at YOUR pace, not the producer's
while (running) {
  const events = await stream.readGroup('geo-workers', { count: 200, block: 2000 });
  for (const e of events) {
    const verdicts = fences.evaluate(e);   // enter/exit/dwell decisions
    if (verdicts.length) await alerts.queue(verdicts);
    await stream.ack(e.id);                // crash before ack? redelivered. after? done once.
  }
  metrics.gauge('consumer.lag', await stream.lag('geo-workers'));
}

The metric that mattered wasn't technical

Fresh location data plus live fence events meant routing decisions could react to *now* — reroute around the stalled cluster, resequence stops while the day was still happening. Measured travel time dropped 35%. Nobody in a product review asked about consumer groups; they asked if the number was real. That's the job: the stream was plumbing, the 35% was the product.

Batch tells you what happened. Streaming lets you do something about what's happening. Price the difference and the architecture picks itself.

You can't fix a river you can't see

Redis held the hot state — live positions, fence status, dedupe keys — and ELK ate every structured event so 'why did this alert fire at 14:32' was a query, not an archaeology dig. When an alert misfires, you replay the exact ping sequence and watch the decision happen. Half of running a stream is being able to see into it.

The full build is in my work section under real-time geo intelligence. But the takeaway travels: if the value of an event decays in minutes, process it in seconds. Everything else can stay a cron job — I still love a good cron job.

Key takeaways

01Choose streaming when staleness costs money — if an event's value decays in minutes, process it in seconds.
02GPS lies: debounce dwell time, discard physics-violating jumps, and trust server receipt time over device clocks.
03Consumer lag is the metric that matters — streams fall behind gracefully where batch jobs fall over.

FAQ

ArchitectureStreamingReal-time