Best practices & pitfalls

1) Topic design: prevent fanout explosions

Do

• Design topics by who consumes (not just who publishes).
• Keep hot topics narrow. Use per-device topics for telemetry; use shared topics sparingly.
• Put tenant/site/device hierarchy early (helps filters and routing).

Avoid

• One “global” topic that thousands of clients subscribe to.
• Excessive wildcards in high-fanout paths (# on broad roots).

Rule of thumb: Always estimate fanout:
effective deliveries/sec = publishes/sec × avg subscribers per publish

2) QoS strategy: pick the lowest QoS that meets business needs

• QoS 0: fastest; can lose messages.
• QoS 1: “at least once”; duplicates are possible.
• QoS 2: “exactly once” in theory—but many managed hubs don’t support it.

At scale, QoS 1 + idempotency wins:

• Add a message id / sequence id in payload.
• Make consumers dedupe safely.

3) Retained messages: use like “last known value,” not history

Retained messages are great for:

• last known device status,
• configuration snapshots,
• “current mode” values.

But they’re not a time-series database.

• Each retained topic consumes broker storage.
• At massive topic counts, retained misuse becomes a memory/disk problem.

4) Shared subscriptions: scale your consumers cleanly

If backend services need to process a high-rate topic stream, shared subscriptions let you run a consumer group-like pattern behind MQTT.
This is often the simplest way to scale processing without adding extra routing layers.

5) Connection churn: design for reconnect storms

At scale, devices don’t reconnect one-by-one—they reconnect in waves:

• power outages,
• cellular carrier resets,
• firmware rollouts,
• broker restarts.

Mitigations

• Randomized reconnect backoff with jitter (clients).
• Staggered fleet rollouts.
• Separate “control plane” topics from “telemetry plane.”

6) Observability: if you can’t measure it, you can’t scale it

Track, at minimum:

• connections (current + churn rate),
• publish rate and bytes/sec,
• subscription count and wildcard share,
• retained count and size,
• offline queue depth,
• auth failures,
• latency percentiles (p50/p95/p99).

Performance, cost, and security considerations

Performance planning: the three numbers to size first

1. Concurrent connections (peak, not average)
2. Peak publishes/sec (burst, not average)
3. Fanout factor (avg subscribers per publish)

Then validate with load tests that include:

• realistic payload sizes,
• realistic topic filters,
• reconnect churn,
• and your actual auth method (TLS certs, tokens, etc.).

Cost modeling: message size matters more than people expect

Managed services often bill or quota by message “chunks.”

• AWS IoT Core: payloads up to 128 KB; metered in 5 KB increments (e.g., 8 KB counts as two).
• Azure IoT Hub: quota math depends on tier; message size used for daily quota calculations is standardized (for many tiers, 4 KB is used as the baseline unit).

Practical takeaway:
If you can reduce average payload from 8 KB → 4 KB, you can materially reduce “billed message units” in some pricing models.

Security: MQTT is secure when you operate it securely

Core must-haves:

• TLS everywhere (prefer mutual TLS for devices).
• Strong device identity + rotation.
• Least-privilege topic ACLs (devices can publish only to their own topics).
• Rate limits and anomaly detection (auth failures, reconnect spikes).
• Regular security testing (OWASP IoT projects are a good baseline).

Real-world use cases (mini case study)

Scenario: A telemetry platform grows from 15k to 180k devices across 12 regions.

Before

• Single broker deployment.
• One “global telemetry” subscription consumed by a single analytics service.
• Retained messages used for both “status” and pseudo-history.

What broke

• Analytics consumer became the bottleneck.
• Fanout spikes during dashboard usage caused latency jumps.
• Retained store grew unpredictably.

After

• Split topic tree by tenant/site/device; moved dashboards to read from a state store (not retained-as-history).
• Introduced shared subscriptions for backend processors to scale horizontally.
• Added reconnect jitter, rate limiting, and operational SLO dashboards.

Result (typical outcomes you should aim for)

• Predictable latency under bursts (no “mystery” spikes).
• Linear scaling of consumers (add replicas instead of redesigning).
• Lower storage volatility by narrowing retained usage.

‍

FAQs

1) What does “MQTT at scale” mean?

It means you can handle peak connections, throughput, fanout, and session state—reliably—while still being able to operate upgrades, failures, and security policies without outages.

2) How many MQTT connections can a broker handle?

It depends on broker, hardware, payload rates, and session settings. Vendors publish high-scale benchmarks (e.g., HiveMQ’s large-scale connection tests and EMQX’s WebSocket scale reports), but you should validate with your own workload shape.

3) What’s the best MQTT broker for large IoT fleets?

Start from your constraints:

• If you want fully managed identity + integration, consider managed hubs (AWS IoT Core / Azure IoT Hub).
• If you need full protocol control, custom routing, or on-prem, consider clustered brokers (EMQX, HiveMQ, VerneMQ).
  Use a load test that matches your fanout and reconnect pattern before deciding.

4) Do I need QoS 2 for IoT?

Often, no. Many managed IoT hubs don’t support QoS 2, so production designs commonly use QoS 1 plus idempotent processing.

5) How do shared subscriptions work in MQTT 5?

Multiple clients share one subscription group so the broker distributes matching messages across them—like a consumer group—improving throughput and fault tolerance for backend processing.

6) Are retained messages bad at scale?

They’re not “bad,” but they’re easy to misuse. Retained messages are perfect for “last known state,” but using them as history can blow up broker storage and complicate ops.

7) How do I load test MQTT?

Use a benchmark tool and simulate:

• realistic payload sizes,
• topic trees and wildcard patterns,
• QoS and session settings,
• reconnect churn,
• auth (TLS, certs, tokens).

Many broker ecosystems provide benchmark tools/guides (e.g., EMQX’s eMQTT-Bench documentation).

8) Is MQTT secure over the internet?

Yes—when you use TLS, strong identity, topic ACLs, and monitoring. Also treat IoT security like a system: test regularly and follow IoT security testing guidance.

‍