While preparing for the AWS SAP-C02, many candidates get confused by streaming data ingestion architectures. In the real world, this is fundamentally a decision about trade-offs between operational resilience, managed service costs, and data loss risk (RPO). Let’s drill into a simulated scenario.
The Scenario #
AceTech Manufacturing operates a large-scale industrial IoT solution with over 10,000 sensors distributed across multiple factories. These sensors send telemetry data using MQTT to an on-premises Apache Kafka cluster. This Kafka cluster processes the streaming data and stores processed results in an Amazon S3 bucket for analytics.
Recently, the on-prem Kafka cluster suffered an unexpected outage. The downtime caused gaps and data loss in the telemetry pipeline, resulting in incomplete datasets and impacting operational decisions. The CTO has tasked the cloud architect team with designing a new solution fully on AWS to improve the system鈥檚 high availability, scalability, and data durability to prevent data loss in future incidents.
Key Requirements #
Design an AWS solution that ensures zero data loss during failures, scales seamlessly with sensor data volume, and minimizes operational overhead while keeping costs reasonable.
The Options #
- A) Deploy two Amazon EC2 instances hosting Kafka in an active-passive setup across two Availability Zones (AZs). Use Amazon Route 53 with failover routing to route sensors to the active Kafka broker鈥檚 domain name.
- B) Migrate the on-prem Kafka cluster to Amazon Managed Streaming for Apache Kafka (Amazon MSK). Place a Network Load Balancer (NLB) in front of MSK brokers with health checks, and configure sensors to send telemetry via the NLB.
- C) Deploy AWS IoT Core to ingest sensor data directly via MQTT, then stream the data to Amazon Kinesis Data Firehose. Use AWS Lambda for data transformation before storing output objects in Amazon S3.
- D) Deploy AWS IoT Core and set up an EC2 Kafka cluster managed by the team. Route sensor telemetry to AWS IoT Core, which forwards data to the EC2-hosted Kafka cluster for processing.
Correct Answer #
Option C
Step-by-Step Winning Logic #
Option C leverages fully managed services鈥擜WS IoT Core and Amazon Kinesis Data Firehose鈥攚hich provide inherent high availability, horizontal scalability, and durable storage. IoT Core handles device connectivity natively over MQTT, removing the complexity of running and maintaining Kafka brokers. Firehose buffers and batches data reliably before delivery to S3.
This combination eliminates the single point of failure that the on-prem Kafka cluster exhibited. The solution scales automatically as sensors grow, reducing risk of data loss during peak loads. Though managed services may incur higher recurring costs compared to EC2-hosted Kafka, the operational cost savings and risk reduction of data loss justify this trade-off, an essential FinOps consideration.
馃拵 Professional-Level Analysis #
This section breaks down the scenario from a professional exam perspective, focusing on constraints, trade-offs, and the decision signals used to eliminate incorrect options.
馃攼 Expert Deep Dive: Why Options Fail #
This walkthrough explains how the exam expects you to reason through the scenario step by step, highlighting the constraints and trade-offs that invalidate each incorrect option.
Prefer a quick walkthrough before diving deep?
[Video coming soon] This short walkthrough video explains the core scenario, the key trade-off being tested, and why the correct option stands out, so you can follow the deeper analysis with clarity.
馃攼 The Traps (Distractor Analysis) #
This section explains why each incorrect option looks reasonable at first glance, and the specific assumptions or constraints that ultimately make it fail.
The difference between the correct answer and the distractors comes down to one decision assumption most candidates overlook.
- Option A: Managing Kafka on EC2 with Route 53 failover is complex and does not guarantee zero data loss during failover. It requires manual patching, monitoring, and recovery, increasing operational risk and cost.
- Option B: Amazon MSK is a great managed Kafka solution but fronting MSK brokers with an NLB for MQTT-based sensor telemetry adds complexity and is not a native IoT ingestion path. Sensors must support Kafka protocols, which many MQTT devices do not.
- Option D: Combining AWS IoT Core with an EC2 Kafka cluster still puts the operational burden on the team to manage Kafka brokers, adding complexity and the same failure risks that motivated the redesign.
馃攼 The Solution Blueprint #
This blueprint visualizes the expected solution, showing how services interact and which architectural pattern the exam is testing.
Seeing the full solution end to end often makes the trade-offs鈥攁nd the failure points of simpler options鈥攊mmediately clear.
graph TD
Sensors["IoT Sensors (MQTT)"] --> AWSIoTCore[AWS IoT Core]
AWSIoTCore --> KinesisFirehose[Amazon Kinesis Data Firehose]
KinesisFirehose --> S3Bucket[Amazon S3 Storage]
KinesisFirehose --> Lambda["Lambda (Data Transformation)"]
Diagram Note: Sensor telemetry flows through AWS IoT Core using native MQTT, streams reliably via Kinesis Firehose with buffering and delivery guarantees, transformed by Lambda before landing in durable S3 storage.
馃攼 The Decision Matrix #
This matrix compares all options across cost, complexity, and operational impact, making the trade-offs explicit and the correct choice logically defensible.
At the professional level, the exam expects you to justify your choice by explicitly comparing cost, complexity, and operational impact.
| Option | Est. Complexity | Est. Monthly Cost | Pros | Cons |
|---|---|---|---|---|
| A) EC2 Kafka (Active-Passive) | High | Medium ($1000-$1500) | Control over Kafka version/config | Operational overhead, failover risks, single point of failure |
| B) Amazon MSK + NLB | Medium-High | High ($2000+) | Managed Kafka, reduced ops vs EC2 | Complex MQTT to Kafka integration, protocol mismatch |
| C) AWS IoT Core + Kinesis Firehose | Low | Medium-High ($1500-$2000) | Fully managed, scalable, near zero data loss, native MQTT ingestion | Slightly higher service fees, some vendor lock-in |
| D) IoT Core + EC2 Kafka | High | Medium ($1200+) | Use IoT Core device management + Kafka features | Maintains EC2 Kafka ops burden, failure risk persists |
馃攼 Real-World Practitioner Insight #
This section connects the exam scenario to real production environments, highlighting how similar decisions are made鈥攁nd often misjudged鈥攊n practice.
This is the kind of decision that frequently looks correct on paper, but creates long-term friction once deployed in production.
Exam Rule #
For the exam, always choose AWS IoT Core to ingest MQTT sensor data at scale with minimal operational complexity.
Real World #
In practice, hybrid setups may link IoT Core to downstream managed Kafka or analytics clusters for specialized processing, but removing single-host Kafka clusters is critical to operational resilience.