The goal was to land raw S3 uploads in Snowflake within minutes using a fully serverless ETL pipeline built on S3 events, AWS Lambda, SQS, and Snowpipe.
I’m driven by lean, pay‑as‑you‑go data flows that stay secure yet scale on demand—this build shows how AWS and Snowflake can cut ingestion lag and simplify ops.
Results
I designed a star‑schema warehouse and wired S3 event triggers to a Python Lambda that cleans each file, stages it, and queues Snowpipe for auto‑ingest. The stack now processes new data in under one minute, with zero servers to patch and costs tied only to actual usage. IAM roles, environment secrets, and JWT checks keep data secure end‑to‑end.
Challenges
Packaging heavy Python libs for Lambda, tuning memory/timeouts, and orchestrating S3 → SQS → Snowpipe notifications required careful IaC tweaks. Balancing schema flexibility with query speed also meant iterating on partition keys and clustering in Snowflake. These hurdles sharpened my skills in serverless limits, dependency layering, and cost/performance trade‑offs.
