Lead Data Engineer
SecurityHQLondon, United Kingdom
- I led the migration from a sharded MongoDB cluster to ClickHouse Cloud, improving analytical query performance by over 1,000x and reducing annual platform costs by 75%.
- I designed a reusable Airflow ingestion framework and utility library. New pipelines reach production in 2 to 4 weeks, down from over a month.
- I mentor five data engineers through code reviews, Airflow DAG design, and database schema decisions.
- I built a production AWS Bedrock workflow with SOC analysts, using structured outputs and Airflow orchestration to reduce manual triage time by 40%.
- I established CI/CD for Airflow DAGs, Docker images, and Devcontainers with GitHub Actions, and use pytest for pipeline testing.
- I built semantic reconciliation with FAISS and Polars to map unstructured inputs to schemas and remove manual work.
- I partner with product managers and security analysts on delivery roadmaps, using proofs of concept to guide platform decisions.
- ClickHouse Cloud
- Apache Airflow
- AWS Bedrock
- Python
- Docker
- GitHub Actions
