About the Company
Trialing Health aggregates clinical trial metadata to match patients with ongoing trials across European hospitals.
The platform operates under strict accuracy requirements, transforming heterogeneous hospital records into clean, standardized schemas.
My Journey There
Trialing aggregates clinical trial data from hundreds of hospitals across Spain and Portugal. The core challenge was data heterogeneity: medical records arrived in wildly different formats, and bad data would instantly compromise trial matching for physicians.
I migrated the infrastructure to an AWS S3 Data Lake orchestrated by Airflow. To keep the database clean without dropping incoming files, I built a Quarantine Zone using Pydantic. Any record failing strict schema validation was automatically routed away from the main pipeline into a staging area for data managers to inspect.
By optimizing task partitioning and S3 bucket layouts, we cut end-to-end data processing latency from two hours down to 40 minutes.
Key Deliverables & Architectures
- [01]AWS S3 Data Lake & Pipeline Tuning: Transitioned clinical trial data infrastructure to an S3 Data Lake orchestrated by Airflow, reducing data-to-insight latency by 67% (120 to 40 minutes).
- [02]Pydantic Quarantine Zone: Built a multi-tier validation layer with a dedicated Quarantine Zone. Invalid hospital records are automatically isolated for manual data manager review rather than breaking the ETL pipeline.
- [03]Medical Data Structuring: Wrote custom ingestion logic (Regex, Pydantic) to parse heterogeneous, unstructured research archives from hospitals across Spain and Portugal into clean standardized schemas.