About the Company
Trialing Health is a high-assurance medical technology company that compiles clinical trial metadata to match patients with ongoing trials across European hospitals.
The platform operates under strict data accuracy constraints, aggregating heterogeneous research data into standardized schemas.
My Journey There
I spearheaded the transition of their infrastructure to a highly structured AWS S3 Data Lake, using Airflow for ETL orchestration.
I reduced the data ingestion-to-analytics latency by 67% (from 120 minutes down to 40 minutes) through workflow tuning and partitioned storage layouts.
To maintain high-quality data governance, I built a Quarantine Zone logic using Pydantic schemas that isolated non-standard records, enabling manual data stewardship before final warehousing.
Key Deliverables & Architectures
- [01]Infrastructure & Latency Optimization: Spearheaded the setup of an AWS S3-based Data Lake and optimized Airflow-orchestrated ETL pipelines, achieving a 67% reduction in data-to-insight latency (from 120 to 40 minutes).
- [02]Quality Assurance & Data Stewardship: Architected a multi-tier validation framework featuring a specialized Quarantine Zone for manual data verification; this ensured strict data integrity and facilitated seamless collaboration between data managers.
- [03]Complex Data Structuring: Engineered a robust ingestion engine using Regex and Pydantic to transform heterogeneous, non-standardized scientific records into a high-fidelity schema, enabling physicians to track drug efficacy and treatment outcomes in real-time.
- [04]Clinical Intelligence Centralization: Designed and deployed a unified medical data platform that aggregated unstructured research archives from hundreds of hospitals and clinical centers across Spain and Portugal.
Chronicles & Artifacts
AWS S3 Data Lake and Apache Airflow orchestrations validation zone routing logic.