I’ve spent 20+ years moving other people’s data at scale, and the pattern never changes: the notebook demo is easy; the job that has to run every night is where the real engineering lives. This page is the shop-floor version of that work — the Hadoop pipeline behind a cardiac-device analytics program, the big-data plumbing inside a Fortune 500 bank, satellite imagery crunched on Databricks and Spark, and a healthcare data lake I helped carry from AWS to Google Cloud. Diagrams, decisions, and a few scars.
Each one learned the hard way, on production systems with real volume behind them.
I put lakes on object storage — S3, ADLS, GCS — with open table formats: Apache Iceberg, Delta, columnar Parquet/ORC. The reason is boring and important: one copy of the data that SQL, Spark, and ML can all read. I settle partitioning, compaction, and schema evolution up front, because retrofitting any of them onto a full lake is misery.
Apache Spark, the Hadoop/MapReduce ecosystem, and Hive, run on Databricks, AWS EMR, AWS Glue, and Azure Synapse. My rule: tune the job that fires at 2 a.m. every night, not the one in the demo — that’s where the data skew, the shuffle spills, and the pager alerts actually live.
BigQuery, Amazon Redshift, Azure Synapse, and Athena/Trino straight over the lake. I model the marts and materialize the aggregates so a dashboard answers in seconds — I’ve watched too many teams accept “the report runs overnight” as a law of physics. It isn’t.
Azure Data Factory, AWS Step Functions, Luigi, Airflow-style DAGs, dbt-style modeling. The property I care about most is re-runnability: every stage idempotent, every failure recoverable, preprocessor and post-processor steps that pick up where they died. If it can’t be re-run half-asleep at 3 a.m., it isn’t finished.
I’ve done feature engineering across hundreds of thousands of features and billions of calculations, fanned out with Ray, Anyscale, and KEDA-driven Kubernetes. A workload that would never finish on one box finishes on a fleet — that’s the data layer sitting under my AI & ML work.
HIPAA-aware handling, lineage, encryption with KMS and Secret Manager — and partition and format choices made with the scan bill in mind. I’ve owned security on enough engagements to know the audit shows up either way; I’d rather design for it than explain to it.
Four architectures from my own whiteboards — a lakehouse, a distributed compute engine, a telemetry pipeline, and a cross-cloud move.
My default shape: sources land as immutable raw files in object storage, Spark and Glue conform them into open Iceberg/Parquet tables, modeled marts feed the warehouse, and BI and ML read the same governed layer. I learned to insist on the immutable raw zone the hard way — it’s the only thing that saves you when a transform turns out to be wrong three weeks later.
flowchart LR
SRC["Source systems -- apps, devices, files, APIs"] --> ING["Ingest -- batch and stream landing"]
ING --> RAW["Raw zone -- object storage, immutable Parquet"]
RAW --> PROC["Spark and Glue -- clean, conform, enrich"]
PROC --> LAKE["Curated lakehouse -- Iceberg and Delta tables"]
LAKE --> WH["Warehouse -- BigQuery, Redshift, Synapse"]
LAKE --> ML["ML and feature store"]
WH --> BI["BI and dashboards"]
A Fortune 500 industrial client needed to test up to 200,000 features across billions of calculations, and the honest single-box estimate was “never.” So I split it: a driver fans work onto a queue, KEDA scales runner pods against queue depth, Ray parallelizes the math inside the fleet, and results land in PostgreSQL where the best-correlated trends rank out. The queue was the design decision that mattered — everything else scales off its depth.
flowchart TD
USER["Data scientist -- ranking job"] --> DRV["Main driver -- reads config, writes DB"]
DRV --> Q["Work queue -- millions of feature jobs"]
Q --> KEDA["KEDA scaler -- watches queue depth"]
subgraph CL["Distributed cluster -- AWS EKS"]
KEDA -->|"scale 0 to N"| POD["Target and model runner pods"]
POD --> RAY["Ray head and workers -- parallel compute"]
end
POD --> DB[("PostgreSQL -- results")]
RAY --> DB
DB --> RANK["Ranked features -- best correlated trends"]
One of my favorite pipelines to have built. Sensors across the airframe stream vibration, engine, and flight data off each helicopter; I landed it on HDFS and crunched it in the Hadoop ecosystem with MapReduce and Hive into flight metrics, surfaced to customers on an interactive dashboard. The hard part was never the math — it was making raw sensor noise mean something to the person reading the screen.
flowchart LR
HELI["Helicopter -- onboard sensors"] --> TEL["Flight telemetry -- vibration, engine, position"]
TEL --> ING["Ingest -- collection and landing"]
ING --> HDFS["Hadoop HDFS -- raw flight data"]
HDFS --> PROC["MapReduce and Hive -- process and aggregate"]
PROC --> METRICS["Flight analytics -- health and performance metrics"]
METRICS --> DASH["Customer dashboard -- per-aircraft insights"]
On a national healthcare technology platform, I worked the mapping problem cloud to cloud: Redshift, EMR, Glue, Athena, and Iceberg/Parquet on the AWS side onto BigQuery, Dataflow, and Cloud Storage on Google’s, with FHIR/HL7 clinical data going through the Google Healthcare API. I leaned on AI-assisted analysis to speed the port, and put an abstraction layer over eventing so the platform wouldn’t be welded to either cloud again.
flowchart LR
subgraph AWS["AWS -- source"]
S3["S3 data lake -- Parquet and Iceberg"] --> EMR["EMR and Glue -- Spark ETL"]
EMR --> RS["Redshift warehouse"]
S3 --> ATH["Athena queries"]
end
S3 --> MIG["Migration -- AI-assisted mapping and validation"]
RS --> MIG
subgraph GCP["Google Cloud -- target"]
MIG --> GCS["Cloud Storage data lake"]
GCS --> DF["Dataflow -- transform"]
DF --> BQ["BigQuery warehouse"]
GCS --> HC["Healthcare API -- FHIR and HL7"]
end
Everything below is something I’ve run in production, not something I’ve only read about.
Six platforms — aviation, healthcare, banking, insurance, industrial. Every one of them made it to production.
The biggest program I’ve run: a 160-developer program, five teams — two onshore, three offshore — building the big-data processing pipeline for a heart-analytics device platform on the Hadoop ecosystem, .NET and SQL Server across an Azure/on-prem hybrid. My job was keeping five teams’ worth of device telemetry converging into one clinical analytics pipeline.
I was Business Solutions Architect on a 60+ person program, owning the architecture of a 10-million-user banking information system — big-data pipelines on Azure and AKS, with the security architecture threaded through every distributed service. Banks don’t forgive: the throughput and the audit trail both had to hold.
Solutions Architect leading a 50-developer team on a property-risk analytics platform: satellite imagery and geospatial data pushed through Databricks, Hive, and Spark with Azure Synapse and Data Factory. I stayed hands-on with the Python/Luigi preprocessor and post-processor stages that bracketed the big-data layer.
At the largest private company in the U.S., I built the distribution layer under a feature-ranking and forecasting engine — hundreds of millions of calculations over data-science models, parallelized with Ray and Anyscale, scaled with KEDA on AWS EKS. The models belonged to the data scientists; making them finish belonged to me.
The full write-upI built the analytics pipeline for helicopter flight data: telemetry off the airframe, collected and processed across the Hadoop ecosystem with .NET, Python, and SQL Server on Linux, then handed back to customers as per-aircraft health and performance dashboards. Sensor noise in, operational insight out.
On a national healthcare technology platform, I worked the AWS-to-GCP port of a clinical data lake — Redshift, EMR, Glue, Athena, and Iceberg/Parquet mapped onto BigQuery, Dataflow, and Cloud Storage, FHIR/HL7 data riding the Google Healthcare API, and AI-assisted tooling shortening the slog.
The full write-upLakehouse design, Spark and Hadoop pipelines, warehousing, distributed ML compute, cross-cloud migrations — I do this work hands-on, remote, Corp-to-Corp. 20+ years in, I still write the code.