Senior Data Engineer — San Francisco Bay Area, CA, US 9+ years · Streaming & AI platforms

Building data platforms that turn noise into signal — at streaming scale.

I'm Tanay Karmarkar — Senior Data Engineer at Meta (Menlo Park). I architect streaming data platforms and AI-native analytics that power trust & safety, ads, and real-time products for millions. Previously Staff Engineer at Delivery Hero, Data Platform lead at NewStore and Expedia.

Book a 30-min call View selected work

9+ Years in data engineering

$600K Annual infra savings delivered

45% Analytical inquiries automated

Tanay Karmarkar

Senior Data Engineer · Meta

[email protected] · San Francisco Bay Area, CA, US

01 — About

Data infrastructure is a product — reliability is the feature.

I build opinionated data platforms that let product and ML teams move fast without breaking trust. My sweet spot is the seam between streaming infrastructure, warehouse modeling, and AI workflows — where a well-designed pipeline saves millions and unlocks decisions that weren't possible before.

Over 9+ years I've led analytics and AI automation for Integrity at Meta, ran data for a multi-market Ads attribution engine across 7 brands / 60+ markets at Delivery Hero, and built the Data Mesh from zero to near-real-time at NewStore.

Streaming-first: Kafka / MSK / PubSub → Dataflow / Kinesis → BigQuery / Snowflake / Redshift
Cost-aware architecture — $600K/yr BigQuery savings, 19% Snowflake reduction
AI-native workflows: agent orchestration, evals, token & tool-call optimization

02 — Selected work LinkedIn —>

Meta · 2025 → Present

Semantic Agentic Platform — Integrity

Global Ads Attribution & Streaming Platform

Staff Engineer

Led data for a multi-market attribution engine (CPC, CPM, Offers) across 7 global brands and 60+ markets with location-based sharding and KV stores. Engineered click / impression tracking for ranking & bidding models. Deployed streaming platform for 90+ engineers — real-time budget exhaustion alerts & live feature engineering.

GCPBigQueryKafkaDataflowGKE

$600K / yr BigQuery savings — tribe-wide optimization 0 → 1 streaming platform · 90+ engineers onboarded

NewStore · 2021 — 2023

Data Mesh on AWS MSK

Senior Data Engineer

Proposed and implemented Data Mesh using AWS MSK, Kafka Connect & Debezium. Cut time-to-market for new data products from 4 weeks to <3 days (−90%). Onboarded domains to BigQuery for near-real-time freshness (<5 min vs 24h batch). Self-service ingestion via Debezium + dbt reduced central bottlenecks by 75%.

AWS MSKKafka ConnectDebeziumBigQuerydbtLooker

−90% time-to-market 24h → <5min freshness

Expedia · 2020 — 2021

Conversation Platform — Real-time Quality Monitoring

Data Engineer II

Built real-time quality pipeline (Jitter, MOS, RTT) via Twilio + Looker contributing to +25% self-service rate. Engineered Python SDK for dashboarding — eliminated multi-table JOINs for Type II dimensions. Rightsized Snowflake warehouses.

PythonSnowflakeTwilioLookerAirflow

94s → 40s queries −19% Snowflake cost ($0.5M / yr)

03 — Technical skills

Languages

Python — Advanced
SQL — Expert
Java — Intermediate
Go — Beginner

Cloud & Infra

GCP: BigQuery, Dataflow, GCS, Pub/Sub, GKE
AWS: Redshift, MSK, S3, Lambda, ECS, Kinesis
Terraform, Kubernetes, Docker
CI/CD · Incident Commander experience

Data Engineering

Kafka, Debezium, Kafka Connect, KSQL, MSK
Airflow, dbt, Snowflake, BigQuery
Data Mesh architecture
Real-time streaming & orchestration tuning

AI & Agents

Agent orchestration & monitoring
Prompt scaffolding & evals
Claude Code · MCP servers
Token & tool-call optimization

04 — Experience

03/2025 — Present
Menlo Park, CA

Senior Data Engineer — Meta Platforms Inc

Integrity Organization

+5% revenue for Meta Verified — led analytics workstream for business & creator verification, reducing platform harm across Facebook & Instagram.
−20% opex & +10pp CSAT — architected data foundation for AI models automating human review & premium support.
Built semantic agentic platform bridging warehouses, CDPs, CRMs, lakes, APIs & MCP servers — 45% of inquiries now autonomous.
AI-native workflows: −52% token usage, −60% wall-clock time, −35% tool calls.

04/2023 — 02/2025
Berlin, Germany

Staff Engineer, Data (Ads) — Delivery Hero SE

Ads Platform · 7 brands · 60+ markets

$600K/yr BigQuery savings — tribe-wide optimization initiative.
Led attribution engine (CPC, CPM, Offers) with standardized benchmarking, location sharding & IP-range KV stores.
Built foundational click/impression tracking for ML ranking & bidding.
0→1 streaming platform for 90+ engineers — real-time budget alerts & live feature engineering.
Mentored 5+ engineers to Senior IC; Incident Commander & CI/CD lead.

10/2021 — 03/2023
Berlin, Germany

Senior Data Engineer, Infrastructure — NewStore Inc

Data Mesh on AWS MSK + Kafka Connect/Debezium — 4 weeks → <3 days to ship data products.
Centralized BigQuery warehouse — 24h batch → <5 min near-real-time freshness.
Self-service ingestion (Debezium + dbt) — −75% central bottlenecks.
Built Data Platform team: hiring rubrics, roadmaps, vendor negotiations (Looker, Sisense).

03/2020 — 06/2021
Seattle, WA

Data Engineer II, Conversation Platform — Expedia Group

Python SDK for real-time dashboarding — eliminated Type II dimension JOINs; 94s → 40s queries.
−19% Snowflake cost ($0.5M/yr) via warehouse rightsizing.
Real-time quality monitoring (Jitter, MOS, RTT) with Twilio + Looker — +25% self-service rate.

02/2019 — 03/2020
Naperville, IL

Data Engineer II — Egen Solutions Inc

100% HIPAA-compliant PII isolation & de-identification pipeline (Airflow) for oncology datasets.
Data Vault modeling for cancer tissue sample analysis.

05 — Education

12/2018

MS in Business Analytics & Information Systems

University of South Florida

05/2017

BE in Electrical Engineering

Shri G.S. Institute of Technology and Science

06 — Blogs & Writing View all —>

Aug 2026 · 12 min Distributed Systems Core Concepts

WTH is Raft?

Leader elections, replicated state machines, and the safety rules that stop two leaders from committing different logs — with an interactive 5-node cluster.