Senior Data Engineer

As a Senior Data Engineer you will design, build and operate the data platform that powers the company: pipelines that ingest and transform very large datasets from external data partners, data models that the whole company queries, and services that make that data available to internal products. You will work on a team that empowers the other teams to use our huge amount of data efficiently. Using a large variety of technologies and tools, you will solve complicated technical problems and build solutions to make our pipelines robust and fault tolerant and our data easily accessible throughout the company.
This is primarily a batch and warehouse role: most of the work is large Spark and SQL pipelines, data modeling, and migrating pipelines from Redshift to Databricks. Streaming is a small part of the job.
Ukraine is mandatory. Our offices are located in Kyiv and Lviv. Teams are located in Kyiv and Lviv and primarily work remotely with occasional offline meetings.
Responsibilities
Design and build batch data pipelines that ingest, validate and transform multi-billion-row datasets from external data providers and internal systems
Model complex real-world data: dimensional models, reference data, and temporal data whose attributes change over time (e.g. slowly changing dimensions), and evolve those models safely as upstream sources change their schemas and semantics
Develop and operate workloads on our lakehouse platform (Databricks / Spark / Delta) and our data warehouse (Redshift), including migrating existing pipelines from the warehouse to the lakehouse
Orchestrate pipelines with Airflow: scheduling, dependencies, retries, backfills and alerting
Prove correctness, not just completion: design parity checks and reconciliation queries when replacing an existing pipeline, run large historical backfills, and investigate data discrepancies down to the row level
Build and maintain Python services and REST APIs that serve data to internal products
Optimize for performance and cost: query tuning, table design, workload management and right-sizing compute
Own what you ship: monitor production pipelines, participate in incident triage and root-cause analysis, and harden systems so the same failure does not happen twice
Collaborate cross-functionally with product managers, data scientists and stakeholders across the company to deliver on product roadmap
Work within an Agile team that releases cutting-edge new features regularly
Take a high degree of ownership and freedom to experiment with new technologies to improve our software
Qualifications
Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
7+ years of work experience as a data engineer
Python as your primary language for pipelines and services in recent years
3+ years of hands-on Spark (PySpark) in production batch pipelines: you wrote the jobs, deployed them through CI/CD and kept them running at scale
Production experience with Databricks: Jobs/Workflows, Delta Lake, and deploying through CI/CD. Notebook-only experience does not count. A strong candidate without Databricks must show deep production PySpark on AWS EMR instead
Expert-level SQL: writing, reading and tuning complex analytical queries against very large tables, and debugging why two result sets disagree
Amazon Redshift in production is strongly preferred. Otherwise, deep hands-on tuning of another MPP columnar warehouse: distribution and sort keys, workload management, and diagnosing slow queries from the query plan
Built and owned production Airflow DAGs: dependencies, retries, backfills and alerting
Ability to design complex data models: normalized, dimensional and temporal (slowly changing dimensions, effective-dated records, point-in-time correctness)
Experience integrating third-party data feeds: schema drift, late or missing deliveries, vendor data-quality defects and versioned reference data
Experience building REST services in Python (FastAPI, Flask, etc.)
Experience developing, maintaining and debugging large server-side code bases
Working knowledge of AWS (S3, IAM, ECS) and Docker
Good engineering practices: unit and integration tests, code review, CI/CD
The desire to take a high level of ownership, learn quickly, keep a high bar for quality and be pragmatic
Able to communicate effectively with U.S.-based teams and work 11:00 AM – 8:00 PM EEST (11:00–20:00)
Nice to Have
Experience migrating legacy pipelines between platforms with strict parity requirements
Experience with advertising, media or measurement industry data
Our Tech Stack
Almost everything we run is on AWS (S3, ECS, EMR, RDS and more)
Python is our primary language; SQL is everywhere
Databricks (Spark, Delta Lake) is our lakehouse platform; Redshift and Postgres are our warehouses and operational databases
Airflow orchestrates our pipelines
Docker for packaging; GitHub Actions and Jenkins for CI/CD
Grafana, Sentry and VictoriaLogs for observability
Datasets measured in billions of rows
Interview Process
Pre-screening (30 mins)
Technical Interview (1h): we will go deep on a Spark pipeline you built and ran in production
System Design (1.5h)
Product Interview (30 mins)
Offer.