UX Data Scientist

This role is ideal for someone who combines statistical rigor with a strong builder mentality — someone energized by ambiguity, fluent in model behavior, and obsessed with what "good" actually means and how to prove it. You will set the standard for how agent experiences are designed, instrumented, and held to a quality bar at Sitetracker — hands-on, technically deep, and relentlessly connected to customer outcomes.

You will be embedded in a scrum team — designing, measuring, and delivering alongside Engineering and Design on a day-to-day basis. You will own day-to-day decisions for your experience area, mentor others on rigor and method, and ship to production. For someone who wants to build, measure, and elevate the craft around them, this is the role.

The Skills You'll Have
Act with Agency
Thrive in fast-moving environments and deliver high-impact outcomes at pace
Independently identify ambiguous challenges and take ownership of driving them to resolution
Champion new ideas, influence stakeholders, and rally support around strategic initiatives
Identify workflows where AI tools and emerging technologies accelerate delivery, and make
building the default way you work
Measurement, Experimentation, and Evaluation
Define what "good" means for an agentic experience and turn it into metrics, evals, and datasets that hold up under scrutiny
Design and analyze experiments — A/B tests, holdouts, shadow runs — with correct sizing, clean reads, and honest conclusions
Build offline and online evaluation systems for LLM behavior across correctness, grounding, safety, tone, and latency
Apply statistical and ML fluency to real, messy production behavior — distinguishing signal from noise and root cause from symptom
Agentic UX and Behavior Design
Turn complex enterprise workflows into agent experiences people can actually use without training
Design context architecture, decision paths, and communication patterns, and reason about how they shape behavior across the full input distribution
Work directly with customers — running pilots, mining usage, and iterating based on how they actually behave, including customers with complex needs or in delicate situations
Make tradeoff decisions across user needs, agent behavior, technical constraints, and business goals, and connect those decisions to commercial outcomes
Competitive & Market Awareness
Maintain running awareness of the competitive landscape across multiple verticals
Identify gaps and opportunities and produce analysis that connects to product and behavior decisions
Track market trends, adjacent products, and emerging tools, connecting what's happening in the market to what Sitetracker should be doing
Regularly contribute to competition and industry Slack channels with substantive observations

Within 90 Days, You'll:
* Collaborate regularly with product management, design, and engineering peers, and have met key stakeholders across the departments you work with

* Be on the phone with customers, learning their workflows, pain points, and how they use Sitetracker and Scout. Know the personas and subverticals

* Deliver work to your scrum team — pushing code, building instrumentation and evals, and demonstrating the builder standard

* Be able to confidently demo Sitetracker's core product in your area and give a solid cross-product demo

* Stand up baseline evals and quality tracking for your area in Watch, and establish what you don't yet measure

* Begin mentoring others and providing substantive feedback in reviews

Within 180 Days, You'll:
* Lead the successful delivery of high-impact agent experiences across complex enterprise workflows

* Own day-to-day decisions for your experience area independently, establishing leading indicators and making tradeoff decisions confidently

* Architect Scout graphs and design sophisticated AI interactions, owning a comprehensive eval suite and tracking quality over time

* Design and run experiments on real agent behavior, and run customer betas and previews independently — including with customers who have more complex needs or stronger opinions

* Actively mentor others and raise the bar through consistent, high-quality reviews of evals, experiments, and agent designs

* Contribute meaningfully to shared evaluation infrastructure and establish higher standards for quality measurement in your domain

Within 365 Days, You'll:
* Point to product successes you have driven that have directly contributed to commercial outcomes

* Be recognized as an expert in your domain — people across the org come to you when they need to understand your experience area or how its quality is measured

* Ship code confidently to production, build the instrumentation and evals that keep quality honest, and operate as a true peer with engineering

* Be trusted with the most complex customer situations and maintain relationships with key customers that you built and maintain on your own

* Shape direction across multiple projects with well-reasoned, evidence-backed positions that influence the roadmap

* Set the pace for delivery in your domain and make others faster by reducing ambiguity early