HiringCafe
Join our community

241 evaluations engineer jobs at 153 companies in Santa Rosa, CA

1mo
Save
Mark Applied
Hide
Evaluations Engineer
San Francisco, California, United States
$130k-$165k/yr OnsiteFull Time
Vals AI
Vals AI: Building enterprise benchmarks for evaluating LLM performance.
Strong engineering fundamentals, professional Python expertise, familiarity with LLMs, Git workflow experience, ability to analyze model error modes and work with cross-functional teams. In-person in San Francisco; relocation/transportation support provided.
Docent, Python, Django, React, AWS, CDK, Git
Job PostingView all
3mo
Save
Mark Applied
Hide
AI Evaluations Engineer
San Francisco, California, United States
$160k-$210k/yr RemoteFull Time
Ellipsis Health
Ellipsis Health: AI-powered voice agent for healthcare care management.
5+ YOE5+ years software engineering; backend systems; production infrastructure; CI/CD; observability; debugging; collaboration across teams.
Python, TypeScript, Java, CI/CD, APIs, Data Pipelines, Observability Tools
Job PostingView all
1mo
Save
Mark Applied
Hide
Evaluations Engineering - Member of Technical Staff
San Francisco, California, United States
$200k-$400k/yr OnsiteFull Time
Simile
Simile: Building AI simulations to predict human behavior at scale.
Several years building production backend and data systems, experience with evaluation pipelines, ML/LLM fluency, strong system design and communication skills.
Job PostingView all
4w
Save
Mark Applied
Hide
Member of Technical Staff - Research Software Engineer - Safety Evaluations Infrastructure
New York City or San Francisco or London
OnsiteFull Time
Reflection AI
Reflection AI: Developing open-source frontier artificial intelligence models and coding agents.
Experience building secure, isolated infrastructure for sensitive model evaluations; strong Python skills; security engineering fundamentals; data pipeline and sandboxing experience.
Python
Job PostingView all
3mo
Save
Mark Applied
Hide
Machine Learning Engineer, Model Evaluations (Speech LLM) - San Francisco
San Francisco, California, United States
$180k-$270k/yr HybridFull Time
Plaud
Plaud: Develops AI-powered voice recorders and automated transcription software.
Python software engineering; building distributed systems, data pipelines, and evaluation harnesses at scale; partner with ML researchers to define benchmarks; build dashboards and monitor model health; debug mid-training anomalies; communicate results clearly.
Python, Distributed systems, Data pipelines, Evaluation harnesses, Dashboards, Weighs & Biases, MLflow
Job PostingView all
2mo
Save
Mark Applied
Hide
Staff Software Engineer, Quantitative Evaluations
Mountain View or San Francisco
$251k-$310k/yr HybridFull Time
Waymo
Waymo: Autonomous driving technology for ride-hailing and logistics.
4+ YOEBS in a quantitative field; 4-7 years experience; strong C++, Python, SQL skills; data analysis; building data pipelines; ML familiarity; strong coding standards.
C++, Python, SQL, Machine Learning, Data Processing, Statistics
Job PostingView all
2mo
Save
Mark Applied
Hide
AI Engineer, Evaluation
San Francisco or New York City
$150k-$250k/yr HybridFull Time
Distyl AI
Distyl AI: Builds AI-native orchestration platforms for enterprise operations.
2+ YOE2+ years software engineering, strong Python, experience with evaluation- or experiment-driven development, ability to encode human judgment into tests/graders, systems-oriented mindset, and willingness to travel 10–50%.
Python, LLM
Job PostingView all
2w
Save
Mark Applied
Hide
Founding Research Engineer
New York City or San Francisco
$215k-$330k/yr OnsiteFull Time
Ambral
Ambral: AI-powered account managers for enterprise customer success teams.
4+ YOE4+ years building production software or ML systems, including 2+ years with RL environments or evaluation/agent infrastructure; strong software engineering, data processing, reproducibility, and evaluation skills.
Job PostingView all
2w
Save
Mark Applied
Hide
Forward Deployed Engineer
Mountain View or San Francisco
$160k-$220k/yr OnsiteFull Time
Granica
Granica: Provides an AI efficiency platform for optimizing enterprise training data.
5+ YOERequires 5+ years in sales engineering, solutions architecture, forward deployed or data engineering; expertise in data platforms, distributed systems, cloud infrastructure, SQL, and programming; enterprise evaluations and deployments experience.
Apache Spark, Apache Iceberg, Delta Lake, Trino, Presto, SQL, Large Tabular Models (LTMs)
Job PostingView all
3w
Save
Mark Applied
Hide
Senior AI Engineer — Agents
San Francisco, California, United States
HybridFull Time
Mira Mace
Mira Mace: AI-powered healthcare advocacy and navigation for Medicare beneficiaries.
Experience building and shipping LLM-powered systems or agents, strong AI fluency (prompting, retrieval, evaluation), production monitoring, and solid software engineering practices.
Python, TypeScript, FastAPI, Next.js, Postgres
Job PostingView all
1w
Save
Mark Applied
Hide
Forward Deployed Engineer
San Francisco, California, United States
$180k-$285k/yr HybridFull Time
Bunkerhill Health
Bunkerhill Health: AI-powered platform automating clinical reasoning and diagnostic workflows.
Strong engineering skills, clinical stakeholder communication, production debugging, AI agent development, evaluation building, systems integration, and ownership of customer outcomes are required.
Job PostingView all
4d
Save
Mark Applied
Hide
Applied AI Engineer
Canada or San Francisco or Raleigh or Boston
HybridFull Time
MaintainX
MaintainX: Provides mobile-first software for maintenance and asset management.
Applied GenAI experience with prompt engineering, structured outputs, RAG, evaluation datasets, production LLM features, multimodal inputs, and model quality, cost, and latency optimization.
LLMX, Attachments, OCR, RAG
Job PostingView all
2w
Save
Mark Applied
Hide
Senior ISP Engineer
San Francisco, California, United States
$175k-$250k/yr HybridFull Time
General Motors
General MotorsNYSE: GM: Manufactures and sells automobiles and automotive parts globally.
4+ YOEBachelor's, master's, or PhD in imaging science, electrical engineering, computer science, or similar; 4+ years of camera ISP experience; ISP tuning, sensor characterization, image quality evaluation, and C/C++ or Python proficiency.
Imatest, C, C++, Python
Job PostingView all
4d
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
OnsiteFull Time
Magic Patterns
Magic Patterns: AI-powered platform for rapid software prototyping and design.
Experience using AI tools, working with AI models or agents, and building evaluation systems, post-training, or context engineering; strong first-principles problem-solving skills.
Job PostingView all
4d
Save
Mark Applied
Hide
Forward Deployed Research Engineer
San Francisco or Singapore
$150k-$250k/yr OnsiteFull Time
Clera
Clera: AI talent agent matching professionals with high-growth startup roles
2+ YOE2–4 years in applied research or forward-deployed engineering; Python, Docker, and Linux proficiency; benchmark and evaluation experience; strong debugging, communication, and independent problem-solving skills.
Python, Docker, Linux
Job PostingView all
2w
Save
Mark Applied
Hide
Machine Learning Engineer
San Francisco, California, United States
$176k/yr HybridFull Time
Bardeen
Bardeen: AI-powered platform for automating browser-based workflows and tasks.
1+ YOEMaster's in computational science and engineering or related machine learning field plus 1 year of experience with deep learning, LLMs, Python, cloud infrastructure, evaluation, monitoring, and distributed inference.
Python, Hugging Face, OpenAI, Gemini, Anthropic, AWS, GCP, OpenAI SDK, Gemini SDK, Anthropic SDK, Hugging Face Transformers, GPT, Mistral, LLaMA
Job PostingView all
1mo
Save
Mark Applied
Hide
Early Access Deployment Engineer
San Francisco, California, United States
$250k-$330k/yr HybridFull Time
OpenAI
OpenAI: Develops artificial intelligence models and generative AI software services.
4+ YOE4+ years software engineering with customer-facing delivery; strong prototyping, production ownership; proficiency in Python and/or JavaScript/TypeScript; cloud deployment experience; practical AI/LLM and evaluation experience.
Python, JavaScript, TypeScript
Job PostingView all
3mo
Save
Mark Applied
Hide
Senior Applied AI Engineer
San Francisco, California, United States
OnsiteFull Time
Alterion
Alterion: Building a control plane for enterprise AI agents.
5+ YOE5+ years in software/ML engineering; strong Python; production services; LLMs experience; model evaluation and monitoring; PyTorch or similar.
Python, PyTorch, LLMs, Machine Learning Pipelines, APIs
Job PostingView all
1mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
HybridFull Time
Abby Care
Abby Care: Trains and pays family members to provide home care.
2+ YOE2+ years professional software/ML or applied AI experience, strong software engineering, experience with language models/AI APIs, backend and data pipeline skills, ability to evaluate models and work with healthcare workflows.
Python
Job PostingView all
2mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
$171k-$242k/yr HybridFull Time
Artos
Artos: AI-powered document authoring platform for life sciences R&D.
2+ YOE2+ years building and deploying AI/ML applications, hands-on LLM experience, backend engineering with Python, API development (FastAPI/Django), cloud container deployment, R&D on model capabilities, and evaluation tooling experience.
Langfuse, LangSmith, FastAPI, Django, AWS, GCP, Azure, Terraform, Pulumi, GitHub Actions, React, Python, OpenAI, Anthropic, Google
Job PostingView all

Explore Jobs

Expand Your Job Search