266 ai evaluation engineer jobs at 158 companies in California
2mo
Save
Mark Applied
Hide
AI Evaluation Engineer
Bucharest or Boulder or Boston or Chicago or Glendora or Richmond or London or Auckland
lei16k-lei19k/mo HybridFull Time
5+ YOEBachelor's in CS/Data/Math, 5+ years software engineering with focus on testing/evaluation or AI/ML tooling, strong Python, experience building evaluation pipelines and LLM eval concepts, familiarity with RAG and production data pipelines.
RAGAS, LangSmith, PromptFlow, Snowflake, Python, RAG
2mo
Save
Mark Applied
Hide
AI Engineer, Evaluation
San Francisco or New York City
$150k-$250k/yr HybridFull Time
2+ YOE2+ years software engineering, strong Python, experience with evaluation- or experiment-driven development, ability to encode human judgment into tests/graders, systems-oriented mindset, and willingness to travel 10–50%.
Python, LLM
2w
Save
Mark Applied
Hide
AI Engineer – Algorithm Evaluation & Agentic Systems
Sunnyvale, California, United States
OnsiteFull Time
Experience evaluating AI algorithms and designing agentic systems, with system-level reasoning, metric definition, engineering rigor, and clear technical communication.
1mo
Save
Mark Applied
Hide
Senior Director - AI Evaluation Platform
Santa Clara, California, United States
$279k-$488k/yr HybridFull Time
Proven leadership of large AI or platform engineering teams, deep expertise in AI evaluation methodologies, production evaluation platforms, strong communication, and hands-on experience with PyTorch or TensorFlow.
PyTorch, TensorFlow
1w
Save
Mark Applied
Hide
Sr. Evaluation Engineer
San Francisco, California, United States
$158k-$218k/yr HybridFull Time
5+ YOE5+ years in software engineering, machine learning, or applied AI; strong Python skills; experience with AI evaluation, LLMs, agents, retrieval-augmented generation, testing, monitoring, and CI/CD.
Python, LangSmith, Arize Phoenix, Braintrust, DeepEval, Ragas, TruLens, OpenAI Evals, MLflow, CI/CD
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
4+ YOEBachelor's plus 6 years or master's plus 4 years in AI/ML development; 6 years programming in Python, Go, Scala, or Java; experience building scalable AI systems and leading engineering teams.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, Google Cloud, Azure, C++, C#, Golang
1mo
Save
Mark Applied
Hide
AI Engineer, Agents and Evaluation
Torrance, California, United States
OnsiteFull Time
Proven production experience shipping agent or tool-use systems, experience self-hosting open-weights models on GPUs, strong orchestration/serving/evaluation skills, and habit of treating hallucination as engineering problem.
vLLM, TGI
1w
Save
Mark Applied
Hide
Software Engineer, AI Evaluation
San Francisco, California, United States
$148k-$232k/yr HybridFull Time
Significant production systems experience; expertise evaluating AI systems, agentic tooling, adversarial testing, statistics, experimental design, and functional UI development for non-engineers.
LangSmith, Braintrust, DeepEval, Ragas, Promptfoo
1w
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
OnsiteFull Time
Experience using AI tools, working with AI models or agents, and building evaluation systems, post-training, or context engineering; strong first-principles problem-solving skills.
1mo
Save
Mark Applied
Hide
Senior AI Engineer — Agents
San Francisco, California, United States
HybridFull Time
Experience building and shipping LLM-powered systems or agents, strong AI fluency (prompting, retrieval, evaluation), production monitoring, and solid software engineering practices.
Python, TypeScript, FastAPI, Next.js, Postgres
1mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
HybridFull Time
2+ YOE2+ years professional software/ML or applied AI experience, strong software engineering, experience with language models/AI APIs, backend and data pipeline skills, ability to evaluate models and work with healthcare workflows.
Python
2w
Save
Mark Applied
Hide
AI Engineer
Mountain View, California, United States
OnsiteFull Time
3+ YOERequires 3+ years of software engineering experience, production LLM systems experience, strong Python and/or TypeScript/Node skills, and experience with LLM APIs, agentic workflows, evaluation, and end-to-end services.
Claude, OpenAI, Python, TypeScript, Node
2mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
$171k-$242k/yr HybridFull Time
2+ YOE2+ years building and deploying AI/ML applications, hands-on LLM experience, backend engineering with Python, API development (FastAPI/Django), cloud container deployment, R&D on model capabilities, and evaluation tooling experience.
Langfuse, LangSmith, FastAPI, Django, AWS, GCP, Azure, Terraform, Pulumi, GitHub Actions, React, Python, OpenAI, Anthropic, Google
2mo
Save
Mark Applied
Hide
AI Engineer
San Diego, California, United States
$150k-$225k/yr OnsiteFull Time
2+ YOEBachelor's degree or equivalent practical experience, 2+ years of software engineering experience, applied AI experience, Python or Java, AWS, retrieval systems, AI orchestration, evaluation, and strong communication skills.
Python, Java, AWS, OpenSearch, Pinecone, Weaviate, Redis, pgvector, Azure AI Search, LangChain, LangGraph, LlamaIndex, Semantic Kernel, OpenAI Agents SDK, N8N, AWS Bedrock Agents, AWS Bedrock Knowledge Bases, LangSmith, Arize Phoenix, OpenTelemetry, Datadog, Splunk, New Relic, CloudWatch, Model Context Protocol (MCP)
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
6+ YOEBachelor's degree plus 6 years or master's degree plus 4 years in AI/ML; 6 years programming with Python, Go, Scala, or Java; cloud AI deployment and team leadership preferred.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, AWS, Google Cloud, Azure, C++, C#, Golang
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or Cambridge or San Jose or New York City
$230k-$286k/yr OnsiteFull Time
6+ YOEBachelor's plus 6+ years (or Master's plus 4+ years) developing AI/ML systems; strong programming (Python, Go, Scala, Java); cloud AI deployment experience; leadership and research application skills.
AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, C++, C#, Golang, AWS, Google Cloud, Azure
4w
Save
Mark Applied
Hide
Founding Engineer (Applied AI)
San Francisco, California, United States
OnsiteFull Time
Experience building document ingestion/extraction, agentic workflows, and evaluation harnesses; familiarity with AI coding tools and working closely with customers.
Claude Code, Cursor
1w
Save
Mark Applied
Hide
Applied AI Engineer
Canada or San Francisco or Raleigh or Boston
HybridFull Time
Applied GenAI experience with prompt engineering, structured outputs, RAG, evaluation datasets, production LLM features, multimodal inputs, and model quality, cost, and latency optimization.
LLMX, Attachments, OCR, RAG
3mo
Save
Mark Applied
Hide
Lead AI Engineer — Productivity Systems
Durham or Miami or Palo Alto or Washington
$12k-$18k/mo HybridFull Time
Proven experience shipping LLM-powered systems in production; strong prompt/context engineering, evaluation and guardrail development, and software engineering skills (Python/TypeScript/Clojure).
Python, TypeScript, Clojure, n8n, Zapier, AWS, Claude Code, Cursor, Copilot, Slack, Google Workspace, Jira, Terraform, RAG, MCP
3mo
Save
Mark Applied
Hide
AI Engineer
San Francisco, California, United States
$180k-$250k/yr OnsiteFull Time
Hands-on experience building LLM-powered features; production engineering in TypeScript/Node and/or Python; experience with multiple LLM providers; strong prompt engineering, retrieval and evaluation; track record of evaluation systems; solid software engineering.
TypeScript, Node.js, Python, AWS Lambda, ECS Fargate, OpenAI, Anthropic, Google Vertex, AWS Bedrock, pgvector
Explore Jobs
Expand Your Job Search
Top Cities in California for AI Evaluation Engineer
- AI Evaluation Engineer jobs in Los Angeles, CA
- AI Evaluation Engineer jobs in San Diego, CA
- AI Evaluation Engineer jobs in San Jose, CA
- AI Evaluation Engineer jobs in San Francisco, CA
- AI Evaluation Engineer jobs in Fresno, CA
- AI Evaluation Engineer jobs in Sacramento, CA
- AI Evaluation Engineer jobs in Long Beach, CA
- AI Evaluation Engineer jobs in Oakland, CA
- AI Evaluation Engineer jobs in Bakersfield, CA
- AI Evaluation Engineer jobs in Anaheim, CA
- AI Evaluation Engineer jobs in Santa Ana, CA
- AI Evaluation Engineer jobs in Riverside, CA
- AI Evaluation Engineer jobs in Stockton, CA
- AI Evaluation Engineer jobs in Chula Vista, CA
- AI Evaluation Engineer jobs in Irvine, CA
- AI Evaluation Engineer jobs in Fremont, CA
- AI Evaluation Engineer jobs in San Bernardino, CA
- AI Evaluation Engineer jobs in Fontana, CA
- AI Evaluation Engineer jobs in Modesto, CA
- AI Evaluation Engineer jobs in Oxnard, CA