249 ai evaluation engineer jobs at 139 companies in Los Gatos, CA
2mo
Save
Mark Applied
Hide
AI Engineer, Evaluation
San Francisco or New York City
$150k-$250k/yr HybridFull Time
2+ YOE2+ years software engineering, strong Python, experience with evaluation- or experiment-driven development, ability to encode human judgment into tests/graders, systems-oriented mindset, and willingness to travel 10–50%.
Python, LLM
2w
Save
Mark Applied
Hide
AI Engineer – Algorithm Evaluation & Agentic Systems
Sunnyvale, California, United States
OnsiteFull Time
Experience evaluating AI algorithms and designing agentic systems, with system-level reasoning, metric definition, engineering rigor, and clear technical communication.
1mo
Save
Mark Applied
Hide
Senior Director - AI Evaluation Platform
Santa Clara, California, United States
$279k-$488k/yr HybridFull Time
Proven leadership of large AI or platform engineering teams, deep expertise in AI evaluation methodologies, production evaluation platforms, strong communication, and hands-on experience with PyTorch or TensorFlow.
PyTorch, TensorFlow
1w
Save
Mark Applied
Hide
Sr. Evaluation Engineer
San Francisco, California, United States
$158k-$218k/yr HybridFull Time
5+ YOE5+ years in software engineering, machine learning, or applied AI; strong Python skills; experience with AI evaluation, LLMs, agents, retrieval-augmented generation, testing, monitoring, and CI/CD.
Python, LangSmith, Arize Phoenix, Braintrust, DeepEval, Ragas, TruLens, OpenAI Evals, MLflow, CI/CD
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
4+ YOEBachelor's plus 6 years or master's plus 4 years in AI/ML development; 6 years programming in Python, Go, Scala, or Java; experience building scalable AI systems and leading engineering teams.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, Google Cloud, Azure, C++, C#, Golang
2w
Save
Mark Applied
Hide
Software Engineer, AI Evaluation
San Francisco, California, United States
$148k-$232k/yr HybridFull Time
Significant production systems experience; expertise evaluating AI systems, agentic tooling, adversarial testing, statistics, experimental design, and functional UI development for non-engineers.
LangSmith, Braintrust, DeepEval, Ragas, Promptfoo
1w
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
OnsiteFull Time
Experience using AI tools, working with AI models or agents, and building evaluation systems, post-training, or context engineering; strong first-principles problem-solving skills.
1mo
Save
Mark Applied
Hide
Senior AI Engineer — Agents
San Francisco, California, United States
HybridFull Time
Experience building and shipping LLM-powered systems or agents, strong AI fluency (prompting, retrieval, evaluation), production monitoring, and solid software engineering practices.
Python, TypeScript, FastAPI, Next.js, Postgres
1mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
HybridFull Time
2+ YOE2+ years professional software/ML or applied AI experience, strong software engineering, experience with language models/AI APIs, backend and data pipeline skills, ability to evaluate models and work with healthcare workflows.
Python
2w
Save
Mark Applied
Hide
AI Engineer
Mountain View, California, United States
OnsiteFull Time
3+ YOERequires 3+ years of software engineering experience, production LLM systems experience, strong Python and/or TypeScript/Node skills, and experience with LLM APIs, agentic workflows, evaluation, and end-to-end services.
Claude, OpenAI, Python, TypeScript, Node
2mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
$171k-$242k/yr HybridFull Time
2+ YOE2+ years building and deploying AI/ML applications, hands-on LLM experience, backend engineering with Python, API development (FastAPI/Django), cloud container deployment, R&D on model capabilities, and evaluation tooling experience.
Langfuse, LangSmith, FastAPI, Django, AWS, GCP, Azure, Terraform, Pulumi, GitHub Actions, React, Python, OpenAI, Anthropic, Google
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
6+ YOEBachelor's degree plus 6 years or master's degree plus 4 years in AI/ML; 6 years programming with Python, Go, Scala, or Java; cloud AI deployment and team leadership preferred.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, AWS, Google Cloud, Azure, C++, C#, Golang
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or Cambridge or San Jose or New York City
$230k-$286k/yr OnsiteFull Time
6+ YOEBachelor's plus 6+ years (or Master's plus 4+ years) developing AI/ML systems; strong programming (Python, Go, Scala, Java); cloud AI deployment experience; leadership and research application skills.
AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, C++, C#, Golang, AWS, Google Cloud, Azure
1mo
Save
Mark Applied
Hide
Founding Engineer (Applied AI)
San Francisco, California, United States
OnsiteFull Time
Experience building document ingestion/extraction, agentic workflows, and evaluation harnesses; familiarity with AI coding tools and working closely with customers.
Claude Code, Cursor
1w
Save
Mark Applied
Hide
Applied AI Engineer
Canada or San Francisco or Raleigh or Boston
HybridFull Time
Applied GenAI experience with prompt engineering, structured outputs, RAG, evaluation datasets, production LLM features, multimodal inputs, and model quality, cost, and latency optimization.
LLMX, Attachments, OCR, RAG
3mo
Save
Mark Applied
Hide
AI Engineer
San Francisco, California, United States
$180k-$250k/yr OnsiteFull Time
Hands-on experience building LLM-powered features; production engineering in TypeScript/Node and/or Python; experience with multiple LLM providers; strong prompt engineering, retrieval and evaluation; track record of evaluation systems; solid software engineering.
TypeScript, Node.js, Python, AWS Lambda, ECS Fargate, OpenAI, Anthropic, Google Vertex, AWS Bedrock, pgvector
3mo
Save
Mark Applied
Hide
Lead AI Engineer — Productivity Systems
Durham or Miami or Palo Alto or Washington
$12k-$18k/mo HybridFull Time
Proven experience shipping LLM-powered systems in production; strong prompt/context engineering, evaluation and guardrail development, and software engineering skills (Python/TypeScript/Clojure).
Python, TypeScript, Clojure, n8n, Zapier, AWS, Claude Code, Cursor, Copilot, Slack, Google Workspace, Jira, Terraform, RAG, MCP
2d
Save
Mark Applied
Hide
AI Engineer
Waterloo or Waterloo or Kitchener or Toronto or New York City or San Francisco
HybridFull Time
Experience with Python web development; shipped AI features; hands-on LLMs or generative AI; evaluation frameworks; vector databases; retrieval architectures; and model cost, latency, and accuracy tradeoffs.
Python, Django, LLMs, Retrieval-Augmented Generation (RAG), vector databases
2w
Save
Mark Applied
Hide
AI Support Engineer
San Jose or Austin
$96k-$148k/yr OnsiteFull Time
Bachelor's degree in engineering, computer science, information systems, or equivalent; experience evaluating technical solutions; AI familiarity; analytical, troubleshooting, communication, and project leadership skills.
LLMs, RAG, AI agents, MCP
1mo
Save
Mark Applied
Hide
Senior Software Engineer, AI Infrastructure - LVM Inference & Evaluation
Redwood City, California, United States
$168k-$205k/yr HybridFull Time
4+ YOE4+ years building infrastructure or production AI systems; strong Python; experience with ML infrastructure, LLM/LVM inference, inference optimization, evaluation frameworks, cloud and GPU workloads; BS/MS or equivalent.
Python, vLLM, Triton Inference Server, CUDA, NCCL, PyTorch, TensorRT, ONNX