American Canyon • 50 milesRemote · Hybrid · Onsite · Field · All Environments
American Canyon • 50 milesRemote · Hybrid · Onsite · Field · All Environments
American Canyon • 50 milesRemote · Hybrid · Onsite · Field · All Environments
American Canyon • 50 milesRemote · Hybrid · Onsite · Field · All Environments
159 ai evaluation engineer jobs at 108 companies in American Canyon, CA
2mo
Save
Mark Applied
Hide
AI Engineer, Evaluation
San Francisco or New York City
$150k-$250k/yr HybridFull Time
2+ YOE2+ years software engineering, strong Python, experience with evaluation- or experiment-driven development, ability to encode human judgment into tests/graders, systems-oriented mindset, and willingness to travel 10–50%.
Python, LLM
1w
Save
Mark Applied
Hide
Sr. Evaluation Engineer
San Francisco, California, United States
$158k-$218k/yr HybridFull Time
5+ YOE5+ years in software engineering, machine learning, or applied AI; strong Python skills; experience with AI evaluation, LLMs, agents, retrieval-augmented generation, testing, monitoring, and CI/CD.
Python, LangSmith, Arize Phoenix, Braintrust, DeepEval, Ragas, TruLens, OpenAI Evals, MLflow, CI/CD
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
4+ YOEBachelor's plus 6 years or master's plus 4 years in AI/ML development; 6 years programming in Python, Go, Scala, or Java; experience building scalable AI systems and leading engineering teams.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, Google Cloud, Azure, C++, C#, Golang
2w
Save
Mark Applied
Hide
Software Engineer, AI Evaluation
San Francisco, California, United States
$148k-$232k/yr HybridFull Time
Significant production systems experience; expertise evaluating AI systems, agentic tooling, adversarial testing, statistics, experimental design, and functional UI development for non-engineers.
LangSmith, Braintrust, DeepEval, Ragas, Promptfoo
1w
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
OnsiteFull Time
Experience using AI tools, working with AI models or agents, and building evaluation systems, post-training, or context engineering; strong first-principles problem-solving skills.
1mo
Save
Mark Applied
Hide
Senior AI Engineer — Agents
San Francisco, California, United States
HybridFull Time
Experience building and shipping LLM-powered systems or agents, strong AI fluency (prompting, retrieval, evaluation), production monitoring, and solid software engineering practices.
Python, TypeScript, FastAPI, Next.js, Postgres
1mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
HybridFull Time
2+ YOE2+ years professional software/ML or applied AI experience, strong software engineering, experience with language models/AI APIs, backend and data pipeline skills, ability to evaluate models and work with healthcare workflows.
Python
2mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
$171k-$242k/yr HybridFull Time
2+ YOE2+ years building and deploying AI/ML applications, hands-on LLM experience, backend engineering with Python, API development (FastAPI/Django), cloud container deployment, R&D on model capabilities, and evaluation tooling experience.
Langfuse, LangSmith, FastAPI, Django, AWS, GCP, Azure, Terraform, Pulumi, GitHub Actions, React, Python, OpenAI, Anthropic, Google
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
6+ YOEBachelor's degree plus 6 years or master's degree plus 4 years in AI/ML; 6 years programming with Python, Go, Scala, or Java; cloud AI deployment and team leadership preferred.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, AWS, Google Cloud, Azure, C++, C#, Golang
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or Cambridge or San Jose or New York City
$230k-$286k/yr OnsiteFull Time
6+ YOEBachelor's plus 6+ years (or Master's plus 4+ years) developing AI/ML systems; strong programming (Python, Go, Scala, Java); cloud AI deployment experience; leadership and research application skills.
AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, C++, C#, Golang, AWS, Google Cloud, Azure
1mo
Save
Mark Applied
Hide
Founding Engineer (Applied AI)
San Francisco, California, United States
OnsiteFull Time
Experience building document ingestion/extraction, agentic workflows, and evaluation harnesses; familiarity with AI coding tools and working closely with customers.
Claude Code, Cursor
3mo
Save
Mark Applied
Hide
AI Engineer
San Francisco, California, United States
$180k-$250k/yr OnsiteFull Time
Hands-on experience building LLM-powered features; production engineering in TypeScript/Node and/or Python; experience with multiple LLM providers; strong prompt engineering, retrieval and evaluation; track record of evaluation systems; solid software engineering.
TypeScript, Node.js, Python, AWS Lambda, ECS Fargate, OpenAI, Anthropic, Google Vertex, AWS Bedrock, pgvector
4d
Save
Mark Applied
Hide
AI Engineer
Waterloo or Waterloo or Kitchener or Toronto or New York City or San Francisco
HybridFull Time
Experience with Python web development; shipped AI features; hands-on LLMs or generative AI; evaluation frameworks; vector databases; retrieval architectures; and model cost, latency, and accuracy tradeoffs.
Python, Django, LLMs, Retrieval-Augmented Generation (RAG), vector databases
1w
Save
Mark Applied
Hide
Applied AI Engineer
Canada or San Francisco or Raleigh or Boston
HybridFull Time
Applied GenAI experience with prompt engineering, structured outputs, RAG, evaluation datasets, production LLM features, multimodal inputs, and model quality, cost, and latency optimization.
LLMX, Attachments, OCR, RAG
1mo
Save
Mark Applied
Hide
Senior Software Engineer, AI Infrastructure - LVM Inference & Evaluation
Redwood City, California, United States
$168k-$205k/yr HybridFull Time
4+ YOE4+ years building infrastructure or production AI systems; strong Python; experience with ML infrastructure, LLM/LVM inference, inference optimization, evaluation frameworks, cloud and GPU workloads; BS/MS or equivalent.
Python, vLLM, Triton Inference Server, CUDA, NCCL, PyTorch, TensorRT, ONNX
2mo
Save
Mark Applied
Hide
Senior AI Engineer
San Francisco or New York City or Vancouver or Amsterdam or London or Paris or Singapore or Tokyo or Colorado or Washington
$211k-$316k/yr HybridFull Time
3+ YOERequires 3+ years of software engineering experience, including 2+ years building production LLM or applied AI systems, plus experience scaling AI products and familiarity with model APIs, agents, retrieval, prompting, tool use, and evaluation.
LLM, model APIs, agent frameworks, retrieval, prompting
1mo
Save
Mark Applied
Hide
Applied AI Engineer, GTM Growth Engineering
San Francisco, California, United States
$230k-$385k/yr OnsiteFull Time
4+ YOE4+ years building reliable production systems, experience with AI agents/LLM applications, evaluation and experimentation, strong backend engineering (Python, APIs, data pipelines), and cross-functional collaboration.
Python, APIs, CRM
5d
Save
Mark Applied
Hide
Founding AI Engineer
San Francisco or United States or Miami
HybridFull Time
Production experience deploying applied LLM systems to real users, strong AI judgment, expertise in evaluations and guardrails, and ability to build and lead an AI team.
Anthropic, AWS Bedrock, Python, Burr, LangGraph, ReAct
2mo
Save
Mark Applied
Hide
AI Agent Engineer
Charlotte or San Francisco or Los Angeles or New York City
$95k-$120k/yr OnsiteFull Time
4+ YOEBachelor's degree or equivalent experience, 4+ years in MLOps/AI/ML engineering or data engineering, hands-on Databricks and MLflow experience, strong Python and SQL skills, experience with agent frameworks and evaluation/monitoring of non-deterministic AI systems.
Databricks, Mosaic AI, Databricks Agent Framework, Databricks Vector Search, MLflow, Python, LangChain, LlamaIndex, AutoGen, SQL, Delta Lake, Unity Catalog, CI/CD, Sigma, Microsoft Power BI, Streamlit, Gradio
3mo
Save
Mark Applied
Hide
Senior Applied AI Engineer
San Francisco, California, United States
OnsiteFull Time
5+ YOE5+ years in software/ML engineering; strong Python; production services; LLMs experience; model evaluation and monitoring; PyTorch or similar.
Python, PyTorch, LLMs, Machine Learning Pipelines, APIs