121 ai evaluation engineer jobs at 86 companies in Vacaville, CA
2mo
Save
Mark Applied
Hide
AI Engineer, Evaluation
San Francisco or New York City
$150k-$250k/yr HybridFull Time
2+ YOE2+ years software engineering, strong Python, experience with evaluation- or experiment-driven development, ability to encode human judgment into tests/graders, systems-oriented mindset, and willingness to travel 10–50%.
Python, LLM
1w
Save
Mark Applied
Hide
Sr. Evaluation Engineer
San Francisco, California, United States
$158k-$218k/yr HybridFull Time
5+ YOE5+ years in software engineering, machine learning, or applied AI; strong Python skills; experience with AI evaluation, LLMs, agents, retrieval-augmented generation, testing, monitoring, and CI/CD.
Python, LangSmith, Arize Phoenix, Braintrust, DeepEval, Ragas, TruLens, OpenAI Evals, MLflow, CI/CD
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
4+ YOEBachelor's plus 6 years or master's plus 4 years in AI/ML development; 6 years programming in Python, Go, Scala, or Java; experience building scalable AI systems and leading engineering teams.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, Google Cloud, Azure, C++, C#, Golang
1w
Save
Mark Applied
Hide
Software Engineer, AI Evaluation
San Francisco, California, United States
$148k-$232k/yr HybridFull Time
Significant production systems experience; expertise evaluating AI systems, agentic tooling, adversarial testing, statistics, experimental design, and functional UI development for non-engineers.
LangSmith, Braintrust, DeepEval, Ragas, Promptfoo
1w
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
OnsiteFull Time
Experience using AI tools, working with AI models or agents, and building evaluation systems, post-training, or context engineering; strong first-principles problem-solving skills.
1mo
Save
Mark Applied
Hide
Senior AI Engineer — Agents
San Francisco, California, United States
HybridFull Time
Experience building and shipping LLM-powered systems or agents, strong AI fluency (prompting, retrieval, evaluation), production monitoring, and solid software engineering practices.
Python, TypeScript, FastAPI, Next.js, Postgres
1mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
HybridFull Time
2+ YOE2+ years professional software/ML or applied AI experience, strong software engineering, experience with language models/AI APIs, backend and data pipeline skills, ability to evaluate models and work with healthcare workflows.
Python
2mo
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
$171k-$242k/yr HybridFull Time
2+ YOE2+ years building and deploying AI/ML applications, hands-on LLM experience, backend engineering with Python, API development (FastAPI/Django), cloud container deployment, R&D on model capabilities, and evaluation tooling experience.
Langfuse, LangSmith, FastAPI, Django, AWS, GCP, Azure, Terraform, Pulumi, GitHub Actions, React, Python, OpenAI, Anthropic, Google
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
6+ YOEBachelor's degree plus 6 years or master's degree plus 4 years in AI/ML; 6 years programming with Python, Go, Scala, or Java; cloud AI deployment and team leadership preferred.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, AWS, Google Cloud, Azure, C++, C#, Golang
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or Cambridge or San Jose or New York City
$230k-$286k/yr OnsiteFull Time
6+ YOEBachelor's plus 6+ years (or Master's plus 4+ years) developing AI/ML systems; strong programming (Python, Go, Scala, Java); cloud AI deployment experience; leadership and research application skills.
AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, C++, C#, Golang, AWS, Google Cloud, Azure
4w
Save
Mark Applied
Hide
Founding Engineer (Applied AI)
San Francisco, California, United States
OnsiteFull Time
Experience building document ingestion/extraction, agentic workflows, and evaluation harnesses; familiarity with AI coding tools and working closely with customers.
Claude Code, Cursor
1w
Save
Mark Applied
Hide
Applied AI Engineer
Canada or San Francisco or Raleigh or Boston
HybridFull Time
Applied GenAI experience with prompt engineering, structured outputs, RAG, evaluation datasets, production LLM features, multimodal inputs, and model quality, cost, and latency optimization.
LLMX, Attachments, OCR, RAG
3mo
Save
Mark Applied
Hide
AI Engineer
San Francisco, California, United States
$180k-$250k/yr OnsiteFull Time
Hands-on experience building LLM-powered features; production engineering in TypeScript/Node and/or Python; experience with multiple LLM providers; strong prompt engineering, retrieval and evaluation; track record of evaluation systems; solid software engineering.
TypeScript, Node.js, Python, AWS Lambda, ECS Fargate, OpenAI, Anthropic, Google Vertex, AWS Bedrock, pgvector
1d
Save
Mark Applied
Hide
AI Engineer
Waterloo or Waterloo or Kitchener or Toronto or New York City or San Francisco
HybridFull Time
Experience with Python web development; shipped AI features; hands-on LLMs or generative AI; evaluation frameworks; vector databases; retrieval architectures; and model cost, latency, and accuracy tradeoffs.
Python, Django, LLMs, Retrieval-Augmented Generation (RAG), vector databases
1mo
Save
Mark Applied
Hide
Applied AI Engineer, GTM Growth Engineering
San Francisco, California, United States
$230k-$385k/yr OnsiteFull Time
4+ YOE4+ years building reliable production systems, experience with AI agents/LLM applications, evaluation and experimentation, strong backend engineering (Python, APIs, data pipelines), and cross-functional collaboration.
Python, APIs, CRM
3mo
Save
Mark Applied
Hide
Senior Applied AI Engineer
San Francisco, California, United States
OnsiteFull Time
5+ YOE5+ years in software/ML engineering; strong Python; production services; LLMs experience; model evaluation and monitoring; PyTorch or similar.
Python, PyTorch, LLMs, Machine Learning Pipelines, APIs
1d
Save
Mark Applied
Hide
AI Engineer
Washington, D.C. or Atlanta or San Francisco
$223k-$259k/yr RemoteFull Time
5+ YOE5+ years of software engineering experience, including production LLM systems. Requires agentic systems expertise, strong Python, developer tooling, evaluation methods, systems fundamentals, ownership, and communication skills.
Python, Pydantic, MCP, OpenTelemetry, vLLM, Rust, Swift, Kotlin, Xcode, iOS simulators, CLIs, CI/CD
2mo
Save
Mark Applied
Hide
AI Security & Control Engineer
London or San Francisco
£115k-£200k/yr OnsiteFull Time
5+ YOE5+ years hands-on security experience; threat modeling, red‑teaming, and attack design for software/AI systems; application/cloud/product security; able to read code, evaluate infra and controls, and produce clear security artifacts.
MITRE ATT&CK, SIEM, DLP, CSPM, CI/CD, Watcher, Claude Code, Cursor, Codex, AppSec
1mo
Save
Mark Applied
Hide
AI Engineer
San Francisco, California, United States
OnsiteFull Time
2+ YOE2+ years industry or research experience in software engineering/ML, production-quality coding in Python, familiarity with ML frameworks, vector stores, hybrid search, and databases; experience with model fine-tuning and evaluation pipelines preferred.
Python, ML frameworks, vector stores, hybrid search, databases
4d
Save
Mark Applied
Hide
Applied AI Engineer
San Francisco, California, United States
$100k-$200k/yr OnsiteFull Time
3+ YOE3+ years in applied ML/AI engineering or a related master's degree, with production LLM experience, open-source model fine-tuning, evaluation frameworks, Python, and modern ML tooling.
Llama, Qwen, Mistral, Fireworks AI, Baseten, Together AI, Python, PyTorch, Hugging Face, LoRA, PEFT, DPO, RLHF, RAG, HIPAA
Explore Jobs
Expand Your Job Search
Top Cities Near Vacaville for AI Evaluation Engineer
- AI Evaluation Engineer jobs in San Francisco, CA
- AI Evaluation Engineer jobs in Sacramento, CA
- AI Evaluation Engineer jobs in Oakland, CA
- AI Evaluation Engineer jobs in Stockton, CA
- AI Evaluation Engineer jobs in Santa Rosa, CA
- AI Evaluation Engineer jobs in Elk Grove, CA
- AI Evaluation Engineer jobs in Hayward, CA
- AI Evaluation Engineer jobs in Roseville, CA
- AI Evaluation Engineer jobs in Concord, CA
- AI Evaluation Engineer jobs in Vallejo, CA
- AI Evaluation Engineer jobs in Berkeley, CA
- AI Evaluation Engineer jobs in Fairfield, CA
- AI Evaluation Engineer jobs in Antioch, CA
- AI Evaluation Engineer jobs in Richmond, CA
- AI Evaluation Engineer jobs in San Leandro, CA
- AI Evaluation Engineer jobs in Livermore, CA
- AI Evaluation Engineer jobs in Citrus Heights, CA
- AI Evaluation Engineer jobs in Napa, CA
- AI Evaluation Engineer jobs in Pleasanton, CA
- AI Evaluation Engineer jobs in Alameda, CA