HiringCafe
Join our community

152 ai evaluation engineer jobs at 111 companies in New York City, NY

5d
Save
Mark Applied
Hide
AI Evaluation Engineer (QA)
New York City or Austin or Miami or Dallas or Phoenix or São Paulo or Toronto
OnsiteFull Time, Contract
Appnovation
Appnovation: Full-service digital consultancy serving organizations through digital strategy, design, development, and support.
4+ YOEBachelor's degree in a technical field or equivalent experience, 4+ years in QA/test engineering, Python and data-science skills, LLM evaluation and statistical analysis experience, and automated test framework expertise.
Python, LLM evaluation frameworks, CI/CD, Test automation frameworks
Job PostingView all
2mo
Save
Mark Applied
Hide
AI Engineer, Evaluation
San Francisco or New York City
$150k-$250k/yr HybridFull Time
Distyl AI
Distyl AI: Architecting the AI-Native Enterprise
2+ YOE2+ years software engineering, strong Python, experience with evaluation- or experiment-driven development, ability to encode human judgment into tests/graders, systems-oriented mindset, and willingness to travel 10–50%.
Python, LLM
Job PostingView all
1mo
Save
Mark Applied
Hide
AI Solutions Engineer
Parsippany or Nashville
$140k-$170k/yr HybridFull Time
MHA
MHANYSE: ROP: Healthcare services and software and alternate-site GPO serving pharmacies, senior-care facilities, and home medical equipment providers.
5+ YOE5+ years software engineering experience; deep experience with agentic AI, prompt engineering, agent orchestration, CI/CD, Azure/M365, AI security, and evaluation frameworks; strong communication and cloud-native skills.
GitHub, Visual Studio Code, GitHub Copilot, Claude, Azure, M365, Azure DevOps, GitHub Actions, SonarQube, Playwright, Jira, Azure Monitor, Application Insights, Model Context Protocol (MCP)
Job PostingView all
2mo
Save
Mark Applied
Hide
Innovation Senior AI Engineer
Boston or Jersey City or New York City or Massachusetts or New York or New Jersey or United States or Poland
$210k-$260k/yr OnsiteFull Time
Brown Brothers Harriman
Brown Brothers Harriman: Global financial services firm and private partnership.
3+ YOE3+ years AI/ML engineering (5+ years product development), hands-on leader, expert Python, production LLM/RAG/agent experience, workflow orchestration, Azure/AWS, vector DBs, evaluation and responsible-AI practices.
Python, Agno, LangChain, LlamaIndex, Temporal, Prefect, Airflow, Azure, AWS, Pinecone, Weaviate, pgvector, LLM, RAG
Job PostingView all
1mo
Save
Mark Applied
Hide
Applied AI Engineer
New York, New York, United States
$220k-$300k/yr OnsiteFull Time
Santé
Santé: AI-powered operating-system serving liquor stores and wine shops with point-of-sale software.
5+ YOE5+ years building and shipping production AI products; strong software engineering with TypeScript and/or Python; full ML/LLM lifecycle experience; experiment design, evaluation, and monitoring; strong collaboration and communication.
TypeScript, Python, Next.js, tRPC, Prisma, PostgreSQL, pgvector, Vercel, Langfuse, Devin, Claude, Codex
Job PostingView all
3w
Save
Mark Applied
Hide
Lead AI Solutions Engineer
Bulgaria or Lithuania or Poland or Romania or New York City
HybridFull Time
Exadel
Exadel: Private software consulting and digital engineering firm helping enterprises build AI-enabled products, platforms, and data systems.
Hands-on Claude Code and software engineering experience; ability to create reusable AI-agent capabilities, design executable workflows, evaluate non-deterministic AI solutions, and communicate in upper-intermediate English.
Claude Code, MCP servers, promptfoo, LangSmith
Job PostingView all
1mo
Save
Mark Applied
Hide
Sr. Manager, Test Automation & AI Evaluation
Newark, New Jersey, United States
$98k-$135k/yr OnsiteFull Time
WebMD
WebMD: Private digital health publisher serving consumers, clinicians, employers, and health plans with medical information.
7+ YOE7+ years in quality engineering/test automation with leadership; hands-on Playwright/Cypress/Selenium; CI/CD integration; experience evaluating AI/LLM systems and defining test strategy.
Playwright, Cypress, Selenium
Job PostingView all
2mo
Save
Mark Applied
Hide
AI Engineer
Springfield or New York City or Boston
$141k-$185k/yr HybridFull Time
MassMutual
MassMutual: Mutual U.S. life insurer providing life insurance, protection, retirement, and investment services to individuals, families, and businesses.
4+ YOE4+ years in data science/AI engineering with production ML/LLM experience, Python and SQL proficiency, ability to build and evaluate AI solutions, and a Bachelor’s in a quantitative field (Master’s preferred).
LLMs, Bedrock AgentCore, AWS Strands, Azure, MCP/A2A protocols, Python, SQL, vector databases, Degreed
Job PostingView all
2mo
Save
Mark Applied
Hide
Staff AI Research Engineer
New York or Pittsburgh
$221k-$331k/yr OnsiteFull Time
Duolingo
DuolingoNasdaq: DUOL: Public U.S. education technology providing free and subscription-based language learning and English proficiency testing.
Proven AI research engineering experience across LLMs, bandits, and ML stack (feature engineering, training, fine-tuning, RL, evaluation, deployment, monitoring); leadership and mentoring experience; advanced degree or equivalent experience.
LLMs, bandit models, large-scale neural network models
Job PostingView all
5d
Save
Mark Applied
Hide
Staff AI/ML Engineer
New York City or Montreal
$230k-$280k/yr HybridFull Time
Tetrix
Tetrix: AI investment intelligence platform for institutional investors in private markets.
10+ YOE10+ years of engineering experience with recent production AI/ML focus, especially LLMs, agents, evaluations, or data extraction pipelines. Strong Python skills required; fintech experience valued.
Python, LLMs, PDFs, Excel
Job PostingView all
3w
Save
Mark Applied
Hide
AI Engineer
New York City, New York, United States
$200k/yr OnsiteFull Time
Optiver
Optiver: Global proprietary trading firm and market maker.
6+ YOEBachelors in CS/AI,6+ years software engineering,production Python experience,LLM/application deployment,model evaluation,agent orchestration,distributed systems,EM integration and strong communication.
Python, LLM, Model Context Protocol (MCP)
Job PostingView all
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or New York City or San Jose or Cambridge
$230k-$286k/yr OnsiteFull Time
Capital One
Capital OneNYSE: COF: A technology-driven bank providing diverse financial services.
6+ YOEBachelor's degree plus 6 years or master's degree plus 4 years in AI/ML; 6 years programming with Python, Go, Scala, or Java; cloud AI deployment and team leadership preferred.
AWS Ultraclusters, Hugging Face, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, AWS, Google Cloud, Azure, C++, C#, Golang
Job PostingView all
1mo
Save
Mark Applied
Hide
Senior Lead AI Engineer (SDK's: Gen AI Evaluation and MCP)
McLean or San Francisco or Cambridge or San Jose or New York City
$230k-$286k/yr OnsiteFull Time
Capital One
Capital OneNYSE: COF: A technology-driven bank providing diverse financial services.
6+ YOEBachelor's plus 6+ years (or Master's plus 4+ years) developing AI/ML systems; strong programming (Python, Go, Scala, Java); cloud AI deployment experience; leadership and research application skills.
AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, Python, Go, Scala, Java, C++, C#, Golang, AWS, Google Cloud, Azure
Job PostingView all
5d
Save
Mark Applied
Hide
AI Engineer
Waterloo or Waterloo or Kitchener or Toronto or New York City or San Francisco
HybridFull Time
Cambio
Cambio: AI-powered commercial real estate operations platform helping owners, investors, and operators manage data, compliance, retrofits, and decarbonization.
Experience with Python web development; shipped AI features; hands-on LLMs or generative AI; evaluation frameworks; vector databases; retrieval architectures; and model cost, latency, and accuracy tradeoffs.
Python, Django, LLMs, Retrieval-Augmented Generation (RAG), vector databases
Job PostingView all
3mo
Save
Mark Applied
Hide
AI Engineer
Plano or Purchase or Chicago
$94k-$156k/yr OnsiteFull Time
PepsiCo
PepsiCoNASDAQ: PEP: Multinational food and beverage.
6+ YOEBachelor in CS/AI/ML or equivalent; 6-8 years software lifecycle; ML/LLM/agent systems; Python/Java; production-grade AI delivery; strong evaluation and automation.
Python, Java, FastAPI, asyncio, Pydantic, LangChain, LangGraph, CrewAI, MLflow, Arise, Docker, Kubernetes, Ray, Kafka, gRPC, Redis, Prometheus, OpenTelemetry, Grafana
Job PostingView all
2mo
Save
Mark Applied
Hide
Senior AI Engineer
San Francisco or New York City or Vancouver or Amsterdam or London or Paris or Singapore or Tokyo or Colorado or Washington
$211k-$316k/yr HybridFull Time
Amplitude
AmplitudeNasdaq: AMPL: Public American software providing AI analytics tools that help businesses build better digital products and experiences.
3+ YOERequires 3+ years of software engineering experience, including 2+ years building production LLM or applied AI systems, plus experience scaling AI products and familiarity with model APIs, agents, retrieval, prompting, tool use, and evaluation.
LLM, model APIs, agent frameworks, retrieval, prompting
Job PostingView all
2w
Save
Mark Applied
Hide
Senior AI/ML Engineer
New York City, New York, United States
RemoteFull Time
Pypestream
Pypestream: Enterprise AI software serving large businesses with chat and voice agents that automate customer service end-to-end.
Build inference infrastructure, own model evaluation pipelines, optimize latency, and integrate orchestration engines with frontier large language models.
Job PostingView all
1w
Save
Mark Applied
Hide
AI-Native Engineer
New York City, New York, United States
OnsiteFull Time
Mirror Physics
Mirror Physics: Private New York AI building drug-discovery software and agents for pharmaceutical scientists.
Exceptional software engineering and system design skills; experience owning complex projects end to end; practical fluency building, deploying, evaluating, and operating modern AI systems; strong product judgment and communication.
AI, distributed compute, databases, data pipelines, backend services, user-facing applications, observability infrastructure, multi-agent systems, agent harnesses, inference infrastructure
Job PostingView all
2mo
Save
Mark Applied
Hide
AI Agent Engineer
Charlotte or San Francisco or Los Angeles or New York City
$95k-$120k/yr OnsiteFull Time
Scout Motors
Scout Motors: Reviving the iconic Scout brand with American-made electric vehicles.
4+ YOEBachelor's degree or equivalent experience, 4+ years in MLOps/AI/ML engineering or data engineering, hands-on Databricks and MLflow experience, strong Python and SQL skills, experience with agent frameworks and evaluation/monitoring of non-deterministic AI systems.
Databricks, Mosaic AI, Databricks Agent Framework, Databricks Vector Search, MLflow, Python, LangChain, LlamaIndex, AutoGen, SQL, Delta Lake, Unity Catalog, CI/CD, Sigma, Microsoft Power BI, Streamlit, Gradio
Job PostingView all
1mo
Save
Mark Applied
Hide
AI/ML Engineer, RL Environments - Asset Hub
New York City, New York, United States
$140k-$200k/yr HybridFull Time
SimpleClosure
SimpleClosure: Technology-enabled business dissolution service managing legal, tax, operational, and administrative closure for founders.
4+ YOEBuilt RL environments or AI-training products; 4+ years engineering experience in RL/AI training or evaluation; strong Python, Docker, CI/test skills; familiarity with evals, verifiers, security and PII handling; BS/MS in CS/ML preferred.
Python, Docker, CI, Playwright, MCP
Job PostingView all

Explore Jobs

Expand Your Job Search