117 software engineering evaluator jobs at 83 companies in New York
8h
Save
Mark Applied
Hide
Remote JavaScript Developer
United States or New York City
$200-$300/hr RemoteContract
3+ YOEAt least 3 years of software engineering experience with systems, infrastructure, or backend development; expertise in listed programming languages, scalable production software, architecture, debugging, and code evaluation.
Python, C, C++, Rust, Go, Java, JavaScript, ReactJS
1mo
Save
Mark Applied
Hide
Senior Software Engineer, Legal — CoCounsel Forward Deployed Engineering
Frisco or New York City or Toronto or Ann Arbor or Eagan or San Francisco or Los Angeles or Irvine or McLean or Washington
$95k-$176k/yr HybridFull Time
5+ YOEJD or formal legal training (or equivalent experience),5+ years software engineering,proficiency in Python,AI/LLM evaluation and prompt engineering,ability to engage legal users and mentor engineers,experience delivering production-grade SaaS solutions.
Python
1mo
Save
Mark Applied
Hide
Senior Engineering Manager, Agent Context
New York City, New York, United States
$264k-$300k/yr HybridFull Time
8+ YOE3+ Mgmt8+ years software engineering experience with 3+ years managing engineers; experience with production search/retrieval or ML-serving systems; strong written communication and evaluation-driven quality practices.
OpenSearch
2w
Save
Mark Applied
Hide
Software Engineer, Integrations
Europe or London or San Francisco or New York City or United Kingdom or Toronto or Montreal or Paris or Berlin or Seoul
$150k-$270k/yr RemoteFull Time
Production software engineering experience, external API integration experience, agentic system evaluation or familiarity with evals, prompt engineering, tool-description tuning, and end-to-end ownership.
AI, APIs
4d
Save
Mark Applied
Hide
Simulation Engineering Manager
New York City, New York, United States
$280k-$425k/yr OnsiteFull Time
Requires prior software, machine-learning, or systems engineering experience; leadership of a small production AI, ML, data, or distributed-systems team; hands-on coding, architecture, evaluation, and operational expertise.
LLM, AI, ML, data, distributed system, multi-agent frameworks, model orchestration, synthetic data, feature stores
2d
Save
Mark Applied
Hide
Staff/Senior Software Engineer, Agent Engineering
New York City, New York, United States
$185k-$235k/yr OnsiteFull Time
Production experience shipping LLM or agent systems beyond prototypes, with expertise in orchestration, tool infrastructure, retrieval, memory, model runtimes, evaluation, tracing, security, and operational control.
Proxi, Helios Rapid Ontology System (H.R.O.S.), Microsoft
1w
Save
Mark Applied
Hide
Software Engineer - Applied AI
New York City or San Francisco
$170k-$225k/yr HybridFull Time
4+ YOERequires 4+ years of software engineering experience, experience shipping AI systems, prompt and context engineering, evaluation design, dependable probabilistic systems, strong communication, and willingness to learn trusts.
Go, Postgres, Temporal, GraphQL, Honeycomb, Braintrust, PostHog, Meticulous, Amp, CI
3mo
Save
Mark Applied
Hide
Backend Software Engineer - Saga
New York or Seattle or Arlington or Rosslyn
OnsiteFull Time
4+ YOE4+ years backend software engineering; experience with GenAI systems, vector databases, LLM integrations, and evaluation harnesses; strong problem solving and ownership.
GenAI, vector databases, LLM integrations, RAG architectures, tool-call frameworks, production-grade software
1mo
Save
Mark Applied
Hide
Principal Software Engineer
Redmond or Mountain View or New York
$143k-$275k/yr HybridFull Time
6+ YOEBachelor's in CS or related and 6+ years engineering experience (or equivalent), coding in .Net/Java/JavaScript/Rust/Python; experience with distributed systems, AI evaluation, and reliability engineering.
.Net, Java, JavaScript, Rust, Python
1mo
Save
Mark Applied
Hide
Software Engineer, ML Products
New York City, New York, United States
$175k-$275k/yr OnsiteFull Time
5+ YOE5+ years experience building ML/agentic systems, expertise in context engineering (RAG, token optimization), evaluation systems, and production-quality agent infrastructure.
3w
Save
Mark Applied
Hide
Software Engineer, Client Solutions
San Francisco or New York City
$150k-$170k/yr HybridFull Time
3+ YOE3+ years professional Python experience, experience with LLM evaluation, relational databases, APIs, ETL, strong communication and critical thinking; BS in CS/Engineering preferred.
Python, FastAPI, SQLAlchemy, Postgres, Docker, Kubernetes, AWS, GitHub Actions, React, FHIR, HL7, EDI
1mo
Save
Mark Applied
Hide
Staff Software Engineer, AI Platform
New York, New York, United States
$231k-$340k/yr OnsiteFull Time
8+ YOE8+ years building backend systems with 1+ year in AI/ML engineering, experience with model integration, platform/SDK development, technical leadership, and evaluation frameworks.
2w
Save
Mark Applied
Hide
Senior Software Engineer, Search AI Rater Quality
Washington, D.C. or New York
$174k-$252k/yr OnsiteFull Time
5+ YOEBachelor's degree or equivalent experience, 5 years of software development, search relevance or information retrieval evaluation, and exploratory data analysis using SQL, R, or Python.
SQL, R, Python, Large Language Models (LLMs)
1w
Save
Mark Applied
Hide
Software Engineer- Benchmarking
San Francisco or New York City
$160k-$210k/yr HybridFull Time
4+ YOERequires 4+ years building complex systems, strong Python, dataset preparation and maintenance, Docker, reproducible environments, and collaboration with researchers; AI evaluation experience is a plus.
Python, Docker, React, Next.js, Tailwind, GitHub, Slack, Notion, Linear, PyTorch, Hugging Face, Harbor, Terminal-Bench, Inspect
1mo
Save
Mark Applied
Hide
Sr. Software Engineer I, Evaluation Services
Chicago or California or Colorado or Florida or Georgia or Illinois or Indiana or Minnesota or Missouri or Montana or New Jersey or New York or North Carolina or Ohio or Oregon or Pennsylvania or South Carolina or Texas or Utah or Vermont or Virginia or Washington or Wisconsin or District of Columbia
$125k-$175k/yr HybridFull Time
5+ YOE5+ years software engineering with API development and partner integrations; strong REST/WebSocket/JSON/OAuth skills; proficiency in TypeScript/JavaScript or Python; excellent troubleshooting and communication.
REST APIs, WebSockets, JSON, OAuth, Postman, curl, browser developer tools, network analyzers, TypeScript, JavaScript, Python, React, Node.js, Swagger/OpenAPI, ReadMe
3mo
Save
Mark Applied
Hide
Senior Software Engineer - AI/ML
New York, New York, United States
$150k-$200k/yr OnsiteFull Time
Senior Python engineer with production ML/LLM experience; strong system design, evaluation pipelines, and ability to mentor; independent, product-focused.
Python, LLM, NLP, Retrieval, Embeddings, Ranking, System Design
2w
Save
Mark Applied
Hide
Head of AI Engineering at AIOS — Remote, $200-$400k/yr + equity
Austin or San Francisco or London or New York City or United States or United Kingdom
$200k-$400k/yr RemoteFull Time
8+ YOERequires 8+ years of software engineering, a technical bachelor's degree, production agent-building experience, systems expertise, evaluation design, safe consequential actions, model judgment, and small-team leadership.
AIOS Agent SDK, Macbook
1mo
Save
Mark Applied
Hide
Member of Technical Staff - Research Software Engineer - Safety Evaluations Infrastructure
New York City or San Francisco or London
OnsiteFull Time
Experience building secure, isolated infrastructure for sensitive model evaluations; strong Python skills; security engineering fundamentals; data pipeline and sandboxing experience.
Python
1w
Save
Mark Applied
Hide
Software Engineer, Applied AI
New York City, New York, United States
$170k-$300k/yr HybridFull Time
Production experience with LLMs or agents, strong backend fundamentals, model evaluation experience, systems thinking, and ability to debug real-world failures and iterate quickly.
React, TypeScript, Python, AWS, Aurora, Postgres, ECS, Fargate, Lambda, OpenSearch, Elasticsearch, Redis, Terraform, Datadog
6d
Save
Mark Applied
Hide
Software Development Engineer, Support Agent Intelligence and Evaluation
Seattle or New York City
$144k-$214k/yr OnsiteFull Time
3+ YOERequires 3+ years professional software development, 2+ years system design or architecture, 1+ year distributed software experience, object-oriented design, and proficiency in C#, C++, Java, Perl, or another programming language. Bachelor's preferred.
C#, C++, Java, Perl
Explore Jobs
Expand Your Job Search
Top Cities in New York for Software Engineering Evaluator
- Software Engineering Evaluator jobs in New York, NY
- Software Engineering Evaluator jobs in Brooklyn, NY
- Software Engineering Evaluator jobs in Queens, NY
- Software Engineering Evaluator jobs in Manhattan, NY
- Software Engineering Evaluator jobs in The Bronx, NY
- Software Engineering Evaluator jobs in Staten Island, NY
- Software Engineering Evaluator jobs in Buffalo, NY
- Software Engineering Evaluator jobs in Rochester, NY
- Software Engineering Evaluator jobs in Yonkers, NY
- Software Engineering Evaluator jobs in Syracuse, NY
- Software Engineering Evaluator jobs in Amherst, NY
- Software Engineering Evaluator jobs in Albany, NY
- Software Engineering Evaluator jobs in Greenburgh, NY
- Software Engineering Evaluator jobs in New Rochelle, NY
- Software Engineering Evaluator jobs in Cheektowaga, NY
- Software Engineering Evaluator jobs in Mount Vernon, NY
- Software Engineering Evaluator jobs in Schenectady, NY
- Software Engineering Evaluator jobs in Utica, NY
- Software Engineering Evaluator jobs in White Plains, NY
- Software Engineering Evaluator jobs in Clay, NY