HiringCafe
Join our community

24 batch data engineer jobs at 20 companies in San Francisco, CA

PromotedHiringCafe
Founding Machine Learning / AI Search Engineer
Cupertino, CA, US
$160k-$310k/yr On-SiteFull Time
HiringCafe
HiringCafe: Building a 100× better job search engine to take on Indeed and LinkedIn.
Build the ML and AI search behind HiringCafe — ranking, recommenders, retrieval, and LLM agents that surface jobs people would never find on their own.
Python, PyTorch, Elasticsearch, LLMs
Help us #BeatIndeed & #BeatLinkedIn
2d
Save
Mark Applied
Hide
Staff Data Engineer
Sunnyvale, California, United States
$200k-$260k/yr OnsiteFull Time
RADAR
RADAR: Real-time inventory tracking using RFID and computer vision technology.
8+ YOE8+ years data/analytics engineering experience, strong SQL and Python, experience with batch and streaming pipelines, data modeling, data quality, testing, monitoring, and mentoring engineers.
Airflow, Apache Beam, Python, SQL, Apache Spark, Dagster, dbt, Kafka Streams, Flink, Looker, Tableau, Git, Snowflake, Databricks, BigQuery, Docker
Job PostingView all
6d
Save
Mark Applied
Hide
Principal Data Engineer, People Data
Bellevue or Chicago or San Francisco or Washington
$197k-$271k/yr HybridFull Time
Okta
OktaNASDAQ: OKTA: Provide secure identity management and authentication for enterprises.
10+ YOE10+ years data engineering experience; expert SQL, ETL/ELT, MPP databases, AWS services; HR/People data experience; experience with batch and realtime pipelines, lakehouse architectures, and data quality frameworks.
SQL, Airflow, dbt, Snowflake, Redshift, Amazon S3, AWS Lambda, EMR, ECR, EKS, Workday, Spark, Kafka, Databricks, Iceberg, Delta
Job PostingView all
3d
Save
Mark Applied
Hide
Staff Data Engineer
San Francisco or New York
$185k-$195k/yr HybridFull Time
Butterfly Network
Butterfly NetworkNYSE: BFLY: Handheld whole-body ultrasound scanners powered by semiconductor technology.
5+ YOE5+ years data engineering experience, strong dbt and Databricks skills, Python/PySpark, streaming and batch ingestion, CDC pipelines, data quality and observability, and experience integrating with operational systems.
dbt, Databricks, Delta Lake, Delta Live Tables, Python, PySpark, Salesforce, NetSuite, Stripe, Mulesoft, Great Expectations, Unity Catalog, Airflow, Cloud Run, Cloud Functions, BigQuery, AWS, GCP, Terraform, Spacelift, Boomi
Job PostingView all
3mo
Save
Mark Applied
Hide
Senior Data Engineer
Denver or San Francisco or Nashville or Santiago
$157k-$218k/yr OnsiteFull Time
Checkr
Checkr: AI-powered platform for background checks and identity verification.
5+ YOE5+ years engineering experience building scalable data platforms; proficiency in Python, PySpark, SQL; experience with ETL/ELT, batch and stream processing; familiarity with Kafka, Spark, Iceberg, AWS services.
Python, PySpark, SQL, MongoDB, Kafka, Spark, Iceberg, AWS, EKS, EMR, Glue, Athena, S3, Databricks, Snowflake
Job PostingView all
1mo
Save
Mark Applied
Hide
Senior Data Engineer
San Francisco or New York City
HybridFull Time
Beast Industries
Beast Industries: Produces digital media and consumer goods for MrBeast brands.
8+ YOE8+ years building production data platforms, expertise in batch and streaming pipelines, data modeling, ETL/ELT, reliability, observability, and privacy-safe foundations for AI/ML.
Job PostingView all
1mo
Save
Mark Applied
Hide
Sr Data Engineer
Glendale or San Francisco or Seattle
$139k-$204k/yr HybridFull Time
The Walt Disney Company
The Walt Disney CompanyNYSE: DIS: Produces media content and operates global theme parks.
5+ YOE5+ years data engineering experience (1+ year senior/lead), building streaming and batch pipelines, strong Python/Java/SQL, experience with Kafka/Flink/Spark/Kinesis, embedding/vector stores and orchestration tools, bachelor's in CS or equivalent.
Python, Java, SQL, Kafka, Flink, Spark, Kinesis, Pinecone, Weaviate, FAISS, pgvector, Airflow, Dagster, LangChain, LangGraph, Datadog, Prometheus, Databricks, S3, Glue, MWAA
Job PostingView all
1w
Save
Mark Applied
Hide
Senior Staff Data Engineer
San Jose or Draper
$166k-$245k/yr HybridFull Time
BILL
BILLNYSE: BILL: Automated financial operations software for small and midsize businesses.
8+ YOELead architecture and delivery of data platform capabilities (ingest, lake, streaming, feature store, query, graph, search). Requires distributed systems, streaming and batch experience, SQL and Python, and technical leadership.
Starburst, Databricks Feature Store, Neo4j, OpenSearch, Kafka, Flink, Spark Streaming, Airflow, dbt, Spark, Glue, Apache Iceberg, Delta Lake, Trino, Presto, Python, SQL
Job PostingView all
3mo
Save
Mark Applied
Hide
Software Engineer - Data Infrastructure
San Francisco, California, United States
$160k-$225k/yr HybridFull Time
Fable Security
Fable Security: AI-driven platform managing human risk and employee security behavior.
3+ YOE3+ years in data infrastructure or platform engineering; strong real-time and batch processing knowledge; Spark, Kafka, dbt, Airflow; cloud (AWS/GCP); startup environment.
Spark, Kafka, dbt, Airflow, AWS, GCP
Job PostingView all
1w
Save
Mark Applied
Hide
Software Engineer - X Data
Palo Alto, California, United States
$125k-$400k/yr OnsiteFull Time
xAI
xAI: Develops advanced artificial intelligence systems to understand the universe.
3+ YOE3+ years software engineering experience; expertise in Python,Rust,Scala,Go or Java; experience with data pipelines, realtime and batch processing, and distributed systems.
Python, Rust, Scala, Go, Java, BigQuery, Trino, Clickhouse, Flink, Kafka, Spark, Scalding, SQL
Job PostingView all
3d
Save
Mark Applied
Hide
Member of Technical Staff - Data Platform
New York City or San Francisco or London
OnsiteFull Time
Reflection AI
Reflection AI: Developing open-source frontier artificial intelligence models and coding agents.
Experienced data engineer with production-grade pipeline experience for large-scale batch and streaming workloads; strong debugging, cost/performance optimization, and data quality/governance skills.
Spark, Flink, Beam, Airflow, Dagster, Kafka, PubSub, Parquet, Iceberg, Delta Lake, BigQuery, Snowflake, Great Expectations
Job PostingView all
5d
Save
Mark Applied
Hide
Member of Technical Staff — Data Ingestion & Quality
San Francisco, California, United States
OnsiteFull Time
Causal Labs
Causal Labs: Building physics-based causal AI models for predictive weather intelligence.
Experience building large-scale data pipelines and QA systems, familiarity with streaming and batch ingestion, vendor collaboration, and strong problem-solving and domain learning skills.
Apache Spark, Ray, Beam
Job PostingView all
1mo
Save
Mark Applied
Hide
Data Engineering Manager, Data & ML Platform
San Francisco or Montreal or Bengaluru
$220k-$330k/yr HybridFull Time
Hinge Health
Hinge HealthNYSE: HNGE: Digital provider of musculoskeletal care and physical therapy
5+ YOE2+ Mgmt5+ years data engineering; 2+ years managing engineering teams; 2+ years building ML platform capabilities; experience with batch & streaming systems and tools such as Kafka, Flink, Spark; proficiency with Python, SQL, dbt, Databricks, and AWS.
Kafka, Flink, Spark, Python, SQL, dbt, Databricks, AWS, Delta Lake, MLflow, Unity Catalog
Job PostingView all
2d
Save
Mark Applied
Hide
Staff Software Engineer, Data
Sunnyvale, California, United States
$224k-$322k/yr OnsiteFull Time
Intuitive
IntuitiveNASDAQ: ISRG: Robotic-assisted systems for minimally invasive surgery.
8+ YOEBachelor's or Master's degree, minimum 8 years software engineering experience building data-intensive services, expertise in Java/Go/Python, stream and batch processing, data pipelines, APIs, CI/CD, containers, and AWS.
Java, Go, Python, Apache Flink, Apache Spark, Apache Airflow, REST, GRPC, Postgres, DynamoDB, Gitlab CI/CD, AWS, Docker, Kubernetes, Amazon ECS, Amazon EKS
Job PostingView all
1mo
Save
Mark Applied
Hide
Senior Manager, Content Promotion & Distribution Data Engineering
Los Angeles or Los Gatos
$525k-$950k/yr OnsiteFull Time
Netflix
NetflixNASDAQ: NFLX: Provider of global streaming entertainment and video content.
7+ Mgmt7+ years leading data engineering teams; expertise in data modeling, batch/streaming pipelines, data quality, and multi-modal media pipelines for ML/GenAI; strong stakeholder communication and people management.
S3, Spark
Job PostingView all
1mo
Save
Mark Applied
Hide
Lead AI Engineer, Data Solutions
San Francisco, California, United States
$173k-$260k/yr HybridFull Time
Salesforce
SalesforceNYSE: CRM: Sells cloud-based customer relationship management and business software solutions.
6+ YOE6+ years in AI/ML engineering or applied data science; strong Python in production; ML model development and deployment; data pipelines (ETL/ELT, batch or streaming); APIs and backend systems; experience with LLM-powered systems and agent workflows; familiarity with Spark, Airflow/Dagster, Snowflake/BigQuery.
Python, Spark, Airflow, Dagster, Snowflake, BigQuery
Job PostingView all
1mo
Save
Mark Applied
Hide
Software Engineer, Platform
San Francisco, California, United States
$200k-$250k/yr OnsiteFull Time
Duckbill
Duckbill: SaaS platform for enterprise cloud financial planning and analysis.
Proven experience building data systems and ETL for batch and streaming; strong Python and SQL; experience with data warehouses/lakehouses/OLAP, columnar databases, cloud storage, and data quality practices.
Python, ClickHouse, Airflow, Parquet, S3, Flask, React, OpenTelemetry (OTel), HTTP APIs, SQL
Job PostingView all
1mo
Save
Mark Applied
Hide
Member of Technical Staff (Software Engineer, Data Platform)
San Francisco or New York City or Palo Alto
$220k-$405k/yr OnsiteFull Time
Perplexity
Perplexity: AI-powered search engine providing conversational answers with citations.
5+ YOE5+ years software engineering experience (8+ for staff), production data infrastructure and batch/streaming experience, Airflow/Dagster, Python plus another backend language (Go/TypeScript), ML/AI workflow support, data quality and observability knowledge.
Databricks, Snowflake, Spark, Kafka, Flink, Airflow, Dagster, dbt, Iceberg, Delta Lake, ClickHouse, Kinesis, PubSub, Python, Go, TypeScript
Job PostingView all
1mo
Save
Mark Applied
Hide
Senior Technical Solutions Engineer
San Bruno or Boston or Dallas
$162k-$182k/yr HybridFull Time
Verily
Verily: Developing data-driven technologies for clinical research and precision health.
6+ YOEBA/BS in CS or equivalent, 6+ years experience with Python and/or Java, SQL proficiency, experience with life science/biomedical data and batch workflows, strong communication and project management skills.
Python, Java, SQL, Google Cloud Platform, Amazon Web Services, Microsoft Azure, Docker, Linux
Job PostingView all
3d
Save
Mark Applied
Hide
Software Engineer, Cloud Infrastructure (Multiple Seniority Levels)
San Carlos, California, United States
$135k-$260k/yr HybridFull Time
Beacon AI
Beacon AI: Developing an AI-powered-pilot for safer flight operations.
Design, implement, and operate scalable, secure AWS and LLM infrastructure; experience with LangChain, retrieval systems, data pipelines, and observability best practices.
AWS CDK, Terraform, GitHub Actions, CodeBuild, CodePipeline, VPC, PrivateLink, IAM, KMS, Secrets Manager, AWS Bedrock, SageMaker, ECS, EKS, Lambda, Batch, LangChain, Step Functions, Airflow, OpenSearch Serverless, Aurora PostgreSQL, pgvector, Pinecone, S3, DynamoDB, Glue, Lake Formation, Glacier, Athena, CloudWatch, OpenTelemetry, Redis, Bedrock Guardrails, Triton Inference Server, TensorRT-LLM, EventBridge, SQS, Python
Job PostingView all
1mo
Save
Mark Applied
Hide
Senior Machine Learning Engineer
Glendale or San Francisco or New York
$142k-$199k/yr HybridFull Time
The Walt Disney Company
The Walt Disney CompanyNYSE: DIS: Produces movies, operates theme parks, and provides streaming services.
5+ YOEExperience building and operating large-scale data/ML systems, strong distributed systems fundamentals, streaming/batch technologies (Kafka, Kinesis, Spark, Flink), Python, plus working knowledge of Java/Scala/Go/C++, cloud-native (AWS, containers, Kubernetes, IaC), observability, and collaboration skills; Bachelor's or Master's required.
Kafka, Kinesis, Spark, Flink, Python, Java, Scala, Go, C++, AWS, Kubernetes, containers, IaC tools
Job PostingView all

Explore Jobs

Expand Your Job Search