HiringCafe
Join our community

265 pyspark developer jobs at 112 companies in San Francisco, CA

2w
Save
Mark Applied
Hide
Sr. Software Engineer - PySpark
Scottsdale or San Francisco or Chicago or New York City
$118k-$152k/yr HybridFull Time
Early Warning Services
Early Warning Services: Operates payment and risk solutions for the financial industry.
5+ YOEBachelor's-level education typical; minimum 5 years software engineering experience; strong with Git, Spring/Spring Boot, SQL, REST, Docker/Kubernetes, Kafka, automated testing and DevOps practices.
Git, Git flow, Spring, Spring Boot, SQL, Confluence, CI/CD, Docker, Kubernetes, Kafka, Hadoop, Hive, XML, JSON, YAML, Redis, HBASE, AeroSpike, REST
Job PostingView all
1mo
Save
Mark Applied
Hide
Associate Data Engineer
Redwood City, California, United States
$138k-$164k/yr HybridFull Time
Stanford University
Stanford University: A private research university providing higher education and research.
5+ YOEBachelor's degree plus 5 years relevant experience. Expertise in SQL, Python/PySpark, cloud data services, data modeling, ETL/ELT, and strong communication skills.
Advanced SQL, Advanced Python, SQL, Python, PySpark, S3, Redshift, Glue ETL, IAM, DMS, Appflow, VPC, AWS Services, SnowFlake, FiveTran, KAFKA, Airflow, Oracle Cloud, Scala, R
Job PostingView all
1mo
Save
Mark Applied
Hide
(USA) Senior, Software Engineer
Bentonville or Sunnyvale
$90k-$180k/yr OnsiteFull Time
Walmart
WalmartNYSE: WMT: Operates a chain of hypermarkets, discount stores, and grocery stores.
3+ YOERequires a bachelor's degree and 3 years of software engineering experience, or 5 years of experience. Requires Python, PySpark, SQL, shell scripting, GCP, data engineering, and AI workflow expertise.
Python, Scala, Apache Spark, Airflow, Google Cloud Platform (GCP), Dataproc, Cloud Scheduler, Cloud Storage, BigQuery, Apache Hudi, GitHub Actions, Terraform, PySpark, SQL, Jenkins, Azure DevOps, Cloud Functions, Cloud Dataflow, Vertex AI, LangChain, LangGraph, RAG, Agentic AI, SAP, WCAG 2.2 AA
Job PostingView all
4d
Save
Mark Applied
Hide
Palantir Forward Deployed Engineer
Albany or Arlington or Atlanta or Austin or Beaverton or Bentonville or Boston or Carmel or Charlotte or Chicago or Cincinnati or Cleveland or Columbus or Culver City or Denver or Des Moines or Detroit or Hartford or Houston or Irving or Kirkland or Miami or Milwaukee or Minneapolis or Morristown or Mountain View or Nashville or New York City or Oklahoma City or Overland Park or Philadelphia or Pittsburgh or Raleigh or Redmond or Sacramento or San Diego or San Francisco or Scottsdale or Seattle or St. Louis or St. Petersburg or Walnut Creek or United States
$54k-$235k/yr HybridFull Time
Accenture
AccentureNYSE: ACN: Global professional services firm providing consulting and technology solutions.
3+ YOERequires 3+ years in data engineering, cloud platforms, and Python, PySpark, or Java; 1+ year with Palantir Foundry and AIP; bachelor's degree or equivalent experience.
Palantir Foundry, Palantir Gotham, Palantir Artificial Intelligence Platform (AIP), Python, TypeScript, PySpark, Java, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), Git, GitHub, GitLab, Jenkins, JavaScript
Job PostingView all
3d
Save
Mark Applied
Hide
Senior, Software Engineer - Python
Sunnyvale, California, United States
$117k-$234k/yr OnsiteFull Time
Walmart
WalmartNYSE: WMT: Multinational retail operating discount stores and supermarkets.
3+ YOEBachelor's degree plus 3 years or 5 years of software engineering experience; expertise in Scala, Python, PySpark, Java, Apache Spark, distributed data pipelines, CI/CD, and cloud data ecosystems.
Scala, Apache Spark, Python, PySpark, GCP Dataproc, Grafana, Open Observe, Java, Airflow, Kafka, Cassandra, GCS, Kubernetes, WCNP, CI/CD, Web Content Accessibility Guidelines (WCAG)
Job PostingView all
3w
Save
Mark Applied
Hide
Software Engineer AI/ML Ops
Pleasanton or New York City or Los Angeles
$145k-$182k/yr HybridFull Time
BlackLine
BlackLineNASDAQ: BL: Provides cloud-based financial close and accounting automation software.
2+ YOE2+ years in Python/Java/Scala, experience building PySpark ETL, ML/LLM production pipelines, cloud (GCP/AWS/Azure), ML frameworks, orchestration, containerization, and observability.
PySpark, FiveTran, Plaid, TensorFlow, PyTorch, scikit-learn, Airflow, Kubeflow, Vertex AI, MLflow, GCP, AWS, Azure, LangChain, LangGraph, ADK, Prometheus, Grafana, Newrelic, Docker, Kubernetes, Bash, Python, Java, Scala
Job PostingView all
2w
Save
Mark Applied
Hide
Senior Software Engineer, Data Engineering
San Mateo, California, United States
$243k-$295k/yr HybridFull Time
Roblox
RobloxNYSE: RBLX: Platform for creating and playing user-generated 3D digital experiences.
8+ YOE3+ Mgmt8+ years building scalable ETL pipelines, expertise in SQL/PySpark/Scala, experience with cloud platforms and ETL orchestration, 3+ years leading data engineering, strong debugging and collaboration skills.
Airflow, Dagster, Luigi, Google Cloud Composer, SQL, PySpark, Scala, AWS, GCP, Azure
Job PostingView all
1mo
Save
Mark Applied
Hide
Staff Data Engineer - Quickbooks Capital and Lending
Mountain View, California, United States
$203k-$274k/yr OnsiteFull Time
Intuit
IntuitNASDAQ: INTU: Provides financial software for accounting, tax, and personal finance.
7+ YOE7+ years data engineering experience with Python, PySpark, Spark, SQL, Data Bricks/DataBricks, AWS, Elastic Search, Kafka; BS/MS in CS or equivalent; production support and on-call experience; SDLC familiarity.
Python, PySpark, Spark, SQL, Data Bricks, DataBricks, AWS, Elastic Search, Kafka
Job PostingView all
2mo
Save
Mark Applied
Hide
Palantir Forward Deployed Engineer
Chicago or Milwaukee or Dallas or Columbus or Kirkland or Cincinnati or New York or Cleveland or Oklahoma City or Austin or Albany or St. Petersburg or Hartford or Pittsburgh or St. Louis or Miami or Sacramento or Raleigh or Minneapolis or Mountain View or Scottsdale or San Francisco or Morristown or Denver or Boston or Philadelphia or Des Moines or Overland Park or Los Angeles or Charlotte or Walnut Creek or Carmel or Seattle or Houston or Arlington or Atlanta or Redmond or Bentonville or Beaverton or Nashville or Detroit or San Diego
$54k-$235k/yr HybridFull Time
Accenture
AccentureNYSE: ACN: Global provider of management consulting and technology services.
3+ YOE3+ years data engineering; 1+ year Palantir Foundry experience; 1+ year prompt engineering/AIP; 3+ years cloud (AWS/Azure/GCP); 3+ years Python/PySpark/Java; Bachelor's in CS/Engineering or equivalent; strong communication.
Palantir Foundry, Palantir Gotham, Artificial Intelligence Platform (AIP), Python, TypeScript, PySpark, Java, AWS, Azure, GCP, Git, GitHub, GitLab, Jenkins
Job PostingView all
2mo
Save
Mark Applied
Hide
Machine Learning Engineer III - Document Intelligence
Pleasanton or Vancouver
$160k-$240k/yr HybridFull Time
Workday
WorkdayNASDAQ: WDAY: Provides cloud-based software for financial and human capital management.
3+ YOE3+ years building and deploying production ML systems, 2+ years Python, experience with deep learning, NLP/LLMs, PyTorch/TensorFlow, PySpark/SQL, cloud-native deployment, and scalable ML pipelines.
PyTorch, TensorFlow, LangChain, LangGraph, Python, PySpark, SQL, Docker, K8s, AWS, GCP, PEFT, DeepEval, RAGAS, DSPy
Job PostingView all
1mo
Save
Mark Applied
Hide
Data Engineer
New York or Los Angeles or San Francisco
$140k-$175k/yr HybridFull Time
Regard
Regard: A that builds AI-powered software to improve clinical documentation and advance patient care.
3+ YOEBS or equivalent, 3+ years data engineering experience, Python and SQL proficiency, experience with PySpark and distributed data processing, AWS experience preferred, LLM-assisted development familiarity, and willingness to participate in on-call support.
S3, Apache Iceberg, EMR, PySpark, Dagster, Kubernetes, ClickHouse, PostgreSQL, FastAPI, Metabase, Python, SQL, AWS, AWS Athena, EHR, OMOP CDM, HIPAA, LLM
Job PostingView all
1w
Save
Mark Applied
Hide
Student Worker - Machine Learning Engineer - Data Mining & VLM
Foster City, California, United States
$30/hr OnsiteContract
Zoox
ZooxNASDAQ: AMZN: Developing autonomous robotaxis for urban ride-hailing services.
Currently enrolled in a relevant BS or MS program; strong Python, PySpark, and SQL; ML/VLM evaluation and production LLM/VLM experience; available 40 hours weekly for at least three months.
Python, PySpark, SQL, Scala, Git, Spark, LLM, VLM, RAG
Job PostingView all
3mo
Save
Mark Applied
Hide
Machine Learning Software Engineer Intern - Winter 2027
San Francisco or New York City
HybridInternship, Temporary
Rippling
Rippling: Unified platform managing workforce HR, IT, and finance operations
Currently enrolled in an M.Sc. or Ph.D. in computer science or related field; strong backend programming skills (Python, PySpark, PyTorch); ML experience, LLM familiarity preferred; excellent communication and learning mindset.
Python, PySpark, PyTorch, LLMs
Job PostingView all
1w
Save
Mark Applied
Hide
Software Engineer
San Francisco or Seattle or Maryland or New Jersey or New York or Alaska or Hawaii or Maine or Mississippi or North Dakota or South Dakota or Vermont or West Virginia or Wyoming
$118k-$183k/yr HybridFull Time
DocuSign
DocuSignNASDAQ: DOCU: Provides electronic signature and agreement management software solutions.
3+ YOEBachelor's degree in a relevant field, 3+ years of software development, and experience with search engines, distributed processing, public cloud services, and CI/CD tools.
C#, Java, C++, Go, Elasticsearch, Solr, Apache Spark, Kafka, Azure, AWS, GCP, Git, Jenkins, Azure DevOps, Docker, Kubernetes
Job PostingView all
1w
Save
Mark Applied
Hide
Software Engineer
San Francisco or Seattle or Maryland or New Jersey or New York or Alaska or Hawaii or Maine or Mississippi or North Dakota or South Dakota or Vermont or West Virginia or Wyoming
$112k-$183k/yr HybridFull Time
DocuSign
DocuSignNASDAQ: DOCU: Provider of e-signature and intelligent agreement management software.
3+ YOEBachelor's degree in a relevant field, 3+ years of software development, and experience with large-scale data systems, search engines, distributed processing, and public cloud services.
Azure, Docker, Kubernetes, Elasticsearch, Solr, Apache Spark, Kafka, AWS, GCP, Git, Jenkins, Azure DevOps, C#, Java, C++, Go
Job PostingView all
1mo
Save
Mark Applied
Hide
Senior System Software Engineer
Santa Clara or Champaign or Redmond
$184k-$357k/yr OnsiteFull Time
NVIDIA
NVIDIANASDAQ: NVDA: Designs GPU-accelerated computing and artificial intelligence hardware.
9+ YOE9+ years software development experience, 5+ years data platform development, proficiency in C++, Java, Scala, CUDA; BS/MS/PhD in computer science or equivalent; experience with Apache Spark and supporting enterprise customers.
C++, CUDA, Java, Scala, Apache Spark, Velox, Presto, Apache Arrow, Apache DataFusion, RAPIDS, CCCL, UCX
Job PostingView all
2mo
Save
Mark Applied
Hide
Data Engineer
San Francisco, California, United States
$140k-$230k/yr OnsiteFull Time
ReadyOn
ReadyOn: AI-native labor operating system for frontline enterprise workforces.
5+ YOE5+ years data engineering experience; strong SQL; Python; TypeScript; Airflow; AWS Glue; PySpark; Snowflake/PostgreSQL; backend frameworks; observability and data quality mechanisms.
Python, TypeScript, Apache Airflow, PySpark, AWS Glue, Snowflake, PostgreSQL, NestJS, GraphQL, AWS
Job PostingView all
3mo
Save
Mark Applied
Hide
Senior Data Quality Engineer
Palo Alto, California, United States
$67-$88/hr HybridFull Time
Stanford Health Care
Stanford Health Care: Provides comprehensive medical care, specialized treatments, and clinical research.
3+ YOEBS/BA required. 7+ years reporting/analytics/BI/healthcare tech OR 3+ years cloud datalake development (PySpark on Databricks) with DevOps/CI/CD. Strong SQL, data modeling, metadata, MDM and governance experience.
Databricks Unity Catalog, Databricks, PySpark, Notebooks, GitHub, SQL, Azure, CI/CD, DevOps
Job PostingView all
2d
Save
Mark Applied
Hide
Manager, Developer Relations - Open Source
San Francisco, California, United States
$172k-$237k/yr OnsiteFull Time
Databricks
Databricks: A unified platform for data analytics and artificial intelligence.
8+ YOE2+ MgmtRequires 8+ years in developer relations or hands-on technical roles, 2+ years managing DevRel teams, open source contributions and governance experience, technical expertise in data platforms, and exceptional communication skills.
Apache Spark, Delta Lake, Unity Catalog, Apache Iceberg, MLflow, Omnigent, Lakeflow, Lakebase, Genie Agents, AI/BI, Databricks Data + AI Platform
Job PostingView all
2w
Save
Mark Applied
Hide
Informatica PowerCenter & IDMC (IICS) Developer – Migration Specialist
Pleasanton, California, United States
OnsiteContract
Delan Associates
Delan Associates: Provides engineering and professional services to government agencies.
13+ YOE13+ years experience with Informatica PowerCenter and IDMC (IICS), ETL development and migration, SQL/Oracle/SQL Server, CDI, performance tuning, testing, troubleshooting, deployment and production support.
Informatica PowerCenter, Informatica IDMC (IICS), Cloud Data Integration (CDI), ETL, SQL, Oracle, SQL Server, Azure Data Factory, Databricks, Azure Synapse, Python, PySpark, Agile/Scrum
Job PostingView all

Explore Jobs

Expand Your Job Search