About the Role

We are looking for a Data Pipeline Engineer with strong expertise in Databricks, AWS Glue, Python, SQL, and MongoDB to design, develop, and optimize scalable data pipelines. The ideal candidate should have experience building cloud-native ETL solutions, transforming large datasets, and delivering high-quality data for analytics and business applications.

Key Responsibilities

• Develop and maintain scalable ETL/data pipelines using Databricks and AWS Glue.
• Build data ingestion and transformation workflows for structured and semi-structured data.
• Write efficient PySpark, Python, and SQL code for large-scale data processing.
• Integrate data from multiple sources, including MongoDB and relational databases.
• Implement data quality, validation, and reconciliation checks.
• Optimize pipeline performance, reliability, and scalability.
• Collaborate with architects, analysts, and application teams to deliver data solutions.
• Support production deployments, monitoring, and troubleshooting.

Required Skills

• 5+ years of experience in Data Engineering or ETL development.
• Strong hands-on experience with Databricks and PySpark.
• Experience with AWS Glue and AWS data services.
• Proficiency in Python and SQL.
• Experience working with MongoDB.
• Knowledge of ETL, data modeling, and data warehousing concepts.
• Familiarity with Delta Lake/Lakehouse architecture is preferred.
• Experience with Git and Agile development practices.

Good to Have

• Delta Lake
• Apache Spark optimization
• AWS S3
• CI/CD pipelines
• Healthcare or Benefits domain experience