Job Description:
Job Description: Data Engineer – Databricks & Microsoft Fabric
Location : Chennai, India
Experience : 6 to 8 years
Role Summary
We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, Microsoft Fabric, PySpark, SQL, and cloud-based data engineering. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines, lakehouse solutions, and analytics-ready data models.
The role requires strong experience in building end-to-end data engineering solutions, working with structured and semi-structured data, implementing data quality controls, and supporting enterprise reporting and analytics platforms.
Key Responsibilities
Data Engineering and Pipeline Development
• Design, build, and maintain scalable data pipelines using Databricks, PySpark, Spark SQL, and Microsoft Fabric.
• Develop batch and incremental data ingestion pipelines from multiple source systems.
• Build and optimize Bronze, Silver, and Gold layer data models using lakehouse architecture.
• Implement ELT/ETL workflows for data transformation, enrichment, validation, and publishing.
• Work with structured, semi-structured, and unstructured data formats such as CSV, Parquet, JSON, Delta, and XML.
Databricks Development
• Develop notebooks, jobs, workflows, and reusable components in Azure Databricks.
• Implement Delta Lake features such as schema evolution, merge/upsert, time travel, and optimized storage.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement partitioning, caching, indexing, and cluster optimization strategies.
• Troubleshoot job failures, performance bottlenecks, and data quality issues.
Microsoft Fabric Development
• Build data solutions using Microsoft Fabric Lakehouse, Warehouse, Data Factory, Pipelines, Notebooks, and OneLake.
• Develop and manage data pipelines in Fabric for ingestion, transformation, and orchestration.
• Work with Fabric SQL endpoints, semantic models, and Power BI integration.
• Support migration or modernization of existing data platforms into Microsoft Fabric.
• Implement reusable data engineering patterns and framework-based development in Fabric.
Data Quality, Governance, and Security
• Implement data validation, reconciliation, exception handling, and audit controls.
• Define and apply data quality rules including null checks, duplicate checks, referential checks, and cross-field validations.
• Maintain data lineage, metadata, source-to-target mapping, and technical documentation.
• Ensure data pipelines comply with enterprise security, access control, and governance standards.
• Support integration with data governance tools such as Microsoft Purview, where applicable.
DevOps and Production Support
• Implement CI/CD practices for notebooks, pipelines, SQL scripts, and configuration files.
• Use Git-based version control and deployment processes across environments.
• Monitor production jobs and resolve incidents within agreed timelines.
• Prepare runbooks, deployment guides, operational support documents, and handover materials.
• Collaborate with architects, business analysts, data analysts, and reporting teams to deliver reliable data solutions.
Required Skills
Technical Skills
• Strong hands-on experience in Azure Databricks.
• Strong experience in Microsoft Fabric components such as Lakehouse, Warehouse, Data Factory, Pipelines, Notebooks, and OneLake.
• Proficiency in PySpark, Spark SQL, Python, and SQL.
• Strong knowledge of Delta Lake, lakehouse architecture, and medallion architecture.
• Experience with cloud storage and data platforms, preferably Azure Data Lake Storage, Azure SQL, Synapse, or Fabric OneLake.
• Experience in data ingestion from databases, APIs, files, SFTP, cloud storage, and streaming sources.
• Good understanding of data modeling, dimensional modeling, and analytics-ready data structures.
• Experience with performance tuning of Spark jobs and SQL queries.
• Experience in job scheduling, monitoring, logging, and error handling.
• Knowledge of CI/CD, Git, Azure DevOps, and deployment automation.
Preferred Skills
• Experience with Power BI and semantic model integration.
• Experience in migrating workloads from legacy ETL tools, Synapse, ADF, or Databricks to Microsoft Fabric.
• Knowledge of Microsoft Purview for data cataloging, lineage, and governance.
• Experience in building reusable data engineering frameworks.
• Exposure to real-time or near-real-time data processing.
• Azure certifications or Databricks certifications are preferred.
Roles and Responsibilities Summary
• Build and maintain scalable data pipelines using Databricks and Microsoft Fabric.
• Develop lakehouse solutions using Bronze, Silver, and Gold architecture.
• Perform data transformation, validation, reconciliation, and publishing.
• Optimize Spark workloads and SQL queries for performance.
• Implement data quality, audit, monitoring, and exception handling frameworks.
• Support deployment, production monitoring, incident resolution, and documentation.
• Collaborate with cross-functional teams to deliver enterprise data and analytics solutions.
Required Experience
• 6 to 8 years of overall experience in data engineering, ETL/ELT, or data platform development.
• At least 3+ years of hands-on experience in Databricks / PySpark.
• At least 1+ year of hands-on experience or strong working knowledge of Microsoft Fabric.
• Experience working in enterprise-scale data platforms and analytics projects.
• Experience in Agile delivery models and production support environments.
Educational Qualification
Bachelor’s degree in computer science, Information Technology, Engineering, Data Analytics, or a related discipline.
Good to Have Certifications
• Microsoft Certified: Fabric Analytics Engineer Associate
• Microsoft Certified: Azure Data Engineer Associate
• Databricks Certified Data Engineer Associate / Professional
• Microsoft Certified: Azure Fundamentals
Key Deliverables
• Production-ready data pipelines and notebooks.
• Optimized Databricks and Fabric workloads.
• Bronze, Silver, and Gold layer data models.
• Data quality and reconciliation reports.
• Source-to-target mapping and technical design documents.
• Deployment guides, runbooks, and support documentation.
Recruiter Name: Srinija Adapa
Recruiter Email ID: [email protected]
Get empowered by NTT DATA Business Solutions!
We transform. SAP® solutions into Value
NTT DATA Business Solutions is a fast-growing international IT company and one of the world’s leading SAP partners. We are a full service provider delivering everything from business consulting to implementation of SAP solutions, including hosting services and support.
