Senior Databricks Data Engineer

Job Description

Senior Databricks Developer is responsible for designing, developing, optimizing, and operating data solutions on the Databricks platform. The role contributes to the implementation and evolution of the organization's Data Lakehouse by developing data ingestion, transformation, processing, and delivery processes for reporting, advanced analytics, and AI/ML initiatives. The individual will have end-to-end responsibility for data pipelines and will act as a technical reference within the team.

Responsibilities 

Data Development and Integration

• Develops and maintains data ingestion, transformation, and publishing processes using Databricks and PySpark.
• Builds and optimizes ETL/ELT pipelines for large volumes of data.
• Implements Delta Lake structures and data models for operational and analytical consumption.
• Develops incremental loading, historization, and CDC (Change Data Capture) mechanisms.
• Integrates data from multiple sources, including Oracle, SQL Server, APIs, files, banking applications, and others.

Data Architecture and Modeling

• Participates in defining the Data Lakehouse architecture.
• Proposes scalable and high-performance data models.
• Contributes to the development of coding standards and platform architecture.
• Ensures data traceability and consistency throughout the entire processing flow.

Data Quality and Operations

• Implements automated data quality controls.
• Develops data reconciliation and validation processes.
• Configures process monitoring, alerting, and logging.
• Analyzes and resolves production incidents.
• Manages data reloads, reruns, corrections, and recovery processes.

Optimization and Performance

• Optimizes PySpark and SQL processes for cost efficiency and performance.
• Identifies and eliminates processing bottlenecks.
• Optimizes the use of Databricks resources.
• Contributes to the automation of development and deployment processes.

Job Requirements

• Minimum 5 years of experience in Data Warehouse, Big Data, or enterprise data platform development.
• Minimum 2-3 years of hands-on experience with Databricks.
• Demonstrated experience in complex Data Warehouse, Data Mart, or Data Lake projects.
• Experience developing end-to-end data pipelines.
• Strong knowledge of Data Lakehouse architecture.
• Experience implementing historization, CDC, reconciliation, and Data Quality processes.
• Ability to take full ownership of technical deliveries and provide technical leadership to the team.

Mandatory Tech Skills:

• Azure Databricks/PySpark
• SQL avansat
• Delta Lake
• Git & Jira
• Data Modelling
• ETL / ELT
• Data Quality Frameworks
• Pipeline Orchestration

Nice to have technical skills: 

• Azure Data Factory
• Kafka
• Event Streaming
• Python avansat
• CI/CD
• Unity Catalog
• Data Governance
• Machine Learning pe Databricks
• Qlik Sense

Banking Experience (a Major Advantage)

• Knowledge of Retail Banking products and processes.
• Experience with financial reporting and management reporting.
• Knowledge of risk, compliance, and regulatory reporting processes.
• Understanding of banking data flows and enterprise analytics ecosystems.

The preceding job description had been designed to indicate the general nature and level of work performed by employees within this classification. It is not designed to contain or be interpreted as a comprehensive inventory of all duties and responsibilities required of employees assigned to this job.
What we offer:You will find here not only a challenging and interesting workplace, but also a rewarding work experience, with competitive compensation and benefit packages:

• Soft skills and technical training for current and emerging products and technologies;
• Career development: opportunity to grow within the team;
• Special referral bonuses for recommending your friends;
• Private Medical subscription;
• A monthly budget for flexible benefits program that includes public transportation reimbursement and meal tickets, sport & wellness, telecom & electronics, children & parents, holiday & culture, tourism;
• Flexible working hours and work from home, aligned with project needs;
• Friendly team who is eager to meet you.

What's next:It's best to apply today, because job postings can be taken down and we wouldn't want you to miss this opportunity. In case you need further information, just send us a message at [email protected] and we'll be happy to assist! Diversity & InclusionHere at the Stefanini Group, we value plurality and equity, regardless of race, sexual orientation, disability, age, ancestry, religion, gender, and nationality. We understand and encourage the importance of being you! About usWe are the Stefanini group, a global tech consulting company of Brazilian origin that believes in the power of people to transform businesses through technology. We are present in over 40 countries and operate with the purpose of co-creating solutions TOGETHER WITH OUR CLIENTS that accelerate results and improve the experience of people and organizations. Here, we like to say that technology is not the end, but the means: what really matters are the people who drive it all. Our mindset is AI First, meaning we invest in cutting-edge technology in everything we do, focusing on results for our clients. We are a company, A GROUP, that breathes collaboration and offers a dynamic environment where you will learn by doing, grow alongside the team, and have space to contribute with ideas and projects. More than just talking about digital transformation, we believe in real transformation that starts with people and impacts real businesses. If you are looking for a place to develop, innovate, and be part of something bigger, the Stefanini Group is your place. We want to inform you that there are currently scams targeting job seekers by falsely using our company's name, Stefanini. We sincerely apologize for any confusion or inconvenience this may have caused. Please remember that legitimate job offers from Stefanini will always come through official channels, including direct communication with our trained recruiters. If you receive any unsolicited messages requesting payment or personal information, please disregard them. If you suspect you've been targeted, please contact us immediately at [email protected] for verification. Key Points to Remember: * Legitimate job offers only follow interviews conducted with our hiring managers or clients.* We will never ask for payment at any stage of the recruitment process. Stay vigilant and feel free to reach out for verification. Your safety and security are our top priorities. Thank you for your understanding and cooperation.
#LI-HYBRID