Save Job
Posted 2d ago

Senior Data Engineer - Teradata/Databricks & AI/ML

@ NTT DATA
Bangalore, Karnataka, India
OnsiteFull Time
Responsibilities:ingesting logs, building pipelines, analyzing usage
Requirements Summary:Experience ingesting and analyzing Teradata DBQL logs, building Databricks pipelines, integrating Autosys/DataStage/MagicWand metadata, applying AI/ML and LLMs for data classification and cost-reduction analysis.
Technical Tools Mentioned:Teradata DBQL, Databricks, Autosys, DataStage, MagicWand, SQL, ResUsage, AI/ML, LLM
Save
Mark Applied
Hide Job
Report & Hide
Job Description

•    Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
•    Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
•    Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
•    Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
•    Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
•    Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
•    Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
•    Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.


Job Segment: Developer, Database, Business Intelligence, Data Warehouse, SQL, Technology