Description
AWS operates one of the largest networks in the world, supporting compute and storage for hundreds of services and millions of customers. Within that organisation, our team owns the data, metrics, and operational tooling that engineers use every day to keep the network running, mitigate incidents quickly, and continuously improve how we operate at scale.
We are looking for a Data Engineer to join a small, senior team and help build and maintain the pipelines, data models, and datasets that power these decisions. You will work alongside experienced data engineers, network engineers, data scientists, and software engineers who will help you grow your skills while you contribute to real production systems from day one. You should be enthusiastic about distributed data systems, eager to learn, and comfortable asking questions when requirements are unclear.
A component of this role involves platform migration work. You will contribute to the migration of legacy data warehouse workloads onto our cloud-native data platform, working with senior engineers to analyse dependencies, coordinate with customers, and implement clean decommissioning of deprecated assets. You will also help build and maintain the data quality and monitoring tooling that gives downstream engineers confidence in the data they consume.
You should have solid written and verbal communication skills, as you will work closely with diverse teams across multiple time zones. You should be curious, willing to dive into unfamiliar problem spaces, and passionate about working with operational data at scale.
Key job responsibilities
- Develop and maintain ETL/ELT pipelines on AWS using services such as Glue, EMR/Spark, Lambda, Redshift, and S3.
- Build physical data models and optimize data pipelines for datasets in the team's domain, with guidance from senior engineers on logical data model and architectural decisions.
- Contribute to the migration of legacy datasets, jobs, and stored procedures onto the strategic data platform, including dependency analysis and implementation of cutover plans designed by senior engineers.
- Develop and maintain data quality checks, monitoring, and alerting for production pipelines.
- Implement reporting and analytics infrastructure for internal customers, including dashboards, data catalog entries, and documentation.
- Work with data scientists, network engineers, and product managers to understand requirements and deliver data solutions.
- Participate in design reviews and code reviews. Seek feedback on your designs and code, and provide constructive review comments to peers.
- Support on-call rotation for the team's data systems with mentorship from senior engineers. Troubleshoot pipeline failures and escalate when needed.
- Document your solutions clearly to ensure ease of use and maintainability by others.
A day in the life
Most days are code. Some days are pipeline work: implementing a new Glue job from a design spec, troubleshooting a failing Spark job, or optimising a slow Redshift query. Some days are migration work: cataloguing dependencies on a deprecated table, or implementing a cutover plan. You will get your code reviewed by senior engineers and learn the team's data architecture through hands-on work. On-call weeks are supported by senior team members; off-call weeks let you focus on project delivery.
About the team
Our team builds the data and operational tooling that helps a global network engineering organisation move from reactive incident response toward proactive, measurement-driven operations. We are small, senior, and distributed between Sydney and the United States. Our customers are mostly internal: network engineers, data scientists, and software teams who rely on our datasets to automate network operations and drive availability. We invest in mentoring engineers and helping them grow their skills across the full data engineering stack.
We are looking for a Data Engineer to join a small, senior team and help build and maintain the pipelines, data models, and datasets that power these decisions. You will work alongside experienced data engineers, network engineers, data scientists, and software engineers who will help you grow your skills while you contribute to real production systems from day one. You should be enthusiastic about distributed data systems, eager to learn, and comfortable asking questions when requirements are unclear.
A component of this role involves platform migration work. You will contribute to the migration of legacy data warehouse workloads onto our cloud-native data platform, working with senior engineers to analyse dependencies, coordinate with customers, and implement clean decommissioning of deprecated assets. You will also help build and maintain the data quality and monitoring tooling that gives downstream engineers confidence in the data they consume.
You should have solid written and verbal communication skills, as you will work closely with diverse teams across multiple time zones. You should be curious, willing to dive into unfamiliar problem spaces, and passionate about working with operational data at scale.
Key job responsibilities
- Develop and maintain ETL/ELT pipelines on AWS using services such as Glue, EMR/Spark, Lambda, Redshift, and S3.
- Build physical data models and optimize data pipelines for datasets in the team's domain, with guidance from senior engineers on logical data model and architectural decisions.
- Contribute to the migration of legacy datasets, jobs, and stored procedures onto the strategic data platform, including dependency analysis and implementation of cutover plans designed by senior engineers.
- Develop and maintain data quality checks, monitoring, and alerting for production pipelines.
- Implement reporting and analytics infrastructure for internal customers, including dashboards, data catalog entries, and documentation.
- Work with data scientists, network engineers, and product managers to understand requirements and deliver data solutions.
- Participate in design reviews and code reviews. Seek feedback on your designs and code, and provide constructive review comments to peers.
- Support on-call rotation for the team's data systems with mentorship from senior engineers. Troubleshoot pipeline failures and escalate when needed.
- Document your solutions clearly to ensure ease of use and maintainability by others.
A day in the life
Most days are code. Some days are pipeline work: implementing a new Glue job from a design spec, troubleshooting a failing Spark job, or optimising a slow Redshift query. Some days are migration work: cataloguing dependencies on a deprecated table, or implementing a cutover plan. You will get your code reviewed by senior engineers and learn the team's data architecture through hands-on work. On-call weeks are supported by senior team members; off-call weeks let you focus on project delivery.
About the team
Our team builds the data and operational tooling that helps a global network engineering organisation move from reactive incident response toward proactive, measurement-driven operations. We are small, senior, and distributed between Sydney and the United States. Our customers are mostly internal: network engineers, data scientists, and software teams who rely on our datasets to automate network operations and drive availability. We invest in mentoring engineers and helping them grow their skills across the full data engineering stack.
Basic Qualifications
- 1+ years of data engineering experience
- Experience with data modeling, warehousing and building ETL pipelines
- Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala)
- Experience with one or more scripting language (e.g., Python, KornShell)
- Experience with data modeling, warehousing and building ETL pipelines
- Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala)
- Experience with one or more scripting language (e.g., Python, KornShell)
Preferred Qualifications
- Experience with big data technologies such as: Hadoop, Hive, Spark, EMR
- Experience with any ETL tool like, Informatica, ODI, SSIS, BODI, Datastage, etc.
Acknowledgement of country:
In the spirit of reconciliation Amazon acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.
IDE statement:
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
- Experience with any ETL tool like, Informatica, ODI, SSIS, BODI, Datastage, etc.
Acknowledgement of country:
In the spirit of reconciliation Amazon acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.
IDE statement:
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
