Description
In this role, you'll contribute across the stack: developing ingest pipelines, building scalable REST APIs, and facilitating data exploration and understanding. The platform supports large-scale data ingestion, complex queries, and interactive analysis. While your primary focus will be on the data-pipeline layer, you’ll collaborate closely with other sub-teams to ensure end-to-end functionality and performance. We’re looking for someone excited to work across the system and to improve team processes and tooling, especially for faster integration of new data sources.
Responsibilities
Lead the design and implementation of data-processing workflows
Manage all aspects of the data-processing lifecycle (collection, discovery, analysis, cleaning, modeling, transformation, enrichment, validation)
Develop and maintain data models and JSON Schemas to ensure integrity and consistency
Collaborate with analysts and engineers to meet data requirements
Manage and optimize data storage/retrieval in Elasticsearch and Dgraph (plus MongoDB and Redis)
Orchestrate dataflow using Apache NiFi
Mentor teammates on best practices for data processing and software engineering
Use AI platforms to support hybrid automated/manual data transformation, code generation, and schema management
Work with analysts, product owners, and engineers to ensure solutions meet operational needs
Propose and implement process improvements for faster delivery of new data sources
Required Skills & Experience
Strong data-wrangling and dataflow background (discovery, mining, cleaning, exploration, enrichment, validation)
Proficiency in JSON and JSON Schemas (or similar)
Solid data-modeling experience
Experience with NoSQL databases (Elasticsearch, MongoDB, Redis, graph DBs)
Familiarity with dataflow tools such as Apache NiFi
Extensive experience in Python or Java (both preferred)
Experience using generative AI for code and data transformation
Git for version control; Maven for build automation
Comfortable in a Linux development environment
Familiarity with Atlassian tools (Jira, Confluence)
Strong communication and teamwork skills
Nice to Have
Experience with various corporate data formats
Knowledge of Kafka or RabbitMQ
Proficiency in Java/Spring (Boot, MVC/REST, Security, Data)
AWS (EC2, S3, Lambda) experience
API design for data services
Frontend experience (modern JS + Vue.js or similar)
CI/CD (e.g., Jenkins), automated testing (JUnit)
Docker, Kubernetes, and other containerization tech
DevOps tools (Packer, Terraform, Ansible)
Qualifications
12+ years of relevant experience and a B.S. in a technical discipline
(Four additional years of experience may substitute for a degree)
