Lead Data Engineer

Key Responsibilities

• Understanding the business requirements
• Writing code and implementing the proposed solutions
• Creating data pipelines, versioning and change management
• Manage the complexity inherent in versioned data pipelines
• Develop ETL/ELT processes to help extract and manipulate data from multiple sources.
• Design, build and maintain batch or real-time data pipelines in production.
• Automate data workflows such as data ingestion, aggregation, and ETL processing.
• Logging and instrumentation of pipelines and services.
• Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
• Train and mentor junior engineers.

Job Description

Data Engineer

• NoSQL-to-RDBMS Mapping: Flatten and normalize complex, nested JSON data from MongoDB into structured relational tables.
• Pipeline Development: Build and manage reliable ETL pipelines to ingest data from MongoDB, process it via Snowflake, and load it into MySQL.
• Database Optimization: Design, index, and tune target MySQL databases to ensure high performance and strict data integrity.
• Snowflake Utilization: Leverage Snowflake's variant handling capabilities to efficiently stage, parse, and transform semi-structured data.

Technical Stack

• Proficiency in MongoDB, MySQL, Snowflake SQL, good to have Python (including ETL frameworks like rivery or informatica).