Senior Data Engineer

What You’ll Do

• Design and implement a scalable Lakehouse environment. Own the lifecycle of efficient storage, including schema evolution, partition transformation, and snapshot management.
• Build and manage real-time and batch data ingestion pipelines using Kafka and Spark.
• Deploy, manage, and scale data workloads (such as Spark executors) using Kubernetes (EKS).
• Develop complex ETL workflows. Optimize jobs for large-scale data processing focusing on memory management and shuffle optimization.
• Manage and monitor end-to-end data lifecycles using orchestration tools.
• Write and tune high-performance queries against the Lakehouse. Implement optimization strategies, compaction, and data skipping to reduce latency and cloud costs.

Requirements

• 5+ years in Data Engineering, with at least 2 years focused on Lakehouse or modern Data Lake environments.
• Hands-on experience with AWS ecosystems, async/queue processing in a production environment and working with a Lakehouse.
• Expert in workflow management, data transformation, storage and exchange at scale.
• Proven experience with events driven job processing. Mastery in data ingestion, processing large-scale data and performance tuning.
• Experience with Kubernetes and infrastructure as code.

Nice to Have

• Experience with query engines like Starrocks or Druid.
• Familiarity with building and monitoring dashboards on K8s-based data clusters.

Perks & Benefits

• Medical, dental, vision, and basic life insurance.
• Flexible PTO and company paid holidays.
• Retirement programs.
• 1% charitable giving program.

Compensation

• Base pay: $115,000 - $175,000. The salary range listed here has been provided to comply with local regulations and represents a potential base salary range for this role. Please note that actual salaries may vary within the range above or below, depending on experience and location. We look at compensation for each individual and base our offer on your unique qualifications, experience, and expected contributions. This position may also be eligible for other types of compensation in addition to base salary, such as variable bonus and/or stock bonus.