Your opportunity
As a Lead Engineer for our Data Platform, you will be responsible for leading the design, implementation, and continuous improvement of our data infrastructure. Your expertise with Apache Iceberg, data lakes, OpenSearch, and distributed query engines like Trino will be instrumental in building a scalable, efficient, and robust platform that supports our analytics and data science initiatives.
What you'll do
- Lead the architecture and development of our Data Platform, ensuring scalability, performance, and reliability.
- Drive the integration of Apache Iceberg to manage large-scale analytic datasets with high concurrency and versioning capabilities.
- Architect and optimize data lakes for efficient data storage, retrieval, and management.
- Implement and fine-tune OpenSearch for search and analytics use cases across large datasets.
- Deploy and manage distributed query engines like Trino to enable high-performance data analysis across various data sources.
- Collaborate with cross-functional teams to understand data needs and deliver solutions that empower data-driven decision-making.
- Establish best practices for data modeling, ETL processes, and data governance.
- Lead performance tuning, benchmarking, and optimization efforts to meet the demands of complex data workloads.
- Stay abreast of emerging technologies and industry trends in the data platform space, evaluating and recommending new tools and approaches.
- Mentor and guide a team of talented data engineers, fostering a culture of technical excellence and innovation.
This role requires
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- Minimum of 8 years of experience in data engineering, with at least 2 years in a leadership role.
- Proven expertise in managing and architecting data lakes and implementing data lake house patterns.
- Strong experience with Apache Iceberg or similar table formats.
- Proficiency with search and analytics engines like OpenSearch or ElasticSearch.
- Hands-on experience with distributed query engines such as Trino, Presto, or Apache Spark.
- Solid understanding of cloud-based data storage and computing services (AWS, Azure, GCP).
- Experience with data pipeline and workflow management tools
- Good understanding of distributed databases and query processing.
Bonus points if you have
- Active contributor to open source communities like apache iceberg and OpenSearch
- Prior experience working on distributed database and query processing.
- Experience with containerization and orchestration technologies (Docker, Kubernetes).
- Certifications in data engineering or cloud platforms.
