Prashanthi
Ramamurthy
Senior Engineering Manager at Adobe
"Supercharge your Dynamic Data Transformations at Scale with Apache Spark"
"Supercharge your Dynamic Data Transformations at Scale with Apache Spark"
24 - 5:50 PM EDT/New York (See in local time)
Add to Calendar
04/24/2024 5:50 PM
04/24/2024 06:10 PM
America/New_York
#WTGC2024
"Supercharge your Dynamic Data Transformations at Scale with Apache Spark"
#WTGC2024
"Supercharge your Dynamic Data Transformations at Scale with Apache Spark"
https://app.events.ringcentral.com/events/women-in-tech-global-conference-2024/reception
https://www.womentech.net/vfairs
"Supercharge your Dynamic Data Transformations at Scale with Apache Spark"
"Supercharge your Dynamic Data Transformations at Scale with Apache Spark"
Don’t miss out and join visionaries, innovators, and thought leaders from all over the world at the Women in Tech Global Conference.
Vote by Sharing
Unite 100 000 Women in Tech to Drive Change with Purpose and Impact.
Do you want to see this session? Help increase the sharing count and the session visibility. Sessions with +10 votes will be available to career ticket holders.
Please note that it might take some time until your share & vote is reflected.
Session: Supercharge your Dynamic Data Transformations at Scale with Apache Spark
This talk outlines our development of a scalable data transformation pipeline suitable for both batch and streaming scenarios. We'll discuss the challenges and solutions encountered while scaling to handle hundreds of terabytes daily, shifting from row-level transformations in Apache Spark to using Catalyst Expressions for complex, nested column transformations and DSL-based custom functions.Key Takeaways
- Building Dynamic Data Transformation pipelines for Big Data Processing
- Leveraging Spark internals and Spark Catalyst optimizations to achieve complex nested data transformations at scale
- Trade-offs and learnings from our journey
Bio
Engineering leader at Adobe with extensive experience in leading and architecting scalable, web applications and big data systems. In my 16+ years of experience, I have successfully delivered multiple projects from concept to customers. I am passionate about solving complex problems, designing data models and providing robust and scalable software solutions with customer focus. I am deeply committed to mentoring and actively participate in women in tech forums at various universities. Empowering and uplifting women in the tech industry is a cause close to my heart.