About

Navdeep Agarwal is a highly skilled engineer with over 14 years of experience in designing, developing, and deploying highly scalable data analytics platforms. Currently working as the Lead Data Infrastructure and Engineering at Hotstar, he has a proven track record of building and managing large-scale data platforms. Navdeep's expertise lies in handling and processing web-scale data, microservices, machine learning, key-value stores, asyncio, and more. He has experience in making data lakes, highly efficient Hadoop clusters, data pipelines, and highly scalable web applications. At Hotstar, Navdeep migrated the AWS based data platform stack to an in-house Hadoop based stack, which resulted in a cost reduction of 10 times with better query performance. He also developed real-time state management of 2 billion profiles in HBase at Hotstar scale using Spark. Navdeep was responsible for building a compute isolated Hadoop stack from scratch to a 200 odd nodes cluster with complete automation, which made it possible to scale different teams of different sizes independently. Navdeep's past experience includes working as a Big Data Architect / Co-Founder at DataOrc, where he helped organizations take control of their data and drawing meaningful insights by setting up resilient, cost-effective, and performant data platforms. He was also the Lead Data Engineer at Helpshift, where he designed and developed highly scalable data ingestion pipelines, aggregations platform, reliable API's to expose aggregated data on web-scale data both in streaming as well as batch fashion in the Hadoop Ecosystem. Navdeep holds a Bachelor's degree in Information Systems from Birla Institute of Technology and Science, Pilani. His tech stack includes infra, Hadoop, web, AWS, Spark, ML, microservices, ETL, big data, senior software engineer, data engineering, Azure, and mobile. Overall, Navdeep is a highly motivated and skilled engineer who is passionate about designing and developing things from scratch. He has a strong track record of building and managing large-scale data platforms and is always looking for new challenges to take on.