About

Prakhar Jain is a highly skilled Senior Software Engineer with over 7 years of experience in software development. He is currently working at Databricks, where he is passionate about working on databases and distributed systems. Prakhar's latest work revolves around making Hadoop and Spark stack run on cloud platforms such as AWS, Azure, and Oracle BMC in a cost-effective and performant manner. Prior to joining Databricks, Prakhar worked at Microsoft as a Senior Software Engineer, where he focused on making Apache Spark performant, resilient, scalable, and cloud-native. He improved Spark cluster downscaling by building features like RDD Cache decommissioning and Shuffle offloading, and contributed the same to OpenSource. Prakhar also improved Spark TPCDS performance by more than 2X by researching, designing, and implementing multiple SQL optimizations such as Pre-Aggregation, CNF-DNF Predicate pushdown, Better Sort order selection, Join reordering improvements, Inner to Semi join conversion, Smart shuffle key selection, Distinct pushdown, and Adaptive Hash Aggregates in Apache Spark. At Qubole, Prakhar served as a Member of Technical Staff, Spark, where he designed and developed features such as direct writes, parquet metadata caching, executor packing, proactive shuffle data cleanup, recover partition improvements, and smart executor sizing, which improved Spark performance and cluster utilization. He also made SQL optimizations in Spark Catalyst optimizer such as Skew join and UDF pushdown and designed and developed a new Spark Datasource connector to support HiveAcid tables in Spark. Prakhar holds a Bachelor of Technology (B.Tech.) degree in Computer Science from the Indian Institute of Technology, Bombay. He is highly skilled in software engineering, Spark, Databricks, AWS, SQL, Hadoop, and Azure. With 9.04 years of relevant experience, Prakhar is a valuable asset to any organization looking to leverage his expertise in databases and distributed systems.