About

Harshad Hegde is a Senior Data Engineer at Byju's with over 4 years of relevant experience in the field of Data Engineering. His expertise lies in building scalable data pipelines, data governance, and data quality frameworks. He is proficient in various technologies such as Python, Spark, SQL, AWS, Kafka, and Snowflake. At Byju's, Harshad is responsible for building a Data Platform with the goal of centralizing organization-wide data onto Snowflake. He owns and manages the Snowflake account and builds batch data pipelines using PySpark (AWS Glue jobs) for data ingestion. He also builds near real-time data pipelines from Postgres, MySql, and MongoDB to Snowflake using Kafka Connect. Harshad implements Data Governance on the data platform, including data classification, tagging, and dynamic masking (Snowflake), and cataloging using DataHub. He has also built a data quality framework that ensures consistent data quality across all data sources in the platform. His tech stack includes Python, SQL, PySpark, Kafka, Kafka Connect, Snowflake, and various AWS services such as Glue, S3, Redshift, RDS, Kinesis, Lambda, EC2, ECS, and MSK. Prior to Byju's, Harshad worked as a Data Practitioner at Brilio, where he built scalable data pipelines for ingesting data from RDBMS systems to a data lake. He migrated data from Legacy Systems to AWS S3 to facilitate querying from AWS Athena and Redshift. Harshad developed an Automation Test Framework to test Data Quality at all the layers of the data lake. He also worked on performance tuning of Spark jobs to process large data sets efficiently. His tech stack at Brilio included Snowflake, AWS, SQL, Kafka, Python, Spark, ECS, AWS Glue, MySQL, MongoDB, Postgres, Amazon Athena, HDFS, ML, and Apache Hive. Harshad holds a Bachelor of Engineering degree in Computer Science & Engineering from Nitte Meenakshi Institute of Technology.