About

Nikhil K. is an experienced Data Engineer with a strong background in building and maintaining data pipelines. He is currently working at Epsilon as a Big Data Engineer, where he is responsible for building both streaming and batch data pipelines to support 300B records per day. Nikhil has a proven track record of improving ORC file and Kafka compression, thus reducing HDFS Disk footprint by 3 times using techniques like tuning linger and batch size, Sticky Partitioner, Sorting high cardinality columns, among others. He is also skilled in troubleshooting and debugging data pipelines on a rotation basis. In his current role as a Software Engineer ll at Epsilon, Nikhil is building a framework to migrate Map Reduce jobs to Spark. He has also built a Kibana Dashboard for Spark Job metrics using ELK stack. Nikhil's past experiences include working as a Cloud Data Engineer at Hashmap, where he was responsible for integrating On-prem Data Sources (Oracle, Netezza) and Cloud datasources (Google Analytics, S3, APIs) with Snowflake. He also wrote Matillion ETL jobs and supported a range of business consumption patterns. At Lam Research, Nikhil built a Spark batch application for parsing complex JSON files, ingesting data into HBase, and creating HDF5 files using Java. He also built a Python library for loading, querying, and performing analytical operations on time-series HBase data using SQL-like syntax. Nikhil has a Bachelor of Engineering degree in Information Technology from Army Institute of Technology, Pune. He is skilled in various technologies, including Spark, Kafka, HDFS, Snowflake, SQL, AWS, ETL, Big Data, Java, Python, Apache Airflow, Hadoop, and Apache Hive. With 4.70 years of relevant experience, Nikhil is a highly skilled and dedicated Data Engineer who is always looking to improve and optimize data pipelines.