About

Rahul Dey is a highly skilled Big Data Engineer with over 4 years of experience in the IT industry. He has a strong understanding of Big Data Technologies, particularly Spark (Core, SQL, Streaming), and has developed highly scalable Data Pipelines hosted on AWS EMR and on-premise Hadoop clusters. Rahul is proficient in programming languages such as Scala, Python, and SQL and has a good understanding of the Hadoop ecosystem, including HDFS, Hive, Sqoop, and Apache Kafka. Rahul has hands-on experience with AWS services such as S3, Glacier, Redshift, Athena, Lambda, Step Function, QuickSight, Elastic MapReduce, Glue, CloudWatch, EC2, RDS, and IAM. He has developed a Central Logging and Monitoring System using the ELK stack (Elasticsearch, Logstash, Filebeat, and Kibana) to monitor data pipelines across different Hadoop Clusters. Additionally, Rahul has developed a Real Estate web application using Python’s Framework Django. Currently, Rahul is a Senior Software Engineer - Big Data at Bosch, where he is responsible for developing Spark applications in Scala for processing streaming data and loading it to Hive. He is also responsible for tuning the performance of Spark jobs to complete data processing within desired time intervals, generalizing the solution such that the jobs work in different clusters, and developing UNIX/python scripts for deployment and scheduling Spark jobs in clusters. Rahul has rolled out and set up data pipelines for Production plants, created and maintained Databases in MPP SQL query engine Impala, and automated response plans for the Incidents raised by the Data Analyst Team. He has also created a monitoring and alerting system using Elastic stack for critical data pipelines and implemented new methodologies to the existing data pipelines to improve performance. Rahul conducts regular knowledge sharing sessions on System Architecture for new stakeholders and collaborates with central teams for tasks and standards. Rahul holds a Bachelor of Engineering degree in Electronics and Communications Engineering from Nitte Meenakshi Institute of Technology. He is proficient in Big Data, Software Engineering, SQL, Data Engineering, Spark, AWS, Hadoop, Python, HDFS, Amazon Athena, MapReduce, Scala, Elastic Search, Apache Hive, Web, Django, and has relevant experience of 6.12 years.