About

Sunil Pandey is a Microsoft Certified Azure Data Engineer Associate Professional with over 7 years of experience in the IT Industry, with a focus on Big Data Engineering. He has expertise with various tools in the Hadoop Ecosystem, including Hive, HDFS, Sqoop, Spark, Kafka, Yarn, Oozie, and Zookeeper. Sunil has in-depth understanding of Spark architecture, including Spark Core, Spark SQL, Dataframe, and Spark Streaming. He has imported data from AWS S3 into Spark RDD and performed transformations and actions on RDD's, using Avro, Parquet, and ORC data formats to store data in HDFS. Sunil has extensive experience in working with Amazon Web Services (AWS) using EC2, EMR, RDS, S3, and Redshift. He has hands-on experience in working with Azure Cloud and its components like Azure Data Factory, Azure HD Insights, Azure Data Explorer, Event Hub, Azure Data Lake Storage V2, CosmosDB, and Databricks. Sunil is also experienced in Data Warehousing ETL concepts using Informatica Power Center, OLAP, and OLTP. Sunil has optimized existing algorithms in Hadoop using Spark Context, Spark SQL, Dataframe, and Pair RDD's. He has processed complex/nested JSON and CSV data using Dataframe API. Sunil has built Azure Data Factory pipelines using Python SDK and built Azure EventHub and Stream Analytics architecture fulfilling the near-real-time analytics requirement. He has also built Azure Data Factory pipeline using Azure Databricks for transformation storing into Azure Synapse Analytics. Currently, Sunil is working as a Lead Software Engineer at Wells Fargo. In the past, he has worked as a Senior Data Engineer at Tech Mahindra and as a Senior Associate at Societe Generale. Sunil possesses a wide range of technical skills, including Spark, Azure, SQL, Data Engineering, Software Engineer, Senior Software Engineer, Kafka, HDFS, Python, Cassandra, AWS, Hadoop, Databricks, Scala, Apache Hive, Web, Big Data, ETL, test, Javascript, Jupyter, Selenium, dotNET, ASP.NET, Microservices, and Search. With almost 10 years of relevant experience, Sunil is a highly skilled and experienced professional in the field of Big Data Engineering and Data Warehousing.