About

Archit Soni is an experienced lead data engineer with over 11 years of experience in big data development and engineering. He has a demonstrated history of working on mission-critical, fast-paced, and agile projects. Archit is skilled in various technologies such as AWS, Databricks, Hadoop, Python, SQL, Spark, Spark Streaming, Kafka, PySpark, Azure, and TIBCO Spotfire (BI / Data Visualization tool). Currently, he is working as a Lead Data Engineer at Marketcast, where he is responsible for building a data platform on a scale of HOLLYWOOD! He is working on a social engine data pipeline using Python, AWS Lambda, AWS Eventbridge, AWS Batch, and AWS Kinesis that retrieves data from different social media platforms in raw format, extracts relevant attributes, and ingests them in S3 as JSON format. Another pipeline runs that takes the JSON files and ingests it into Elasticsearch using AWS Lambda and AWS Kinesis Firehose. In the past, Archit has worked with Tibco as a Senior Technical Consultant and Technical Consultant Lead - PSG. He has also served as a Tech Lead - Data Engineering for Mercedes AMG F1 Team, where he created a Big Data solution that serves as a single source of truth for data for all departments in Mercedes. His solution reduced manual effort of finding files and then running analysis on them to just creating their query in their custom GUI and getting results either in CSV, Matlab, or Parquet formats. The same query can be seen in real-time in Spotfire as well to further enhance the EDA and Predictive analytics. The business value of this project is to reduce the time it takes for Analysts to run their models, hence saving costs up to £15MM annually. Archit has pursued a B.Tech in Information Technology from Dr. A.P.J. Abdul Kalam Technical University and a Data Science Masters Program from Edureka. He has also completed his Senior Secondary education in PCM IT from DAV Public School. Archit has a diverse tech stack, including AWS, Data Engineering, Spark, Python, Databricks, Hadoop, SQL, Big Data, Azure, HDFS, Elastic Search, Kafka, ECS, NoSQL, Kubernetes, Docker, DevOps, Amazon Athena, CI/CD, Redis, Apache Hive, Snowflake, AWS Glue, test, ML, Data Scientist, Research Scientist, Integration, Senior Software Engineer, and infra.