About

Arpit Solanki is a seasoned Data Engineer with 5.18 years of relevant experience in solving Data Engineering problems at Atlan. He is currently working as a Backend/Data Engineer at Lightbeam.ai. Arpit is passionate about working on open source technologies and has contributed to PyData's Dask and Presto. Arpit's most recent work at Atlan involved building a Data Quality Platform that allows users to set up checks and calculate metrics such as median, null values, and sensitive data (PII detection credit card phone number, etc.) on their data to ensure only the right data goes for consumption. He used Spark as the processing engine and Kubernetes as infrastructure to build this platform. Currently, Arpit is building a Data Virtualization Platform to support querying over heterogeneous data sources with central authorization and caching for faster querying with less cost. He is using open source projects such as Apache's Calcite, Ranger, Spark, etc. to build it. At Atlan, Arpit has also built data pipelines (ETL) using Apache Airflow, data lakes, and data versioning in lakes using Spark, Presto, and Hive. He has also built developer tooling for Atlan data platform users and plugin-based generic data ingestion framework with change data capture (CDC). Arpit has completed his Engineer's Degree in Computer Science from the Indian Institute of Information Technology Vadodara (IIIT) in 2018. He has a strong technical background in Data Engineering, Backend, Spark, Apache Airflow, Kubernetes, Kafka, ETL, Apache Hive, and infra. If you'd like to work together or just discuss anything related to Data Engineering, drop Arpit an email at solankiarpit1997@gmail.com. You can also find him on Github, Stackoverflow, and Codementor. Arpit has also participated in a Community broadcast interview on Presto.