About

Mukesh Kumar is a seasoned Tech Lead and Senior Data Engineer with over 9 years of experience in Crawling and Extraction, Large-scale Data Management, and Distributed Algorithms. He is currently serving as a Senior Data Engineer at DataWeave, where he is responsible for building and managing the distributed crawler infrastructure that crawls 2000+ websites round the clock. Mukesh's specialties include Crawling and Extraction, Data Pipeline Architecture, and Data Analysis. He has extensive experience in managing scale, crawling policies and priorities, and handling frequent changes in website structures. At DataWeave, he manages scale by distributing tasks to many lightweight workers using Celery. He also queues URLs to be crawled based on crawling policies and priorities and makes network requests to get data and persist data content for the URL into the cache system so that extraction process can be run on the top of these data to get actionable data points. Mukesh has worked with several renowned companies in the past, including Marlabs, where he served as a Solutions Architect. He has also worked with DataWeave in the past as a Senior Data Engineer. At DataWeave, he built the distributed crawler infrastructure that crawls 2000+ websites round the clock. Mukesh holds a Bachelor of Engineering degree in Computer Science from Shree Institute of Science & Technology. He has also completed his 10th and 12th from Sainik School Tilaiya and a basic education in Mathematics, Science, and Social Science from Vidya Niketan. Mukesh's tech stack includes Data Engineering, Senior Software Engineer, Infra, Web, and RabbitMQ. With a relevant experience of 12.12 years, Mukesh has proven himself to be a valuable asset in the field of data engineering and distributed algorithms.