About

Ayush Verma is a highly experienced Senior Software Engineer at Microsoft with over 8 years of relevant experience in the field. He is currently responsible for building the Aria pipeline, one of the largest big data telemetry pipelines in the world, which processes millions of events per second from billions of devices worldwide and supports the business of many customer-facing products at Microsoft like Windows, Xbox, Office, Bing, and Skype. Ayush has developed scalable backends and distributed systems using a long list of technologies including Java, Python, AWS, RESTful microservices, TDD, Flink, Spark, Spring, and many data stores like PostgreSQL, MySQL, Redis, Elasticsearch, RockDB, S3, and transports like SQS, Kafka, Kinesis, and more. He has also worked as a Senior Software Engineer at Skyscanner, where he was responsible for designing several key components of a new self-sustaining business data pipeline. Ayush has also pioneered solutions to ingest billions of daily events, spread across millions of parquet files and merge them into a data lake, with deduplication, roll-ups, schema evolution, real-time enrichments, and a resilient deployment with automatic recovery and horizontal scalability. He has also designed and delivered a highly scalable solution for converting protobuf messages of varied, evolving schemas to parquet format, by introducing custom extensions to Flink and proto-parquet writers and utilizing multiplexed stream processing on AWS Kinesis with Kinesis streams. Ayush has a Bachelor of Engineering (B.E.) degree in Computer Engineering from Netaji Subhas Institute of Technology (NSIT). He has a strong grasp of various technologies, including AWS, microservices, Java, data engineering, Spark, big data, Redis, MySQL, Elastic Search, Kafka, Spring, and Python. Overall, Ayush is a highly skilled and experienced software engineer with a proven track record of delivering scalable and robust solutions to complex data engineering problems.