Amit Ranjan
Development
Dubai, United Arab Emirates
Skills
Data Engineering
About
The candidate's skills align with System Developers and Analysts (Information and Communication Technology). The candidate also has skills associated with Database Specialists (Information and Communication Technology). The candidate has 12 years of work experience.
View more
Work Experience
Senior Data Engineer
Spotify
January 2020 - Present
- Tech Leader in the data team responsible for all the Ubiquity devices of Spotify including Car, Desktop, Smart Speakers. Led the data part of the initiative to generate better content recommendations, which resulted in ~1% uplift in the consumption. The initiative involved leveraging advanced analytics and machine learning to optimise content suggestions, significantly enhancing the overall user experience. Owned a major ML-backed service which serves over 1K req/s impacting 200M users daily. Implemented strategic improvements to elevate the service's coverage and precision, resulting in a notable enhancement for over 60 million Monthly Active Users (MAU), equivalent to 12% of Spotify's entire user base. Pioneered the development and management of strategic, high-quality data products, catering to diverse needs such as reporting, machine learning, and experimentation across the organisation. Ensured these products adhered to rigorous governance standards, contributing to a more agile and data-centric culture. Championed the growth and adoption of a data-centric culture by spearheading impactful data strategy initiatives and delivering targeted training sessions to multiple teams. Achieved a robust Net Promoter Score (NPS), reflecting the positive reception and effectiveness of the data-driven strategies implemented. Optimised Apache Beam/Scio data pipelines, GCS and BQ storage footprints leading to heavy cost savings on cloud infrastructure. Architected and implemented a canonical dataset to give the full view of consumption on Spotify across all devices, which empowered teams with a unified and comprehensive understanding of user behaviour, facilitating more informed decision-making and strategic planning. Technologies: Apache Beam, Scio, GCP, BigQuery, Scala, Python, Luigi, Data Management, Data Strategy
Big Data and Spark Trainer
Various Companies
January 2015 - December 2023
- Instructor in online education platforms upGrad, Simplilearn and AcadGild, to provide industry-relevant programs and training, so that professionals and freshers can develop new, deployable skills in data engineering. Prepared study materials and conducted the live sessions. Delivered data training to hundreds of industry professionals. Reached more than 500,000 learners on YouTube with original content dedicated to Python Created a top-rated course (4.5+ rating) for in-depth, hands-on driven exposure to the features and concepts of Spark Core with tips on tuning its performance. Available at Udemy: Apache Spark Core and Structured Streaming In-Depth Technologies: Hadoop, Hive, Oozie, Airflow, AWS, Pig, Sqoop, Flume, HBase, Spark, Kafka, Advanced
Senior Data Engineer | Careem
Various Companies
January 2018 - December 2020
- Member of Technical Staff responsible for development of a customised, scalable and modular data platform to enable batch and real time access from diverse systems. Developed platform products and capabilities to migrate away from the managed solutions (like New Relic, AWS Kinesis, AWS Glue, AWS Athena), leading to cost savings of over $100K per month. Optimised spark applications and in-house ETL solution to reduce run-time and cost by over 30%. Reduced data availability from 1 day to 15 mins by spearheading real-time processing efforts and seamlessly integrating Delta Lake, resulting in the establishment of a cutting-edge Near Real-Time Data Warehouse (DWH). Built a data platform from scratch to enable collection, transformation and compliant access to company's data. Technologies: Spark, Kafka, Elastic Search, Hive, HBase, Presto, Java, Scala, Python, AWS, Docker, Jenkins,
Data Platform Engineer | Rakuten
Various Companies
January 2017 - December 2018
- Developer in the data platform team aimed at adopting a scalable and modern tech stack. Managed stakeholders' requirements with the best possible strategy. Implemented a modern data platform from scratch. Revamped the existing data ingestion, transformation, access, cataloguing, governance, documentation and observability.
Senior Systems Engineer
Infosys
January 2012 - December 2017
- Bhubaneshwar, India 2012 - 2017 Consulted various clients on adoption of open-source technologies to work with data at scale. Migrated analytical workloads from proprietary tools like Teradata to the on-premise open-source Hadoop ecosystem leading to scalability at a fraction of the original cost. Implemented and customised solutions related to Map-Reduce, Hive, Oozie and Autosys. Developed real time streaming application using Spark Streaming, Kafka, HBase and Hive. Awarded with an MFG Rising Star in 2015 for exceeding customers' expectations and delivering high-quality projects. Technologies: Spark, Kafka, HBase, Shell Scripting, Python, MongoDB, Pig, Java, Quartz scheduler, Hive, Tez, Python, Shell Scripting, S3
Education
