Ganesh Shankar
Development
Banglore, India
Skills
Data Engineering
About
GANESH SHANKAR's skills align with Programmers (Information and Communication Technology). GANESH also has skills associated with System Developers and Analysts (Information and Communication Technology). GANESH SHANKAR has 11 years of work experience.
View more
Work Experience
Senior Data Engineer
Stillfront Group
October 2023 - Present
- Germany from GCP Cloud Fusion Pipeline to DBT/Snowflake. Implemented the outlined project plan, starting with the migration from the traditional source of GCP CloudSQL MySQL to the AWS S3 target. Worked on the analysis of existing data pipelines. Worked on the bug fixes of existing pipelines. Worked on the migration of existing ETL jobs to DBT based jobs using Snowflake, Airflow, DBT.
Senior Data Engineer
FREE NOW
May 2022 - October 2023
- Germany using PySpark, Databricks and Airflow, AWS. • • Managing the platform using terraform, CI/CD pipelines • Design & Implemented unnecessary table cleanup DAGs & generate alerts to slack • using databricks, Airflow Worked on UDF using JAVA and calculation of metrics of databricks job using • • overwatch, promethus, Grafana.
Senior Data Engineer
NCS GROUP Pte Ltd
June 2021 - April 2022
- Singapore • S3, RDS, Azure blob, ADLS, ADF, databricks, cloud-composer, GCS, Big query Worked on migration project from Oracle to Big Query • • Worked on managing infrastructure of GCS services such as bucket, service • account, Big query, cloud-composer using Terraform Worked to ingest data using post REST API and publish data to Pub/Sub, do • • transformation using cloud functions and dump it BigQuery Created Ranger policy to give access to AD groups and users to hive tables and • • views
Senior Associate
Ernst & Young Advisory Pte Ltd
April 2019 - May 2021
- Prepare Pyspark job to move data from one layer to another layer.Transformation •Singapore, Singapore • using Spark based on the business use cases and write the output to another layer. Involved in Production maintenance and daily health check up of cluster and • • daily jobs
Senior Data Consultant
Accenture Pte Ltd
December 2016 - April 2019
- Singapore • and continuous client engagement for offshore team. Technology stack- Spark, Scala, Collibra, Java, Mainframe, S3, Airflow •
Software Engineer
Zaloni Technologies
June 2014 - November 2016
- India • Wrote hive queries
Associate Software Engineer
Sigma Infosolutions
January 2013 - December 2013
Education
