John Shu
Development
TX, United States
Skills
DevOps
About
John Shu's skills align with Programmers (Information and Communication Technology). John also has skills associated with System Developers and Analysts (Information and Communication Technology). John Shu has 14 years of work experience.
View more
Work Experience
Staff Data/Platform Engineer
CME Group
January 2022 - December 2023
- Skills: Python, Java, C/C++, C#, Go, Bash/Shell, SQL, AWS, Kubernetes, Helm, Docker, Terraform, Ansible, Shipa, Jenkins, ArgoCD, Prometheus, Grafana, Datadog, AWS DMS, Airflow, DBT, EMR, Datapipeline, Glue, Kafka, Spark, Flink, Snowflake, Postgres, RDS, DynamoDB, SNS, SQS, VPC CME Group - Global Derivatives Futures & Option Trading, Jan 2022-Dec 2023- Staff Data/Platform Engineer (Data Governance) Designed, built and deployed container-based architectures, implemented container workloads for managing annotation and attribution for migrated trading data from AWS to GCP. Argo, docker, python, GCP Dataplex EDTS- External Data Transfer Service - Contributed in the development of enterprise level batch processing and streaming framekwork with Spark, Kafka and Flink. Later helped to improved performance and scaling by 40%. Dataplex - Designed and built federated Argo Workflows and Cronjobs to manage data operations in GCP, from ingestion, landing zone, lakes etc.Used Terraform (IaC) to maintain and provision AWS/GCP infrastructures. Building and configuring enablements in Terraform/Terragrunt for end-users of hybrid-cloud CICD platform. Automating Data Catalog entries creation and tagging in GCP Maintained and upgraded Spark Cluster Jobs handling financial data processing, transformations and insight extraction. PySpark, Pandas, Python and other machine learning libraries. Maintained and upgraded Jenkins and Argo Workflow Pipelines for Sustained Scalability and Robustness, with a marked decreased in failure rates Assisted other data teams with DMS architecting and deployment for nightly data migration with scaling using Glue, Redshift, Bigquery, Aurora, DBT, CloudWatch, Splunk, EKS. Artifact management and deployment with Bamboo and Nexus.
Sr. DevOps/Data Engineer
JP Morgan Asset Wealth Mgmt
June 2018 - December 2021
- Deployed and Managed Federated Kubernetes Clusters for Asset Wealth Investment Applications with 60~80 Billion Dollar worth Application Analytics Data - VPC, Multiple AZs, Pub, Priv-Subnets, Sec Grps, ELB, Rte 53, WAF, Shield Built, Upgraded and maintained Data Pipeline and Cluster handling 300GB worth of Data monthly, AWS Lambda, DBT for Data Transformations, Redshift, Glue, Kafka, Spark, EMR, Aurora, EKS for data jobs, etc. Architected and Built Continuous Integration/Delivery (CICD) pipelines using webhooks and triggers (Git, Terraform, Jenkins, ArgoCD, Maven, SonarQube, Snyk, Docker, Nexus, Ansible, Shipa, Helm, Kubernetes, Prometheus and Grafana, AWS EKS) Designed and Built container-based architectures, implemented container workloads for and provided prescriptive guidance about which container orchestration solution to use on AWS, adopt methods to make individual microservices into a sustainable, automated chain and allowing rapid build, test, and release for code while minimizing errors. Building and deploying various micro services in EKS. Creating and maintaining namespaces, config maps, secrets, service, ingress, RBAC in Kubernetes. Terraform (IaC) to deploy, provision and manage AWS/GCP infrastructures, VPC, Multiple AZs, Pub, Priv - Subnets, Sec Grps, ELB, Rte 53. Writing, maintaining, updating (iac) Terraform manifest, Ansible playbooks, and K8s manifest files and Automation Scripts. Implementing Docker and Kubernetes for the runtime environment of the CI/CD system to build, test and deploy. Managed multitier application and configured Route 53 to route traffic to backend application through ELB and Nginx Ingress Controller. Implemented quality gates in SonarQube dashboard to ensure error free codes during build and deployment. Also led and participated in Development, writing unit test cases, Code Review, Build & Release (Python, Java) and maintaining legacy trading applications (C++, C#). Performed package management, upgrades/patches, troubleshooting, maintenance. Monitoring of micro-services and applications using Prometheus and Grafana. Maintained and upgraded event processing architecture patterns with event trigger, logging, analysis, visualization, notification and responding, SNS, Kinesis, Kafka (MSK), ELK Languages: Python, Java, SQL, Go, C/C++, C#, Go, Bash, Shell, PowerShell, Yaml, JSON and Groovy AWS Platform: Terraform, CloudFormation, Lambda, IAM, AMI, S3, VPC, EC2, ELB, RDS, SNS, SQS, SES, Route53, NAT, Route Table, DynamoDB, CloudFront, Service Catalog, Auto Scaling, Trusted Advisor, CloudWatch, Glue, Athena and Sage Python - django, flask, pandas, numpy, scipy, json, requests, sqlalchemy, beautifulsoup, hdf5 and general ETL pipelines and data analytics Java - Spring Tool Suite, Maven, Hibernate, Struts, Monetaboot, MyBatis, ehCache, Intellij. MVC and Microservices Architecture Framework Databases - Snowflake, Postgres, DynamoDB, Redshift, BigQuery, Toad, Microsoft SQL Studio with multiple DB's in UAT, DEV and PROD. Security Scanning - Snyk, SonarQube, OSS, S3, Blackduck, and other Vulnerability Scanners
Data Engineer & Scientist
UTSA
September 2014 - December 2018
- Building/Maintaining AWS Data Infra to Manage ingested Data - Lambda, S3, EMR, Spark, Redshift, DynamoDB, Glue, Kinesis, CloudWatch, DMS, DBT, ETL/ELT processes, Athena, Sage, QuickSight, Tableau, Jasper Reports. Extensively used SQL, Numpy, Pandas, Scikit-learn, Spark, Hive etc for Data Analysis and Model Building. Managed and Maintained DMS Data migrations using AWS DMS, Glue, DBT, Data Pipeline, Redshift, DynamoDB etc. Used Python Data Analytics, Sciences and Engineering to build applications to mine and analyze online data with relations to cyber threats both in dark and surface web. Data from dropbox, medhelp, hackforum as well as over 30 other related online forums have been mined and analyzed. Latest project focused specifically on the investigating the outbreak of ransomware both in dark web and surface web forums and finding creative and innovative methods to curb its spread. All projects use NLP, ETL and EDA techniques. Algorithms include LDA, RNN such as LSTM and CNN. General pipelines deal with stemming, stop-word removal, N-gram generation, topic modeling, building training, querying and optimizing models.
Software Engineer
H.E.B Company
August 2013 - September 2014
- Java, Javascript, PHP, Python, HTML, JSP, SQL, Tomcat, Glassfish, Git, Bash and Shell Scripting, Spring and Struts frameworks and much more. Was Responsible for developing software being used by different teams for the smooth functioning of the organization. One of the Highlights of my tenure was being part of the 3-man team that developed the Primo Picks web application for which commercial aired during Super bowl 2014. All projects involved all round involvement in software development life cycle from requirements analysis to deployment and maintenance.
Software Developer
FBD Co. Ltd
June 2013 - August 2013
- Built online engineering lab-requesting web application with PHP framework. Application accepted requests and Engineers logged in and selected requests to work on and kept logs and updated progress till completion.
Research Associate & IT Support
ICITD Research Center
December 2006 - May 2010
- Performed Data Analysis/ Data Visualizations and Research on the growth and spread of the Internet in developing nation. Analyzed the impact of this growth in areas of Cybercrime, Telemedicine, Tele- Education. Also Managed Logistics and Technical Operations, comprehensive IT support including running, updating and installing recent software packages, and debugging and testing different systems and equipment.
Software Developer
Monsanto
January 2009 - August 2009
- Built and deployed KIOSK (Touch Screen) interactive training software in C#, developed Fire-department and co-ops websites, setup plant wide network of print servers for over 150 printers, general IT supports servicing over 500 employees, developed and updated online employee training courses.
Education
Tulane University
University of Texas San Antonio
M.Sc.
September 2010 - December 2012
