Nehal Syed
Development
CA, United States
Skills
Data Engineering
About
Nehal Syed's skills align with System Developers and Analysts (Information and Communication Technology). Nehal also has skills associated with Programmers (Information and Communication Technology). Nehal Syed has 18 years of work experience, with 4 years of management experience, including a mid-level position.
View more
Work Experience
SR DATA ENGINEER
Knott Solutions
May 2023 - February 2024
- Design, development and operationalization of a reusable and modular data & analytics platform using GCP cloud services that enables efficient, consistent, automated value generation from data. • Leads development and execution of highly complex and large-scale data structures and pipelines to organize, collect and standardize data to generate insights and addresses reporting needs • Implemented and managed the ingestion process and export of Healthcare FHIR data into Big Query warehouse, ensuring seamless data flow and accuracy. • Utilized GCP Dataform for applying transformations to enhance the quality and structure of the data. • Collaborated with cross-functional teams to enable the consumption of data by AI models, dashboards, and other analytical tools. • Utilized Terraform to build and maintain the data platform infrastructure.
SR DATA ENGINEER
Accenture
August 2022 - May 2023
- Led the technical migration of analytical pipelines from Cloudera and Hortonworks on-premises environments to AWS, utilizing various AWS analytical and self-managed services. Conducted in-depth analysis of the existing architecture, resulting in a comprehensive high-level design and migration plan for transitioning standalone Spark ETL pipelines to Kubernetes microservices. • Designed and developed scalable Big Data analytical pipelines using Apache Spark and Amazon Kinesis for both stream and batch processing, facilitating efficient data ingestion into the data lake. • Leveraged Infrastructure as Code (IaC) principles to automate the creation and management of application infrastructure such as Kinesis, EMR, and Kubernetes using Terraform and Ansible. • Established orchestration and deployment pipelines using Docker and Jenkins, ensuring efficient and consistent deployment of microservices and analytical components. • Successfully migrated and deployed the Apache Hive metastore onto Amazon using Glue Data Catalog, enhancing data discoverability and accessibility. nehalsyed@gmail.com LinkedIn
Accenture
September 2019 - May 2023
SENIOR DATA ENGINEER
Accenture
May 2022 - August 2022
- Collaborated with cross-functional teams to learn the intricacies of workloads and queries on Azure Cosmos DB data warehouse, leading to effective performance and cost optimizations. Collaborated with stakeholders to gather requirements and align technical solutions with business objectives, contributing to successfully implementing strategic initiatives. Provided mentorship and technical guidance to team members, fostering a culture of continuous learning and innovation. • Gained a comprehensive understanding of the pipeline architecture and design of trading systems, enhancing insights into data flow and transactional processes. • Identified areas for improvement in data warehouse efficiency and recommended solutions, resulting in a 20% reduction in query response time and 15% reduction in costs • Designed and implemented a cutting-edge streaming trade feeds data ingestion pipeline into the bank warehouse, utilizing Azure Functions and Azure Event Hubs for seamless data consumption and replication. • Demonstrated proficiency in Azure cloud services, ensuring real-time data delivery while maintaining data integrity and security.
PRINCIPAL DATA ARCHITECT
Accenture
March 2020 - April 2022
- Led the GCP Solution Architecture team, helping customers Migrate analytics services to GCP and operate in hybrid mode. Created, maintained, and governed architectural views and blueprints depicting the Business and IT landscape in its current, transitional, and future state. Worked with client technical teams to develop transition roadmaps and architecture patterns to migrate legacy systems to modern cloud architectures. Defined migration factory team processes, roadmaps, and methodologies. Oversaw all teams actively migrating under the big data cloud migration factory. • Led technical discussions with customers and technical teams to understand business and technical requirements to develop and tailor cloud-based solution architectures. • Led the evaluation, and rationalization of existing services (analytics, distributed storage, orchestration, workflows, compute, security) to support the hybrid and multi-cloud environment. • Developed Spark code in Scala using Spark SQL & Data Frames for aggregation. • Defined reference architectures, design and integration patterns, and implementation standards. • Elaborated cloud strategy to develop comprehensive architecture and action plans for cloud implementation. • Secured and instrumented cloud systems to meet strict performance, security, audit, and compliance requirements. • Developed a training program for the internal PSO and client teams to facilitate the migration journey. • Led efforts and hands-on to develop assets like code, FAQs, processes, tutorials, quick start guides, notebooks, etc. • Evaluated organization processes and tools, establishing/updating operational processes to ensure adherence to best practices, compliance, and security requirements in a cloud environment. • Worked with the platform team and oversaw the building of modules that automate and simplify the standing up of big data environments in the cloud.
SR DATA ENGINEER
Accenture
September 2019 - February 2020
- Collaborated closely with data scientists, analysts, and business stakeholders to comprehend requirements, resulting in streamlined data processing workflows and actionable insights. Architected a high-scalable and cost-effective data ingestion solution utilizing Google Cloud Pub/Sub, Apache Airflow, and Google Cloud Storage. • Utilized performance tuning techniques, reducing data processing time by 25% and enhancing overall system efficiency. • Led the design and development of a sophisticated data pipeline to analyze call logs for call anomaly detection, resulting in a 15% reduction in fraudulent activity. • Created a robust microservices-based ETL pipeline using Apache Beam, featuring autoscaling functionality that dynamically accommodated increased containers on the cloud platform. • Designed and implemented comprehensive data cleaning pipelines encompassing transformation, enrichment, PII masking, aggregation, and storage, leveraging cloud services for optimal efficiency. • Orchestrated pipeline workflows using Apache Airflow, crafting DAGs (Directed Acyclic Graphs) for ephemeral pipelines that prioritized maximum cost savings while ensuring data integrity.
Rafay Systems
January 2018 - September 2019
DATA & PLATFORM ENGINEERING LEAD
Rafay Systems
September 2018 - September 2019
- nehalsyed@gmail.com LinkedIn Developed and built centralized telemetry data collection, transport, ingestion, etl services using Amazon Web Services (AWS) and Open Source. Worked with product managers, designers, and engineers in the team to prioritize, plan, and schedule work. System integration and managing the technical relationship with customers. • Developed and implemented streaming data pipelines with Apache Kafka, Kinesis to consumer applications, storage systems, and databases. • Implemented a range of AWS services, including Amazon EMR, Kinesis, Firehose, Lambda, S3, DynamoDB, RDS, and Redshift, to streamline data processing, storage, and analysis. • Worked with the other engineering teams to identify and anticipate changing requirements and opportunities to improve the development process and increase the quality of deliverables. • Created data collection and aggregation pipelines on remote servers using Fluend and Prometheus • Architected and developed reliable, scalable, and performant distributed systems and data pipelines. • Created detailed design documents and presentations for the team decision.
SRE ENGINEERING LEAD
Rafay Systems
January 2018 - September 2018
- (SRE) principles to build highly reliable Cloud and Kubernetes infrastructure, manage and operate at scale, solve technical problems, and automate operational tasks. • Architect and implement solutions that ensure high availability, fault tolerance, and disaster recovery. • Lead design and deployment of monitoring, alerting, and logging systems to detect and respond to performance bottlenecks, outages, and anomalies using prometheus operator, grafana, opsgenie, ELK, fluentd, influxDB, pagerduty
SENIOR CLOUD DATA PLATFORM ENGINEER
Comcast
October 2016 - January 2018
- Partnered with customer organizations to understand their platform challenges and serve as a subject matter expert in cloud infrastructure, performing design reviews and consulting with internal teams to ensure design best practices. Provided leadership in platform resilience techniques across multiple zones and regions. Identified the top cloud data architecture solutions to successfully meet the company's strategic needs. • Designed and developed a secure and organized cloud-based data Platform as a Service. • Architected and automated highly available, scalable, secure compute and cache services to support data streaming. • Led and defined technical standards, governance, and compliance for the cloud platform. • Built, versioned, and managed infrastructure as code using tools, safely and efficiently. • Investigated quickly, performed troubleshooting, and resolved technical issues. • Performance-tuning underperforming systems by identifying root causes and mitigating factors.
Comcast
May 2015 - January 2018
SENIOR CLOUD DATA ENGINEER
Comcast
May 2015 - October 2016
- Spanned full engineering lifecycle from architecture and design, data analysis, automation, software development, QA, capacity planning, and managing the analytics environment. Wrote and interpreted complex queries toward data-driven business solutions. Performed offline analysis of large data sets using components from the Hadoop ecosystem like MapReduce and Spark. Evaluated big data technologies and prototype solutions to improve data processing architecture. • Architected and built distributed, performant, scalable, and reliable data pipelines that ingest device usage and process data at scale and in real time. • Implemented data pipelines that support experiments to answer targeted questions that solve known issues by actively engaging with business partners to understand and prioritize key business challenges. • Implemented pipelines to automate reporting dashboards utilizing established data warehouse structures that track and visually communicate key business metrics and provide actionable insights. • Developed and maintained data engineering best practices and contributed to Insights on data analytics and visualization concepts, methods, and techniques.
SENIOR SOFTWARE ENGINEER
ASSYST Consulting/Cricket Media
March 2008 - February 2015
WEB & APP DEVELOPER
T3 Training & Technology Team
January 2006 - December 2007
Education
Sir Syed University
B.S.
