Syed Faisal Abbas
Development
Islamabad , Pakistan
Skills
Data Engineering
Python
SQL
Snowflake
Azure (Microsoft Azure)
AWS (Amazon Web Services)
Azure Databricks
Docker
Apache Airflow
About
SYED FAISAL ABBAS's skills align with System Developers and Analysts (Information and Communication Technology). SYED also has skills associated with Database Specialists (Information and Communication Technology). SYED FAISAL ABBAS has 8 years of work experience, with 5 years of management experience, including a mid-level position.
View more
Acomplishments
One of my notable accomplishments very early into my career was optimizing a critical query for a client that originally took over a day to run. By implementing a series of optimization techniques, including indexing, statistics collection, and query restructuring, I was able to significantly reduce the query's runtime to just a few hours.
First, I analyzed the query execution plan to identify the most time-consuming operations. I then created and applied appropriate indexes to the underlying tables to improve data retrieval efficiency. Additionally, I ensured that relevant statistics were collected and kept up to date to enable the query optimizer to make informed decisions about the query execution plan.
Furthermore, I restructured the query to minimize the number of data reads and unnecessary joins, thereby reducing the overall processing time. By combining these optimization techniques and fine-tuning the query parameters, I was able to achieve a dramatic improvement in query performance, allowing the client to obtain critical insights from their data in a timely manner.
I spearheaded the design and implementation of several complex data architectures that significantly enhanced data processing efficiency and scalability. By leveraging my expertise in cloud technologies and open-source tools, I successfully integrated multiple data sources, optimized data pipelines, and ensured data quality and governance. These architectures not only met the specific business requirements but also provided a solid foundation for advanced analytics and business intelligence.
Work Experience
Principal Consultant Data Analytics
Systems Limited
September 2021 - Present
- Responsibilities: Designing and architecting complex and scalable data systems that meet business requirements and support analytical needs. Leading and providing technical guidance to a team of data engineers, mentoring junior members, and promoting best practices. Collaborating with cross-functional teams, such as data scientists, analysts, and business stakeholders, to understand data needs and translate them into technical solutions. Evaluating and selecting appropriate technologies, tools, and frameworks to optimize data processing, storage, and analytics. Developing and implementing advanced data engineering techniques, such as data streaming, real-time processing, or machine learning pipelines. Ensuring data governance and data quality standards are implemented and maintained across data pipelines and systems. Managing and optimizing cloud-based data infrastructure, such as AWS, Azure, or Google Cloud Platform. Identifying and resolving performance bottlenecks, data integrity issues, and other technical challenges in data pipelines. Conducting performance tuning and optimization of data systems to enhance efficiency, scalability, and reliability. Staying abreast of industry trends, emerging technologies, and advancements in data engineering to drive innovation and continuous improvement. Projects: British Americal Tobacco, UK: o Developing logic to transform raw data into useful, actionable information. o Building, testing, and maintaining database pipeline architectures. These pipelines ensure data flows efficiently from source to destination. o Working closely with management to understand company objectives and align data engineering efforts accordingly to deliver business value. o Performing data analysis to ensure data quality and accuracy. o Ensuring compliance with data governance policies and maintaining data security. o Technologies used: Snowflake, Azure, ADF, Python, Airflow, SQL Regeneron, USA: o Creating data pipelines starting with NiFi for ingestion from source (Flat Files, Database, SFTP etc.) to S3 buckets. Using PySpark to perform the transformations on the ingested data and finally loading it into S3/RedShift to be consumed. Using Athena to query the data present in S3. o Creating and managing Airflow workflows to orchestrate the ingestion and post ingestion processes. o Using Bitbucket for code versioning and Jenkins for CI/CD. o Technologies used: AWS, Python, NiFi, Airflow, Spark, SQL, Jupyter Notebook, Linux, Databricks, EMR
Data Engineer / Team Lead
AFINITI
November 2020 - September 2021
- Responsibilities: • Designing, building, and maintaining data systems and pipelines for efficient and reliable data processing, integration, and storage. • Developing and implementing data architectures, including data models, schemas, and database structures. • Collecting, cleaning, and transforming data from various sources into usable formats for analysis and reporting purposes. • Building and optimizing data pipelines to ensure efficient data ingestion, processing, and storage. • Implementing data quality and validation processes to ensure data accuracy, completeness, and consistency. • Collaborating with data scientists, analysts, and other stakeholders to understand data requirements and develop solutions that meet their needs. • Troubleshooting and resolving issues related to data pipeline performance, data integrity, and system reliability. • Implementing data security and privacy measures to ensure compliance with regulations and protect sensitive data. • Monitoring and optimizing data systems to improve performance, scalability, and cost efficiency. • Keeping up with industry trends, emerging technologies, and best practices in data engineering Projects: AT&T - Mexico, Santander Bank - Mexico, Claro - Columbia o Creation of Data Pipelines to provide data to the AI models using Talend, MySQL, Python & Java. o Deployment of "Red & Blue architecture" that separates the workspace for the two teams to compete for their models. o Calculating Standard Deviation, Lift, Gain, Incremental etc. to finalize the revenue generated by the models. o Identifying and optimizing bad performing queries in the workflows by analyzing statistics. o Proactively creating Python scripts, SQL functions and stored procedures to automate the daily tasks. o Working with the Customer Support team in case of critical system failures and other technical issues. o Monitoring the daily data loading and re-running the failed jobs. o Effectively working with cross-functional teams to ensure operational and service quality.
Data Warehouse Consultant
Teradata GDC
December 2017 - November 2020
- Responsibilities: • Building and optimizing data pipelines to ensure efficient data ingestion, processing, and storage. • Implementing data quality and validation processes to ensure data accuracy, completeness, and consistency. • Collaborating with data scientists, analysts, and other stakeholders to understand data requirements and develop solutions that meet their needs. • Troubleshooting and resolving issues related to data pipeline performance, data integrity, and system reliability. • Implementing data security and privacy measures to ensure compliance with regulations and protect sensitive data. • Monitoring and optimizing data systems to improve performance, scalability, and cost efficiency. • Keeping up with industry trends, emerging technologies, and best practices in data engineering Projects: BonPrix: o Creating ETL workflows for loading the files to stage, then transforming the data as per the client's requirement to core using: BTEQs, TPTs and Shell scripts. o Identifying and optimizing bad performing queries in the workflows by analyzing statistics, creating and evaluating TASM rulesets and workloads. o Processed data from Hadoop to Teradata for reporting via Spark. o Archived data from Teradata to Hadoop using Sqoop & Spark. o Deployment of Query Grid to communicate between Teradata and Hive clusters. o Performing DBA activities such as Space Management, User Management, Access Rights Management. o Managing the DSA backup/restore jobs; Creating, running and scheduling the jobs and investigating and resolving failures. o Proactively creating shell scripts, macros and stored procedures to automate the daily tasks. o Working with the Customer Support team in case of critical system failures and other technical issues. o Monitoring the daily data loading and re-running the failed jobs.
Web Development Executive
Eikon7 (Pvt.) Ltd
August 2016 - November 2017
- Responsibilities: Designing and developing websites for clients in diverse fields such as E-Commerce, Construction, ERP Software, etc. Responsible for all stages of website creation from initial design and architecture to development, deployment and management. Working closely with internal development and design teams to ensure flawless integration with back-end technology and a great user experience. Providing website maintenance and enhancements. Creating web applications that are consistent across all browsers and platforms. Maintain graphic standards and branding throughout the product's interface. Maintaining continuous contact with all internal and external stakeholders through the project and after completion to make sure features are delivered with high quality and on time.
Education
Bahria University
MS Project Management
September 2022 - January 2024
