Senior Data Engineer

Role Overview We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role covers data ingestion, processing, storage, orchestration, and delivery to Analytics and BI teams within an AWS-based data ecosystem. Responsibilities - Design and maintain batch and real-time data pipelines. - Integrate data from APIs, databases, files, and external sources using Apache Kafka and AWS services. - Develop and orchestrate workflows with Apache Airflow, including monitoring, retry mechanisms, and alerting. - Build and optimize distributed data processing solutions using Apache Spark, PySpark, Spark SQL, and AWS EMR. - Manage Data Lake and Data Warehouse environments using Amazon S3 and Amazon Redshift. - Design analytical data models, including Star and Snowflake schemas. - Prepare and optimize datasets for Analytics and Business Intelligence use cases, particularly for Qlik. - Develop high-quality solutions using Python and SQL, following best practices for Git, code reviews, testing, and documentation. - Ensure data quality, security, governance, and compliance, including AWS IAM, encryption, and access control. - Monitor production data pipelines and proactively address operational and performance issues. Mandatory Requirements - Proven experience as a Senior Data Engineer or in a similar role. - Advanced proficiency in Python and SQL. - Hands-on experience with: - Apache Spark / PySpark - Apache Kafka - Apache Airflow - Strong AWS experience, particularly with: - Amazon S3 - Amazon Redshift - AWS EMR - Experience with Data Lakes, Data Warehouses, ETL/ELT processes, and data modeling. - Strong understanding of batch and streaming architectures. - Experience with performance optimization, Git, testing practices, and production support. Technical Stack - Python - SQL - Apache Spark / PySpark - Apache Kafka - Apache Airflow - AWS EMR - Amazon S3 - Amazon Redshift - Git - Qlik Originally posted on Himalayas