Databricks Engineer

Key Responsibilities - Develop and maintain scalable data pipelines using Databricks, Apache Spark, and PySpark. - Build ETL/ELT workflows for batch and streaming data. - Work with Delta Lake and optimize data processing performance. - Develop and optimize Spark SQL queries and data transformations. - Integrate data from databases, APIs, files, and other source systems. - Implement data quality, validation, monitoring, and error handling. - Collaborate with data engineers, architects, analysts, and business stakeholders. - Implement CI/CD and automated deployment for data pipelines. Required Skills - Strong experience in Databricks and Apache Spark. - Proficiency in Python/PySpark and SQL. - Hands-on experience with Delta Lake. - Experience with Azure, AWS, or GCP cloud platforms. - Strong understanding of ETL/ELT and data warehousing concepts. - Experience with Git and CI/CD tools. - Knowledge of data modeling, partitioning, and Spark performance tuning. - Good understanding of data governance and security practices.