Data Engineer

Role: - Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics. - Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers. - Carry out data modelling in Databricks. - Expose the Delta tables to the visualisation layer. - Ensure that the data pipelines are optimised. Skillsets: - Expert in Apache Spark Structured Streaming and Databricks Auto Loader. - Fluent in PySpark and Databricks SQL. - Proven experience building Databricks architecture patterns. Familiarity with Unity Catalog for data governance and access control.