Middle Data Engineer (6 months' engagement)

N-iX is looking for a Middle Data engineer for 6 months' engagement. You will act as the independent hands-on engineer and the technical owner of data readiness. The role combines pipeline and query engineering with investigation of source, mapping, hierarchy, and validation issues; coordination of market and provider validation cycles; reconciliation of outputs; and delivery of technically ready datasets to downstream consumers. The Data Engineering team builds and maintains pipelines, while mappings and configurations determine cell/model scope; Snowflake extraction jobs run SQL through Databricks, with Blob Storage, validation, Medallion transformations, Power BI review, and Gold delivery to Ekimetrics. The operating model also requires end-to-end triage across MDFA, CDF, WPP, Data Foundation, Redmill, and market stakeholders rather than treating every discrepancy as a coding defect. Key responsibilities: - Independently design, develop, maintain, test, deploy, and optimize ingestion, validation, transformation, aggregation, data-quality, anomaly-detection, and extraction workflows - Develop and tune Python/PySpark packages, SQL extraction queries, Databricks jobs, ADF pipelines, Delta tables, and source-to-target contracts - Own technical readiness for new cells and refreshes: clarify filters and expected scope, implement or update queries, review mappings/configuration, reconcile outputs, and confirm readiness for business validation - Investigate source, NCID, taxonomy, hierarchy, mapping, naming-convention, aggregation, missing-week, outlier, and performance issues across CDF, MDFA/PFME, APIs, manual files, and specific sources - Coordinate technical validation cycles with Product, CMIA/markets, WPP/MDFA, CDF, other data providers, and Ekimetrics; convert reported discrepancies into actionable owners and technical evidence - Drive complex incident resolution across pipelines, data contracts, access, service principals, secrets, networking, Power BI refreshes, and downstream delivery - Review pull requests and test evidence; guide the Junior engineer and delegate scoped engineering/support work without becoming a people manager - Maintain architecture documentation, interface contracts, repository documentation, runbooks, deployment procedures, and support/escalation guidance - Recommend practical automation, reliability, performance, and maintainability improvements while respecting platform standards and business-validation ownership. Must-have technical competencies: - 5–6 years’ professional engineering experience with independent production ownership - Strong Python, PySpark, and SQL, including complex transformations, query optimization, reusable packages, debugging, tests, and reconciliation - Strong ETL/ELT and batch-pipeline engineering across relational warehouses, APIs, object storage, and file-based ingestion - Experience with a cloud data lake/lakehouse, Medallion patterns, schema/interface contracts, data-quality controls, orchestration, observability, and incident recovery - Solid Git engineering practices: branching, pull requests, reviews, automated tests, deployment controls, and documentation - Proven ability to translate business/data requirements into filters, mappings, transformations, validation rules, and operational workflows - Stakeholder-facing problem solving: explain discrepancies, challenge incomplete requirements, establish technical owners, and drive issues to closure. Nice-to-have: - Strong preference for Azure Data Factory, Azure Databricks, Databricks Jobs/API, Delta Lake, Snowflake, Azure Blob Storage/Data Lake, Power BI, GitHub, and Azure Key Vault - Valuable experience with Pandera or equivalent schema-validation frameworks, Streamlit, Managed Identity/service principals, Azure Communication Services, Managed VNet/Private Endpoints, and Dev/Prod release practices - Experience with PFME/media data, syndicated sales, marketing hierarchies, MMM inputs, or multi-market data onboarding is preferred but can be learned. We offer*: - Flexible working format - remote, office-based or flexible - A competitive salary and good compensation package - Personalized career growth - Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more) - Active tech communities with regular knowledge sharing - Education reimbursement - Memorable anniversary presents - Corporate events and team buildings - Other location-specific benefits *not applicable for freelancers