Senior Data Engineer

Job Description

We are a team of tech enthusiasts on a mission to bring together the best minds in IT services and analytics. Our goal? To build cutting-edge IT and Analytical Hubs that empower our partners to become truly data-driven organizations.For our client’s Data Management & Data Foundation area, we are looking for an exceptional Senior Data Engineer – a highly experienced, hands-on engineer who can independently take ownership of complex technical challenges and turn them into robust, scalable solutions.This is not a maintenance-focused or business-as-usual role. You will join a high-seniority engineering group working on strategic and technically challenging initiatives, moving between projects depending on priorities and where the strongest engineering expertise is needed.The wider Data Management organization consists of approximately 30 engineers and technology specialists, while this role will focus primarily on a selected set of high-impact initiatives.Your roleAs a Senior Data Engineer, you will:Design, develop, and optimize complex data solutions and data pipelines on Azure and Databricks.Take end-to-end technical ownership of assigned engineering problems – from understanding the challenge through solution design, implementation, testing, and production deployment.Write high-quality, production-grade code using Python, PySpark, and SQL.Work extensively with Azure Databricks and modern capabilities of the Databricks ecosystem.Explore and implement AI-enabled capabilities within the data platform, including solutions involving Databricks AI features, RAG-based approaches, data discovery, and conversational access to enterprise data.Contribute to the migration and modernization of existing data platforms and solutions into Databricks-based architectures.Work with Unity Catalog and contribute to scalable approaches to data discovery, governance, and access.Solve technically difficult or non-standard problems where there may not be an existing blueprint or ready-made solution.Proactively identify opportunities to improve architecture, performance, scalability, reliability, and maintainability.Challenge existing approaches when you see a better technical solution and actively contribute your own ideas.Collaborate with Data Engineers, Architects, Product Owners, DataOps, QA, DevOps, and other technology teams.Support architectural discussions and contribute to technical design decisions across the Data Management landscape.Move between strategic initiatives as priorities evolve, bringing strong engineering expertise wherever it is most needed.This is a role for someone who enjoys building things, solving difficult engineering problems, and having a real technical impact rather than simply executing predefined tasks.What we are looking forMust-have skills8+ years of professional experience in Data Engineering, Software Engineering, or a closely related field.Strong, hands-on commercial experience with Azure Databricks.Very good knowledge of the Azure cloud ecosystem and cloud-based data architectures.Excellent programming skills in Python – we are looking for a strong coder, not someone who only uses Python for basic notebook development.Advanced PySpark skills and experience working with large-scale data processing workloads.Advanced SQL, including complex transformations, optimization, and analytical/window functions.Strong understanding of data engineering principles, ETL/ELT patterns, data pipelines, distributed processing, and production data platforms.Experience designing and delivering solutions independently, with limited supervision.Ability to understand an unfamiliar technical problem, propose an approach, and drive it through to implementation.Strong troubleshooting and problem-solving skills.Experience working with production-grade engineering practices, including version control, testing, CI/CD, monitoring, and deployment processes.Strong understanding of software development principles and the ability to produce clean, maintainable, well-structured code.High level of ownership, independence, and proactivity.Excellent English communication skills, both written and spoken.Particularly valuableWe will be especially interested in candidates who also bring:Hands-on experience with AI capabilities within Databricks and modern AI-enabled data solutions.Experience with RAG architectures, LLM-based applications, or conversational interfaces built on enterprise data.Experience with Databricks Genie or similar conversational data discovery capabilities.Strong knowledge of Unity Catalog.Experience migrating enterprise data platforms or data management solutions to Databricks.Understanding of data cataloguing, metadata management, lineage, governance, and data discovery.Strong awareness of solution and data architecture and the ability to contribute meaningfully to architectural decisions.Experience designing reusable frameworks, components, or engineering standards rather than only individual pipelines.Perf