In your role as a Data Platform Engineer, you'll guide the development of mission-critical Azure solutions with a focus on Databricks implementation. You'll join a project where we're building a Databricks-centric platform that helps multiple teams work together effectively. This platform builds upon our recently established Azure Cloud environment that uses a modular approach with shared Terraform modules.
What you'll be working on: We've set up a modern Azure Cloud Platform for a new customer. The platform already runs its first workloads, and we're now starting a project to add a Databricks layer that will let multiple customer teams use Databricks together while maintaining proper governance and security.
As a Data Platform Engineer specializing in Databricks, you'll:
- Design and implement a Databricks platform (supporting 50+ users across 5-10 teams) that meets different teams' unique needs
- Set up Unity Catalog at scale for proper data governance, including permission models and access controls
- Turn customer needs into architecture that addresses security, compliance, and operations
- Develop reusable Terraform modules that teams can use to deploy consistent, secure Databricks workspaces
- Integrate Azure data components like SQL Database, Data Factory, Data Lake Storage Gen2, and Cosmos DB
- Create monitoring, logging, and configuration management solutions
- Bridge the gap between Databricks users and engineering teams, translating business needs into technical solutions
- Research and implement tools for data lineage, quality, and management
You will be part of a diverse, cross-functional team, which:
- Works directly with the customer in a collaborative partnership
- Is exceptionally focused on security and reliability
- Automates every aspect of repetitive work
- Provides space to pioneer, share, lead, and shape our growing Databricks practice
What your typical week might look like:
- Talk with customer stakeholders to understand their evolving data platform needs
- Develop and improve Terraform modules for Databricks resources
- Solve performance and cost issues in Databricks workspaces
- Share Databricks best practices with teammates
- Check out new Databricks features and see how they might help the platform
- Join design sessions to evolve the data platform architecture
- Document your designs and operational procedures
Beyond your project team, you'll join our vibrant internal community where we share knowledge, exchange experiences, and support one another. We provide advanced internal training programs covering the latest technology and modern ways of working to achieve high performance in complex IT environments.
This role is perfect for you if you have:
- Eight years of experience with Azure cloud environments
- At least two years of recent Databricks experience in multi-team settings (10+ users)
- Hands-on experience with Unity Catalog and data access controls
- Strong Terraform skills to create modular, reusable components
- Knowledge of common Azure data components
- Experience building secure data platforms
- Curiosity to dig into technical details while keeping the big picture in mind
- The ability to listen to business needs and turn them into technical solutions
- Good communication skills for working with both technical and non-technical people
Nice to have:
- Databricks Azure Platform Architect or Databricks Platform Administrator certification (or willingness to get certified—we'll fully fund and support this)
- Experience as a Data Engineer with Databricks
- Background in building reusable platform components
- Azure expertise in one of the following areas: Azure security architecture and practices, Azure networking concepts and implementations, working with Azure Policies and governance
This is an opportunity to take the lead while working with an established team. If you enjoy diving into technical details while keeping the big picture in mind and can effectively share your knowledge with the broader team, we'd love to hear from you.
Everyone we work with and consider working with has a right to equal treatment. The hiring and appraisal process at Schuberg Philis is designed to be thorough and equitable, implementing fair payment, benefits, and opportunities across all demographics.
It is our desire to be a company that brings together multiple nationalities, cultures, religions, genders, abilities, and talents within and across our teams. We welcome colleagues from diverse backgrounds to join Schuberg Philis and actively support diversity and inclusion in the tech industry.
