Senior Databricks Developer

ai talent • Australia

Visa Sponsorship

This Job is No Longer Active This position is no longer accepting applications

AI Summary

Lead the design and development of large-scale data pipelines using PySpark/Scala within Databricks. Architect and manage the Delta Lake environment. Define and enforce data governance practises within the Lakehouse.

Key Highlights

Lead technical leader for Analytics and Research workloads

Architect and build highly scalable data pipelines

Ensure strict data governance and security for sensitive patient data

Searching for Development & Programming roles that provide visa sponsorship? Connect with international employers through Development & Programming Jobs with Visa Sponsorship opportunities actively seeking talented professionals.

Key Responsibilities

Lead the design and development of large-scale, resilient, and performant ETL/ELT data pipelines using PySpark/Scala within Databricks notebooks and jobs.

Architect and manage the Delta Lake environment, focusing on data ingestion, quality enforcement, and schema evolution for complex public health datasets.

Optimise Databricks clusters, notebooks, and Spark jobs for cost-efficiency and performance.

Define and enforce data governance practises within the Lakehouse, utilizing Unity Catalogue for centralized metadata and access control.

Collaborate closely with government analysts and data scientists to transition analytical models and research findings into scalable, production-ready pipelines.

Champion CI/CD and MLOps practises for Databricks notebooks and workflows, utilizing tools like Azure DevOps or Jenkins.

Mentor and guide junior engineers on Databricks development standards, Spark optimisation, and modern data engineering practises.

Explore our comprehensive directory of visa sponsorship jobs from employers worldwide who are ready to sponsor talented international professionals.

Technical Skills Required

PySpark Scala Delta Lake Databricks Azure SQL Dimensional modelling principles

Interested in opportunities specifically in Australia? Discover our dedicated Visa Sponsorship Jobs in Australia page featuring roles from top employers in this location.

Benefits & Perks

482 visa sponsorship

Job Description

We're partnering with a vital leader in the Government Health sector, dedicated to optimising public health outcomes, resource allocation, and policy analysis through a modern, cloud-native data platform. For the right candidate with the necessary skills and experience, we are pleased to offer 482 visa sponsorship.

Want the full job description? Read the complete details on LinkedIn, the original posting.

Continue on LinkedIn

This is a short excerpt. All rights to the full description belong to its original publisher.