Senior Databricks Developer - Lakehouse & Distributed Analytics

ai talent • Australia

Visa Sponsorship

This Job is No Longer Active This position is no longer accepting applications

AI Summary

Seeking an experienced Senior Databricks Developer to design, build, and optimize large-scale data processing pipelines and lakehouse storage layers for an enterprise client. Key responsibilities include implementing Medallion architectures, deploying production-grade pipelines with PySpark and Delta Lake, and ensuring Unity Catalog governance. Requires extensive commercial experience with Databricks, Spark/PySpark, Delta Lake, and strong Python/SQL skills, with current Australian work rights or eligibility for 482 visa sponsorship.

Key Highlights

Design and build high-performance data processing pipelines and lakehouse storage layers.

Implement Medallion architectures (Bronze/Silver/Gold) and ensure Unity Catalog governance.

Requires extensive Databricks, Spark/PySpark, Delta Lake experience and Australian work rights/visa eligibility.

Searching for Development & Programming roles that provide visa sponsorship? Connect with international employers through Development & Programming Jobs with Visa Sponsorship opportunities actively seeking talented professionals.

Key Responsibilities

Design, build, and maintain production-grade batch and real-time streaming data pipelines using Databricks, Apache Spark / PySpark, and Delta Lake.

Implement scalable Bronze, Silver, and Gold architectural layers, ensuring optimal data partitioning, liquid clustering, and ACID transactional reliability.

Enforce enterprise-grade security, data lineage, and fine-grained access policies using Databricks Unity Catalog.

Optimise Spark execution plans, manage memory configurations, eliminate data skew, and fine-tune warehouse/cluster compute to drive cost efficiency.

Develop automated CI/CD deployment pipelines for Databricks notebooks, modular libraries, and workflows using Databricks Asset Bundles (DABs), Azure DevOps, or GitHub Actions.

Collaborate directly with client data architects, analytics engineers, and machine learning specialists to deliver clean, curated data models for downstream BI and AI consumption.

Technical Skills Required

Databricks PySpark Delta Lake

Explore our comprehensive directory of visa sponsorship jobs from employers worldwide who are ready to sponsor talented international professionals.

Benefits & Perks

Visa and sponsorship support (482 On-Hire, New 482, Temporary & Working Visa Holders)

Opportunity to work on large-scale distributed analytics

Interested in opportunities specifically in Australia? Discover our dedicated Visa Sponsorship Jobs in Australia page featuring roles from top employers in this location.

Nice to Have

Databricks Certified Data Engineer Professional credential.

Experience integrating Databricks with dbt (data build tool) or workflow orchestrators (Airflow).

Cloud platform experience across Azure (Azure Databricks) or AWS.

Exposure to streaming architectures using Apache Kafka, Event Hubs, or Spark Structured Streaming.

Job Description

We are partnering with an enterprise client to accelerate their modern lakehouse roadmap and advance large-scale distributed analytics. We are seeking an experienced Senior Databricks Developer to represent our organisation and take technical ownership of designing, building, and optimising high-performance data processing pipelines and lakehouse storage layers.

Want the full job description? Read the complete details on LinkedIn, the original posting.

Continue on LinkedIn

This is a short excerpt. All rights to the full description belong to its original publisher.