Luxoft company
🔔 Are you already in Poland? If YES this is the project for YOU!
Our benefits:
👩⚕️ Private Medical Care in Luxmed and Life Insurance
🏋️♀️ Multisport Card
👨👧👦 Paid referrals
📚 Self-learning libraries
🛫 Relocation package for seniors and assistance during all process...and MORE!
👉 Location: KRAKOW - hybrid (3 days/week from the office )
Project Description:
We are embarking on an exciting project that requires expertise in a diverse range of technologies. This project aims to enhance our systems' efficiency and performance through the implementation of cutting-edge tools and methodologies. As part of our team, you will be responsible for deploying and maintaining critical infrastructure components, streamlining development workflows, and optimizing system performance. This project offers an opportunity to work with state-of-the-art technologies and contribute to the evolution of our IT landscape.
Responsibilities:
1. Data Engineering and Processing:
• Utilize Databricks (or equivalent platforms) to develop and maintain data engineering solutions.
• Write and optimize SQL queries and Python scripts for data processing and analysis.
2. Data Modeling and Architecture:
• Design and implement data models, ensuring they align with data warehousing principles and Lakehouse architecture.
• Develop and maintain data warehouses to support business intelligence and analytics.
3. ETL Processes:
• Design, develop, and manage ETL pipelines using Azure Databricks (Spark, Spark SQL, Python, SQL).
• Implement ETL/ELT design patterns to ensure efficient data extraction, transformation, and loading.
4. Spark-Based Data Processing:
• Leverage Spark and Python for large-scale data processing tasks.
• Develop and optimize data processing workflows using Spark.
5. Real-Time Data Processing:
• Implement real-time data processing solutions using Delta Live Tables and Spark Streaming.
• Ensure timely and accurate processing of streaming data for real-time analytics and applications.
6. Workflow Design and Optimization:
• Design data processing workflows that are scalable and maintainable.
• Continuously monitor and optimize workflows to improve performance and efficiency.
7. Collaboration and Communication:
• Work closely with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
• Communicate effectively with stakeholders to ensure alignment on data engineering initiatives.
Mandatory Skills Description:
1) Hands on Data Engineering Experience using Databricks (or equivalent) , SQL and Python skills(as a script)
2) Solid understanding of data modelling, data warehousing principles and Lakehouse architecture
3) Expert knowledge of ETL using Azure Databricks (Spark, Spark SQL, Python, SQL) and understand the ETL/ETL design patterns
4) Proficiency in Spark-based Python for data processing.
5) Strong experience in data modeling and designing data processing workflows using Spark.
6) Familiarity with Delta Live Tables and Spark Streaming for real-time data processing.
Nice-to-Have Skills Description:
a) Understanding of ETD (Extract, Transform, Load) product and lifecycle.
b) Reconciliation experience
