Data Engineer Officer/Senior Officer

We improve lives

IDB Invest is the private sector arm of the IDB Group and an international financial institution committed to addressing the development needs of Latin America and the Caribbean. Our mission is to support sustainable enterprises and projects that deliver financial results while maximizing social and environmental impact in the region.

IDB Invest operates in 26 countries, structured across four geographic regions and three industry verticals: Infrastructure and Energy, Financial Intermediaries, and Corporates.

In support of a new transformative business model “Originate to Share” our shareholders recently approved a $3.5 billion capitalization. This will allow IDB Invest to significantly scale up its investments, to approximately $19 billion, and unlocking greater private sector participation to drive development and climate impact across the region.

About This Position

IDB Invest is seeking a Data Engineer Officer/Senior Officer with deep expertise in the Databricks platform to join a team of staff and consultants in the data management and analytics office. The data engineer will collaborate on the design and implementation of large-scale data ecosystems on Databricks, including the Lakehouse architecture, data governance via Unity Catalog, and the integration of structured, semi-structured, and unstructured data to generate enterprise-scale insights.

The position requires a highly motivated, technically advanced candidate with extensive hands-on experience in architecting and operating Databricks-based data platforms. The data engineer's primary role will be to serve as a technical reference within the team, setting standards for the Lakehouse, mentoring junior engineers and contractors, and partnering with data scientists, analysts, and business stakeholders to deliver reliable, performant, and governed data products that support decision-making across multiple domains.

Our Team: The Data Management and Analytics Unit is part of the ITA Division, responsible for defining and implementing IDB Invest's data strategy.

What You’ll Do

• Architect, develop, and operate scalable data pipelines and Lakehouse solutions on Databricks (Delta Lake, Delta Live Tables, Workflows, Unity Catalog) to ingest, transform, and serve data to analytics, AI, and BI consumers.
• Lead the design of the medallion architecture (bronze, silver, gold) and define standards for data modeling, partitioning, performance tuning, and cost optimization on Databricks.
• Build and maintain robust ETL/ELT processes using PySpark, Spark SQL, and Databricks-native tooling, integrating sources such as relational databases, Azure Data Lake Storage (ADLS Gen2), Azure Blob Storage, APIs, and streaming systems.
• Implement and operate data governance, lineage, access control, and data quality frameworks using Unity Catalog and complementary tools (e.g., Collibra).
• Collaborate with Data Scientists and ML Engineers to productionize models, including support for feature engineering, MLflow workflows, and serving on Databricks.
• Diagnose and resolve issues in existing data pipelines, modernize legacy workloads, and lead migrations to the Databricks Lakehouse.
• Establish CI/CD, testing, and observability practices for data pipelines using Databricks Asset Bundles, Git integration, Azure DevOps or GitHub Actions.
• Provide technical leadership, code reviews, and mentorship to junior engineers and consultants, and contribute to the continuous improvement of the team's engineering practices.

Looking to advance your Data Science career with relocation support? Explore Data Science Jobs with Relocation Packages that include comprehensive packages to help you move and settle in your new role.

• Partner with business stakeholders to translate requirements into well-architected data solutions and clearly communicate trade-offs, timelines, and outcomes
  
  

What You'll Need

Education: Master's degree in computer science, data science, software engineering, information systems, or related quantitative field. In lieu of a Master's Degree a Bachelor's Degree and the below certifications may be considered:

• Databricks Certified Data Engineer Professional
• Azure Data Engineer Associate (DP-203)
• Azure Databricks Data Engineer Associate (DP-750)
• Databricks Certified Associate Developer for Apache Spark
  
  

Experience: 4-5 years of professional experience as a data engineer, with a track record of delivering production-grade data platforms.

• 4+ years of hands-on experience designing, building, and operating solutions on the Databricks platform, including Delta Lake, Spark Structured Streaming, Delta Live Tables, Workflows, and Unity Catalog.
• Expert-level proficiency in PySpark, Spark SQL, and Python, with a strong command of Spark performance tuning and optimization.
• Strong experience with the Azure data ecosystem (Azure Data Lake Storage Gen2, Azure Blob Storage, Azure Data Factory, Azure SQL, Event Hubs/Kafka.
• Demonstrated experience designing medallion / Lakehouse architectures and applying dimensional and data vault modeling techniques at scale.

Discover our full range of relocation jobs with comprehensive support packages to help you relocate and settle in your new location.

• Proven experience implementing CI/CD for Databricks (Databricks Asset Bundles, Repos, Azure DevOps or GitHub Actions).
• Solid understanding of data governance, security, lineage, and access control; experience with Unity Catalog required, and experience with Collibra or similar tools is a plus.
• Experience supporting AI/ML workloads on Databricks (MLflow, Feature Store, model serving) and BI consumption layers (Power BI, Tableau, or Databricks SQL).
• Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Databricks Certified Machine Learning Professional) are strongly preferred.
• Excellent communication, collaboration, and mentoring skills; comfortable engaging with both technical and non-technical stakeholders.
• Loves thinking out of the box to solve challenges at an unprecedented scale.
  
  

Functional expertise: : [i] You are able to work in a high-pressure environment while juggling multiple tasks, and effectively prioritize work; [ii] you are able to understand and translate clients' (business units) needs into well-architected solutions, providing services that meet clients' expectations for quality and performance; [iii] you listen actively, obtain necessary input, share ideas, speak persuasively, and convey information in a clear, objective, and concise manner; [iv] you provide technical leadership and mentorship, helping to grow the capabilities of the broader data engineering team.

Languages: You are proficient in written and verbal communication in English and in a second official language of the IDB Group (Spanish, Portuguese or French).

Requirements

Citizenship: No requirements

Consanguinity: You have no family members (up to the fourth degree of consanguinity and second degree of affinity, including spouse) working at the IDB, IDB Invest, or IDB Lab.

Type of contract and duration

International staff contract, 36 months initially, renewable upon mutual agreement.

What We Offer

Interested in relocating to United State? Check out our comprehensive Relocation Jobs in United State page with detailed relocation packages and benefits.

The IDB Group provides benefits that respond to the different needs and moments of an employee’s life. These benefits include:

• A competitive compensation package, including an annual base salary expressed on a net-of-tax basis.
• Leaves and vacations: 24 days of paid time off + 8 personal days + sick leave + gender-neutral parental leave.
• Health Insurance: IDB Group provides employees and eligible dependents with a robust medical benefits program that covers medical, dental, vision, preventive care, and prescription drugs.
• Pension plan: defined benefit pension plan that provides financial security and support employees in planning for their future.
• We offer assistance with relocation and visa applications for you and your family when it applies.
• Hybrid and flexible work schedules.
• Health and well-being: Access to our Health Services Center that provides preventive care and health education for employees.
• Development support: We offer tools to boost your professional profile, such as mentoring, 1:1 professional counseling, training and learning opportunities, language classes, mobility options, among others.
• Other perks: Lactation Room, Daycare Center, Gym, Bike Racks, Parking, Spouse Career Program, and others.
  
  

Our culture

At the IDB Group, we work so everyone brings their best and authentic selves to work while finding their purpose. Our people consistently strive for excellence, and we recognize and celebrate the impact of their contributions.

In our efforts to drive innovation, we intentionally include all voices, cultivate a sense of belonging and champion fairness. We welcome individuals from underrepresented groups to join us and share their unique perspectives.

We ensure that individuals with disabilities are provided reasonable accommodations to participate in the job interview process. If you require an accommodation to complete this application, please email us at [email protected].

Our Human Resources Team reviews carefully every application.

#IDBInvest