Databricks - Data & AI Platform Engineer / Marine & Energy industry

Databricks Consultant – Data and AI Platform Engineer

Role Description

We are looking for a hands-on Databricks Platform Engineer & Data Engineering Consultant with strong expertise in data platform management and a growing focus on AI/ML operations. The consultant will primarily drive platform stability, infrastructure-as-code, and data engineering excellence, while secondarily supporting agentic and ML workloads.

Our project end-customer is a true global leader in smart technologies and complete lifecycle solutions for marine and energy markets. Agile mindset and innovative thinking are highly valued and this has been the most important factor in their global success.

Responsibilities

Platform & Infrastructure Management

• Design, deploy, and manage Databricks workspaces on AWS following company governance and best practices Manage platform infrastructure using Terraform (workspace provisioning, cluster policies, IAM roles, networking)
• User and group management via Terraform and Databricks SCIM integrations
• Cluster management, cost optimisation, and resource governance

Data Engineering & Delta Lake Architecture

• Design and implement Delta Lake medallion architecture (Bronze / Silver / Gold layers)
• Define naming conventions, schema standards, and data asset governance
• Build and maintain CI/CD pipelines with schema evolution support
• Implement data historisation patterns (inserts, updates, deletes / SCD concepts)
• Integrate multiple data sources including AWS Redshift, S3, and Oracle Cloud

Databricks Platform & Catalog Governance

• Register and federate existing data warehouse assets into Unity Catalog (from Redshift, S3, Oracle)
• Define and enforce catalog standards, roles, and data access governance
• Data sharing configuration via Unity Catalog and Delta Sharing workflows
• Advisory for onboarding new analyst users and teams to the platform

MLOps & Machine Learning Engineering

• Support MLflow experiment tracking, model registry, and deployment workflows
• Assist in setting up MLOps pipelines for model lifecycle management
• Advisory on compute and environment configuration for ML workloads

Agentic & AI Engineering

• Participate in the design and implementation of AI agent frameworks on Databricks (e.g. Mosaic AI Agent Framework, LangChain integrations)
• Support agentic workflows and tool integrations within the data platform
• Advisory for AI enablement, including governance and responsible AI practices

Info

Start: asap

Duration: Until the end of 2027

Language: English

Location: Hybrid (office in Helsinki)

Could this be you? Contact via slack katja_witted or email katja.rannisto@witted.com