Senior Data Engineer (Azure Databricks)
UARROW PTE. LTD. Tampines
Design and implement large-scale Azure and Databricks Data Lakehouse solutions.
- Build and optimize ETL/ELT pipelines for batch and real-time streaming workloads using Azure Data Factory, Databricks, and Apache Spark.
- Develop scalable data ingestion frameworks and integrate diverse structured and unstructured data sources.
- Optimize data storage and query performance using Delta Lake, Parquet format, partitioning, and Spark performance tuning techniques.
- Implement robust data governance, security, and access control using Unity Catalog, Azure Key Vault, and least-privilege principles.
- Build and maintain data quality frameworks using Great Expectations or similar validation tools for batch and streaming data.
- Automate pipeline deployment and CI/CD processes using Azure DevOps, Git, and configuration management tools.
- Enable advanced analytics and ML workflows by preparing curated datasets and integrating with ML platforms like MLflow.
- Develop real-time streaming solutions using Kafka, Azure Event Hubs, and Databricks Structured Streaming.
- Create reusable PySpark libraries and frameworks for data curation, reconciliation, notifications, and Delta table automation.
- Migrate legacy Hive or on-prem data sources to cloud platforms while ensuring security, compliance, and performance.
- Monitor pipeline performance, troubleshoot issues, and implement alerting and logging for proactive management.
- Collaborate with cross-functional teams and stakeholders to translate business requirements into scalable data solutions.
- Establish best practices, standards, and documentation for enterprise-wide data engineering and analytics initiatives.
- Mentor junior engineers and promote knowledge sharing within the data engineering team
Trust RecruitTampines
and enabling future AI initiatives.
Job Responsibilities
Data Platform & Engineering
• Build and maintain the Microsoft Fabric data platform for APAC.
• Develop and manage data pipelines from ERP, CRM, and operational systems.
• Design and maintain...
TRUST RECRUIT PTE. LTD.Geylang, 8 km from Tampines
focused on building data pipelines, managing data platforms, supporting reporting and analytics, and enabling future AI initiatives.
Job Responsibilities
Data Platform & Engineering
• Build and maintain the Microsoft Fabric data platform for APAC...
Geylang, 8 km from Tampines
We are looking for a skilled and experienced Data Science Engineer to join our team. The ideal candidate will be responsible for designing, building, and maintaining robust data pipelines to support the processing and analysis of clinical study...