Hiring for Data Engineer-Data Brick
Implement security best practices and data encryption methods to protect sensitive data in both AWS and Databricks, while ensuring compliance with data privacy regulations. Employ Informatica IDMC for data governance and compliance.
Maintain clear and comprehensive documentation of data infrastructure, pipelines, and configurations in both AWS and Databricks environments, with metadata management facilitated by Informatica IDMC. Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and deliver appropriate solutions across AWS, Databricks, and Informatica IDMC.
Identify and resolve data-related issues and provide support to ensure data availability and integrity in both AWS, Databricks, and Informatica IDMC environments.
Optimize AWS, Databricks, and Informatica resource usage to control costs while meeting performance and scalability requirements.
Stay up to date with AWS, Databricks, Informatica IDMC services, and data engineering best practices to recommend and implement new technologies and techniques. Conduct user requirement analysis for the development / implementation of new systems and enhancements to existing systems.Evaluate potential solutions and make recommendations to resolve business problems. Involvement in the system integration testing phase prior to implementation. Plan and coordinate end-user training for any system implementations or enhancements.