Data Engineer_L3

.Hi We are DATA Group and we are searching for the best talent Our goal is to simplify our clients' lives with innovative IT solutions. We operate at global scale and we are expanding to Portugal If you are passionate and have the desire to make the difference, we want to get to know you Join us to be part of this incredible adventure Who are we looking for? YOU Description:Position Summary:We are looking for a Databricks Specialist Consultant with a deep focus on Unity Catalog and Microsoft Purview, responsible for migrating files in Parquet format stored in Azure Storage Account containers to Delta Table format, with the implementation of a layered architecture (Bronze, Silver, Gold) in Databricks. The consultant will also be responsible for integrating these Delta Tables into Unity Catalog with centralised governance via Microsoft Purview, ensuring that the data is accessible and well governed while being used for dynamic reporting in Power BI.Responsibilities:- Expertise in Unity Catalog:- Migrate Parquet files stored in Azure Storage Account to Delta Tables registered in Unity Catalog in Databricks.- Apply granular access control at table, column and schema level using RBAC in Unity Catalog, ensuring compliance and security.- Configure and optimise Unity Catalog to provide centralised governance over all data in Databricks, ensuring that permissions and data lineage are clearly defined.- Advanced integration with Microsoft Purview:- Integrate Unity Catalog with Microsoft Purview for automatic cataloguing, data lineage tracking, and auditing.- Ensure that all data changes, permissions and metadata are visible and auditable via Purview, guaranteeing compliance with regulations such as GDPR and HIPAA.- Implementation of Layered Data Architecture:- Implement Bronze, Silver, Gold architecture in Databricks, with different layers of data for ingestion, transformation and final exposure for reporting.- Create Databricks clusters suitable for each layer, optimising performance and guaranteeing scalability and security in data processing.- Notebook Conversion and Delta Table Optimisation:- Review and migrate existing notebooks that handle Parquet files to use Delta Tables registered in Unity Catalog.- Implement performance optimisations in Delta Tables, using commands such as OPTIMIZE and VACUUM to improve query efficiency and free up space.- Reports and Visualizations with Power BI:- Ensure that data transformed and governed via Delta Tables in Databricks is accessible for real-time reporting in Power BI, using Direct Query to ensure data is always up-to-date.Technical skills required:- Expert in Unity Catalog:- Advanced experience in configuring, managing and optimizing Unity Catalog in Databricks, including access control, security policies and governance