MS Fabric Data Engineer

Deadline for Applications: 22 April 2025, 18:00 (CET)

As part of a broader Digital Transformation Accelerator Program, CGIAR System Organization is transitioning its data infrastructure to Microsoft Fabric. This includes migrating to industry-standard data engineering practices in MS Fabric, including modern architecture based on Medallion architecture, deployment pipeline, GIT integration, and others.

CGIAR has already set up a standard and solid architecture in MS Fabric and built a solution to collect same data domain (personnel) records from different federated data sources to curate them and stack them in a data Warehouse. This solution requires ongoing improvements and requires us to integrate additional data sources to ensure it is in a functional format.

CGIAR seeks a Data Engineer consultant with hands-on experience in MS Fabric to lead the improvements of the existing implementation, but also should help design, implementation, and migrate additional data assets and data solutions to this new architecture.

Scope of Work

·       Enhancements and fixes to existing data solutions based on dataflows and notebooks to collect, curate, correct, stack, and assign global unique ID to stacked records.

·       Drive the adoption and implementation of standardized data engineering practices within Microsoft Fabric.

·       Migrate existing data assets (Power BI dataflows, datasets, warehouses, dashboards) into the new MS Fabric based architecture.

·       Ensure clear separation between data backend and frontend layers using the DEV, TEST, and Production workspaces and medallion architecture (Bronze, Silver, Gold layers).

·       Establish robust deployment pipelines for Fabric artifacts (e.g., Lakehouses, Dataflows, Datasets, Dashboards).

·       Support capacity monitoring, diagnostics, and tuning by leveraging MS Fabric capacity metrics and best practices.

 Qualifications and Experience

Essential:

·       Proven experience in data engineering within Microsoft Fabric ecosystem including Python Notebooks, Warehouses, Lakehouses, Data Pipeline, Deployment Pipeline, and others.

·       Deep understanding of data quality concepts and how to apply them in MS Fabric using Python / Pandas-based Notebooks.

·       Strong understanding and implementation experience of the Medallion architecture pattern.

·       Demonstrated ability to migrate and refactor Power BI assets (dataflows, datasets, dashboards).

·       Proficient in DAX, SQL, and Power Query (M).

·       Familiarity with source control (e.g., Azure DevOps, GitHub) and structured deployment practices.

·       Proficiency in English Language.

Desirable:

·       Experience with MS Fabric capacity planning and monitoring.

·       Familiarity with Fabric Admin roles and workspace governance.

·       Understanding of Microsoft Purview or other data governance tools.

·       DP600 certification or similar.

·       Experience in capacity building / training to bring other technical members to the same level of expertise.

Deliverables

1.       Enhanced and fixed issues on the personnel database solution including the generation of global unique ID, and the inclusion of 6 additional data sources

2.       All existing workspaces are migrated to new architecture (dataflows, dashboards, and warehouses) from legacy workspaces (based on PowerBI capacity) to new workspaces (based on Fabric Capacity) including dataflows, datasets, and dashboards.

3.       Implemented deployment pipelines and Git integration for versioning.

4.       Recommendation on improvements on Fabric utilization based on capacity monitoring dashboard/report and usage analysis.

5.       User guides, technical documentation, and knowledge transfer materials and delivery (training).

Performance Measurement

1.  Key Performance Indicators (KPIs)

·       Documentation quality: Thorough and clear documentation.

·       Training satisfaction: Satsifaction rating from the training delivered to end users.

·       Working solution: Solution is working automatically and free of bugs.

·       Timeliness: The consultant will be evaluated on their ability to meet deadlines and milestones as outlined in the timeline.

2.  Performance Review

Weekly meetings with the consultant will be carried out to review project plan, obstacles, and adjustments to the timeline. During the meetings, feedback will be provided to steer performance and prioritize objectives if needed.

Evaluation Criteria 

·       Certification in MS Fabric (e.g. DP-600)

·       Years of experience with MS Fabric data engineering

·       Years of experience with data science and Python

·       Years of experience with data management and data quality controls using MS Fabric notebooks

Confidentiality

As a condition of engagement with CGIAR the Data Engineer (the “Consultant”) acknowledges and agrees to the following confidentiality obligations relating to the access, use, handling, and protection of personal data and other sensitive information:

1. Confidential Information
   The Consultant understands that, during the course of engagement, they may have access to confidential, proprietary, and sensitive information, including but not limited to:
• Personal data as defined under applicable data protection laws (e.g., GDPR, DPA 2018);
• Technical and business information relating to systems, software, datasets, infrastructure, and security;
• Any information, documentation, or datasets marked or reasonably understood to be confidential.
2. Use and Disclosure
   The Consultant agrees to:
• Use confidential information solely for the purpose of fulfilling their duties and responsibilities;
• Not disclose, share, copy, transmit, or otherwise make available any confidential information to unauthorized persons within or outside the organization, either during or after the period of engagement;
• Handle personal data strictly in accordance with internal policies, contractual obligations, and applicable data protection laws and regulations.
3. Data Protection Compliance
   The Consultant commits to:
• Adhere to the organization’s Data Protection Policy, Acceptable Use Policy, and Information Security Protocols;
• Complete mandatory data protection and information security training as required;
• Promptly report any actual or suspected data breaches, unauthorized access, or misuse of personal data.
4. Return or Destruction of Information
   Upon termination of egagement, or upon request by the organization, the Consultant shall return or securely destroy all confidential and personal data in their possession, in accordance with organizational policies and procedures.
5. Survival of Obligation
   These confidentiality obligations shall survive the termination of the Consultant’s engagement for any reason and remain in effect until such confidential information becomes publicly available through no fault of the Consultant.
6. Breach and Remedies
   Any breach of this confidentiality obligation may result in disciplinary action, including termination of consultancy, and may lead to civil or criminal liability. The organization reserves the right to pursue legal remedies to protect its interests.

All applications must be submitted online by clicking the 'Apply' button below. If you require assistance or face challenges in submitting your application, please email smo-bidding@cgiar.org with the position title in the subject line. Please note we will not accept applications through this email. 

Please ensure that your resume and cover letter are in English and do not contain your marital status, age, or photograph. Documents provided in a language other than English will not be considered.

CGIAR is committed to fair, safe, and inclusive workplaces. We believe that diversity powers our innovation, contributes to our excellence, and is critical for our mission. We offer a multi-cultural, multi-color, multi-generational, and multi-disciplinary, collegial working environment. We consciously create an inclusive organization that reflects our global character and commitment to gender equity. We, therefore, encourage applicants from all cultures, races, ethnicities, religions, sexes, national or regional origins, ages, disability status, sexual orientations, and gender identities.

All received applications will be acknowledged; however, only shortlisted applicants will be contacted. 

We look forward to hearing from you!