National Consultant to Conduct a Comprehensive Technical Assessment of the ''Mandukhai'' chatbot

How you can make a difference:

UNFPA is the lead UN agency for delivering a world where every pregnancy is wanted, every childbirth is safe and every young person's potential is fulfilled.  UNFPA’s strategic plan (2022-2025), reaffirms the relevance of the current strategic direction of UNFPA and focuses on three transformative results: to end preventable maternal deaths; end unmet need for family planning; and end gender-based violence and harmful practices. These results capture our strategic commitments on accelerating progress towards realizing the ICPD and SDGs in the Decade of Action leading up to 2030. Our strategic plan calls upon UN Member States, organizations and individuals to “build forward better”, while addressing the negative impacts of the Covid-19 pandemic on women’s and girls’ access to sexual and reproductive health and reproductive rights, recover lost gains and realize our goals.

In a world where fundamental human rights are at risk, we need principled and ethical staff, who embody these international norms and standards, and who will defend them courageously and with full conviction.

UNFPA is seeking candidates that transform, inspire and deliver high impact and sustained results; we need staff who are transparent, exceptional in how they manage the resources entrusted to them and who commit to deliver excellence in programme results.

Purpose of consultancy:

The purpose of the consultancy service is to conduct a comprehensive technical assessment of the Mandukhai chatbot, evaluating its functionality, performance, and scalability, while also identifying opportunities for enhancements to its current capabilities.

Scope of Work:

Background:  With the support of UNFPA Mongolia, the Facebook-based Mandukhai chatbot was launched in 2021, to deliver accurate and confidential information on sexual and reproductive health (SRH) to adolescents. This AI-powered bot was designed to increase access to SRH information and services and empower its users with the correct knowledge and informed decision-making. Since its installation at the National Center for Maternal and Child Health (NCMCH) and the appointment of a live agent for troubleshooting, Mandukhai demonstrated promising results, achieving an accuracy level of 62% in June 2021, and through ongoing enhancements to its functionality, the chatbot's accuracy improved to 73% by the end of 2022. 
Since its launch, there has been limited information available regarding its performance and level of accuracy in the response of the Mandukhai chatbot including user experience. Additionally, no formal evaluation has been conducted on the bot’s compliance with the data security requirements as per the Mongolia ICT Sector Medium-Term Development Policy (2022-2027), and health-related ethical considerations. Recent advancements in AI technology have the potential to significantly enhance the performance of AI-powered chatbots, even in complex interactions involving sensitive information. Given this potential, the evaluation will also focus on the aspects that the chatbot has “learned” since it was launched. 
Therefore, UNFPA Mongolia is seeking to hire a national individual consultant (IC) to perform a comprehensive technological assessment of the Mandukhai chatbot. The evaluation will utilize specific criteria designed for chatbot performance metrics, focusing on accuracy in response rates, overall user experience, and responsiveness to user needs. To capture a broad spectrum of users’ experiences, the assessment will include a diverse group of users, particularly adolescents from remote areas, ensuring the dataset is representative and inclusive.

Key outputs and activities expected to respond to the objective of the consultancy: 

1. Develop the evaluation design: The consultant will be responsible for conducting a comprehensive desk review of existing documentation and researching best practices and standards for chatbot evaluation metrics. This includes studying use cases of advanced AI-powered chatbots in the medical field. By gathering background information and understanding the objectives, design, and any previous evaluations of the chatbot, the consultant will develop an evaluation matrix. This matrix will feature defined performance metrics derived from past conversation logs, including accuracy or effectiveness, engagement, user interaction, operationalities such as fallback and escalation, and error and context awareness metrics. 

2. Usability Testing: Usability testing is essential for evaluating the chatbot, aimed at ensuring that it is intuitive, efficient, and satisfying for users. The consultant will conduct one-on-one sessions to observe participants as they interact with the chatbot, completing specific tasks. Immediate feedback will be collected through observation, post-session questionnaires, or brief interviews. It is important to recruit adolescents from varied demographics, including differences in age, gender, education level, and geographic location, to participate in the usability testing to ensure comprehensive and relevant results.

3. Scalability and performance testing are crucial for evaluating how well the chatbot manages numerous queries or simultaneous conversations, providing insights into its scalability. Performance testing should encompass the chatbot's depth of understanding of the Mongolian language, AI algorithms, and conversation flow and ability to handle tones, proficiency in handling context, multi-turn conversations, and variations in user input, memory-related functions such as recalling the previous conversation and generating of dashboard and statistics. These tests will help determine the chatbot's strength and readiness to function effectively under varied and demanding usage scenarios.

4. Organize stakeholders’ consultation meetings to validate the findings of the chatbot evaluation and improvement plan with recommended actions for further enhancement of the bot level of AI functions.

5. Provide a final consultancy report: The consultant will provide the final consultancy report of the evaluation for further improvements and required updates to the chatbot based on user interaction and scalability testing results. 

Duration and working schedule:

The duration of this consultancy is expected to be 33 non-consecutive days within July 15 –Sep 10 in Ulaanbaatar, Mongolia. The consultant’s work will be based on deliverables, and it is not tied to the working hours.

Place where services are to be delivered:

The consultant will work from her/his workplace. 

Delivery dates and how work will be delivered (e.g. electronic, hard copy etc.):

The selected consultant is required to provide the following deliverables to ensure that the chatbot meets the desired standards for functionality, user satisfaction, and overall effectiveness. 

1. Desk review report along with the bot evaluation matrix, by July 26, 2024.  

2. A draft findings of the bot usability and performance testing analysis, by August 20, 2024. 

3. National validation workshop report with presentations (findings and Scalability Plan) with recommendations, by September 1, 2024 

4.  A draft consultancy report in both English and Mongolian for stakeholders' feedback including UNFPA Mongolia, by September 5, 2024. 

5. Final reports for the UNFPA approval, by September 10, 2024

 All deliverables will be submitted in English and Mongolian and shall follow the structure and detailed outlines agreed with the UNFPA Country Office. 

Monitoring and progress control, including reporting requirements, periodicity format and deadline:

The consultant should work in close consultation with the UNFPA SRHR team and provide progress updates through weekly email as per the schedule jointly confirmed upon the start of the consultancy.

The reporting requirements, formats, and deadlines will be refined during the first briefing session.

Expected travel:

Local travel is expected to 2 provinces and 2 UB remote districts. For local travel outside of Ulaanbaatar, UNFPA will arrange transportation and Daily Subsistence Allowance (DSA) according to its travel policy. 

Required expertise, qualifications and competencies, including language requirements: 

Education:  

• Postgraduate degree in computer science, data science, mathematics, or statistics 

Knowledge and Experience: 

• At least 3-5 years of professional experience as a program or software developer including an understanding of natural language processing (NLP) and machine learning (ML) algorithms and Experience with bot development platforms. 
• Proficiency in analyzing user interactions and feedback to improve bot functionality and user satisfaction.
• Proven experience with analytics tools to monitor bot performance user engagement and survey design. 
• Proven experience in troubleshooting and resolving issues.
• Awareness of the ethical implications of bot interactions, especially in sensitive areas such as health.
• Experience in working with the United Nations and engaging with government and ministries, as well as CSOs, is desirable. 
• A proven track record of producing high-quality reports and policy recommendations.

Languages: 

• Mongolian native speaker
• Technical level of English (both in reading and writing), including the ability to review professional materials.

Competencies: 

• Exemplifying integrity and accountability. 
• Excellent teamwork skills.
• Coordination and facilitation skills.
• Adequate level of organizational, time management, and communication skills.
• Ability to work under pressure with strict timelines. 
• Knowledge of human rights, sexual and reproductive health, gender equality, population, and research in general.

Inputs / services to be provided by UNFPA or implementing partner (e.g. support services, office space, equipment), if applicable:

Logistics: 

• It is anticipated that the consultancy will be governed by an Individual consultancy contract under standard UNFPA General terms and conditions for Individual consultants.
• The UNFPA does not require him/her to work in the UNFPA office. However, weekly meetings with a team are required for the timely consultation and guidance needed.
• The consultant will use his/her computer. If necessary, the consultant can use the office printer and scanners.

Schedule of payment:

• 30% of the payment will be paid upon completion and submission of deliverables 1 & 2 accepted by UNFPA. 
• 70% of the payment will be paid upon completion and submission of deliverables 3,4, and 5 accepted by UNFPA.

UNFPA reserves the right to withhold up to 30% of the total fee in the case that the deliverables are not submitted on schedule or do not meet the required standard. Copyright and ownership of all documents produced will remain with UNFPA.

UNFPA Work Environment:

UNFPA provides a work environment that reflects the values of gender equality, diversity, integrity and healthy work-life balance. We are committed to ensuring gender parity in the organization and therefore encourage women to apply. Individuals from the LGBTQIA+ community, minority ethnic groups, indigenous populations, persons with disabilities, and other underrepresented groups are highly encouraged to apply. UNFPA promotes equal opportunities in terms of appointment, training, compensation and selection for all regardless of personal characteristics and dimensions of diversity. Diversity, Equity and Inclusion is at the heart of UNFPA's workforce - click here to learn more.

Disclaimer:

Selection and appointment may be subject to background and reference checks, medical clearance, visa issuance and other administrative requirements. 

UNFPA does not charge any application, processing, training, interviewing, testing or other fee in connection with the application or recruitment process and does not concern itself with information on applicants' bank accounts. 

Applicants for positions in the international Professional and higher categories, who hold permanent resident status in a country other than their country of nationality, may be required to renounce such status upon their appointment.