Exploring inspection reports: hackathon prototype turns into a practical tool

30 March 2022

shutterstock_1954864822.jpg

©James Dalrymple/Shutterstock.com
A primary school teacher in a rural school in Ghana.

Flashback to January 2021 at the time of the first IIEP-UNESCO hackathon – Hacking EDplanning. During 48 hours, a group of six students from a computer science school developed a digital prototype of a tool that could automatically analyze a large volume of inspection reports, using the semantic technologies of artificial intelligence. The project won the hackathon’s Public Prize, marking just the start of this story. Sixteen months later, the actual development of the tool is underway, in collaboration with the research lab of Paris Graduate School of Digital Innovation, Epitech. A few days before the second edition of the Hacking EDplanning, we caught up with them to see how this digital tool can meet the needs of inspectorates and educational administrations across the world.

Imagine a digital tool that enables a one-click analysis, to discover the main trends that emerge from thousands of school inspection reports - and how they evolve over time or by geographical area. This is precisely one of the challenges that IIEP presented to the hackathon participants last year. Several groups took up the subject, based on thousands of inspection reports made available by open access by Ireland. Among them was a team of six students from Epitech, the Paris Graduate School of Digital Innovation. Based on natural language processing (NLP) technologies, their project – called Athena – came out on top, winning the tied Public Prize.

Automating an analysis that would be impossible to do manually

Although the prototype created in 48 hours was not directly functional at the end of the event, the initial results were particularly promising. IIEP then proposed to Epitech Digital Methods Laboratory to continue the collaboration to develop an open source tool, which would be easy to use and accessible for free by any ministry of education and school inspectorate around the world. In the months that followed, Sonico Samedy, a data scientist and former Epitech apprentice, finalized the computer code, the 'thematic model' and the web interface of the tool, under the guidance of the school's research lab and IIEP development team.

This tool makes it possible to identify on a large scale the topics that emerge from inspection reports, when and how key words appear in texts, etc. By visualizing this data on a map, the General Inspectorate can quickly identify areas where certain issues monitored by inspectors are more present than elsewhere.

"The challenges of our hackathons respond to real needs of the education systems we work with. This tool for exploring inspection reports, co-constructed with ministries, will help educational authorities to better monitor the implementation of their interventions, to ensure a more personalised follow-up, or to keep an eye on the most sensitive subjects.”

Amélie A. Gagnon, Head of the Development Team at UNESCO IIEP

A customizable tool for a variety of uses

Salaries, new teaching practices, measures to reduce absenteeism or to promote inclusion –the subjects investigated by inspectors are as numerous as they are diverse. They depend on national priorities, educational context, policies, and more. By aggregating data from the reports, the IIEP tool can provide an overview of teacher support for recent educational reforms. Or it can identify specific needs in different regions or emerging concerns of teachers. "The tool we are developing is not autonomous: iterative work sessions are necessary to control and guide the sequencing done by the code. The machine can feed human analysis and open up new avenues for improvement,” says Amélie A. Gagnon.

picture1.jpg

Christine François, Digital Delegate at the Nancy-Metz inspectorate

"Today, we have a colossal mass of inspection reports: they concern not only teaching staff but also schools, in addition to those issued by the Inspectorate General. Our evaluation work is very qualitative: the French educational institution writes a lot and constantly evaluates itself. The challenge here is to find a way of aggregating all these reports to bring out the main trends. With IIEP, this is the first time I have heard of a digital tool focused on the semantic dimension and vocabulary for the observation of pedagogical practice, beyond quantitative data. Automating the analysis would be extremely beneficial.

We could then cross-reference these trends according to disciplines, or according to the specificities of institutions, to monitor the impact of public policies or anticipate training needs. At my level, this platform would enable me to cross-reference my working hypotheses with the reality of what is observed by inspectors. For example, to what extent has digital education significantly favoured collaborative work, customized learning or playful practices in the classroom? We could find out by exploring the mass of inspection reports.”

"Automating the analysis of inspection reports with a semantic tool would be extremely beneficial to us."

Christine François, Digital Delegate at the Nancy-Metz inspectorate, France

What is natural language processing?

NLP is a branch of artificial intelligence. It is a set of technologies aimed at understanding human language as it is written - or spoken.  By aggregating digitally encoded and anonymized documents, IIEP's tool seeks to capture and track broad trends, and in no way to serve as a controlling function.

As part of the development of the tool, Epitech is currently experimenting with different NLP approaches, such as clustering and Bayesian methods, to refine the relevance of the results. The next step is now to involve education sector staff and future users to co-build and sharpen the platform's functionalities according to the needs of the field, and to make the tool work in real conditions.

"Epitech seeks not only to train technical experts but above all young people who are aware of the impact they have on the social world around them. The effect on the environment of course, but also on everything that makes society work, education in particular," says Nicolas Bourgeois, Director of the Digital Methods for Human and Social Sciences research laboratory at Epitech. "This project with IIEP is completely in line with our approach, which consists of mobilizing technical skills in the service of human and social sciences. In this case, to improve tools for the education sector.”

"Epitech seeks to train young people who are aware of the impact they have on the social world around them."

Nicolas Bourgeois, Director of the Epitech research laboratory

On the flip side, a research project on inspection reports

The technical and methodological development of this innovative tool is linked to a broader IIEP research project. It aims at establishing a conceptual framework and developing case studies to analyze the role of inspection reports as a key instrument for improving the quality of education. "Almost all education systems in the world have a school inspectorate. However, their role and the objectives of the reports they produce are not necessarily very clear," says Carolina Albán Conto, Coordinator of the IIEP-UNESCO Dakar research and development programme.

While the development of a tool is essential to enable inspection systems to quickly extract information to identify problems and develop strategies, in many countries this is not sufficient. One of the challenges of this research is to analyze the institutional capacities of inspection systems. Under what conditions do inspectors' reports really serve to improve the quality of education - and not only to monitor the compliance of schools with standards? Based on this central question, the research project will study the cases of six countries: France, Ghana, Ireland, Morocco, Uganda, and Tanzania. In some of these countries, steps have already been taken to try to align inspection reports with educational quality issues. The conceptual framework and findings of these studies will also inform the functionality and thinking around the next steps in the development of the digital platform.

"Developing a digital tool is essential but not sufficient. We assume that the capacities of inspection systems must be further developed in favour of quality of education."

Carolina Albán Conto, Coordinator of the research and development programme of the IIEP-UNESCO Office for Africa in Dakar

Other ideas from HackingEDplanning 2021 have also been put into practice. Among them, an innovative methodology to calculate and map school-age populations, now used by researchers and development partners, notably through the WorldPop platform.

The second edition of the hackathon will take place from 6 to 8 May 2022, with six new challenges.

Register for Hacking EDplanning 2022