Project summary

Competence in mathematics has been identified at EU level as one of the key competences for personal fulfilment, active citizenship, social inclusion, and employability in the knowledge society of the 21st century. Concerns about low student performance, as revealed by international surveys, led to the adoption in 2009 of an EU-wide benchmark in basic skills, which states that by 2020 the share of 15-year-olds with insufficient abilities in reading, mathematics and science should be less than 15%. Low achievement in mathematics is a common concern for all European countries. It is an issue associated not only with the effectiveness of teaching and learning, but also with providing an equitable system of education.

An extensive review of research evidence on what works for children with mathematical difficulties has concluded that interventions should ideally be targeted towards an individual child's particular difficulties. Individual support has proven to have significant impact on children's performance, and reporting progress on achievement is important for improving mathematics skills. ICT can be of crucial help here. Digital learning environments used for stepwise solving homework are just as effective as individual tutors, and have the added advantage of giving teachers information about progress of individual students.

The objective of this project is to develop flexible support for detailed diagnostics of mathematical competences of pupils, and to use this in four existing digital testing and practicing environments in mathematics education. The diagnostics summarise the knowledge, skills, and competences of a pupil, and can be used by teachers to identify underachievers. With this information, teachers can direct their attention to help pupils overcome particular difficulties. Pupils can use diagnostic reports to get an accurate understanding of their own performance and progress. Also the system itself can use the diagnostic information for adapting the environment to the level of an individual.

Most digital testing and practicing tools in mathematics education review only a final answer, and do not look at the steps taken to reach the final answer. However, intermediate steps provide essential information for a detailed assessment of student skills. Assessing intermediate steps also corresponds to the pen and paper practice, and is a wish from the mathematics education sector. This project studies the automatic assessment of intermediate steps in mathematics education for domains such as ‘Numbers’ and ‘Relationships’. We develop general feedback and assessment services that can be used by existing digital learning environments. These services give accurate diagnoses of intermediate steps and use these to determine mathematical knowledge, skills, and competences.

The expected results of the project are threefold. (1) We develop innovative technology for calculating detailed diagnostics in mathematics education, offered as an open, reusable set of feedback and assessment services. The diagnostic information is calculated automatically based on the analysis of intermediate steps. (2) The detailed diagnostics are fully integrated into three fullblown, advanced digital environments for practicing mathematics: the DME, Math-Bridge, and Pepite. Also, the Dutch test player Facet, which is developed for the national intermediary diagnostic test for all secondary schools, will be connected to the services in a proof of concept implementation. (3) We design and execute pilot and evaluation studies for assessing the effectiveness of the detailed diagnostics offered by the services in the three environments. The studies involve approximately 375 pupils and 25 teachers at different schools in different countries.

The desired impact is to improve math education and the state of the art in digital testing and practicing environments. The detailed diagnostics let pupils learn mathematics more effectively. The diagnostics support teachers in helping their pupils efficiently, identifying underachieving pupils, and providing individual support where this is most needed. This project can accelerate the introduction of digital tools in mathematics education, and start off a process in which more digital environments for testing, practicing, and learning mathematics make use of a shared set of (feedback and assessment) services. The collaborative design and development of such a shared set of high-quality services, and the innovation that such a collaboration brings, can help to tackle persistent problems in math education across Europe.

The project’s longer term benefits can be substantial because of the already established professional networks of the partners. In particular, we will involve policy-makers, national testing agencies, publishing companies of math textbooks (e.g. a textbook that uses the DME), developers of other learning environments, and associations for mathematics teachers (e.g. Sesamath).

Tasks

Below are some of the tasks that have been developed for the project.

Making a square

Matryoshka doll

Pattern

Magical trick?

Project results

• Assessment software. The software has been released under the terms of the Apache License 2.0, which is a permissive free software license.
• Data set with free-form answers collected in the evaluation studies: please open the README file.
• Selected tasks and competencies and Translations for tasks. Also see some examples of tasks.
• Fine-grained Cognitive Assessment based on Free-form Input for Math Story Problems. In Proceedings of European Conference on Technology Enhanced Learning (EC-TEL) 2018, pages 262-276, 2018. Also see the poster.
• Student Models to Generate Automated Feedback on Intermediate Steps in Solving Mathematical Problems. Abstract for the 14th International Conference on Technology in Mathematics Teaching (ICTMT14), 2019. Also see the poster.

Advise-Me pre-ICTMT14 workshop: A step forward in stepwise automated diagnostics of math tasks

Who: For mathematics educators and math education researchers interested in assessment and digital technology

When: Monday morning, July 22, 2019, 9:30-12:00

Where: Essen, Germany, co-located with the 14th International Conference on Technology in Mathematics Teaching (ICTMT14): the workshop will take place in the Weststadt carree room WSC-N-U-2.03 (Address for navigation systems Thea-Leymann-Str. 9, 45127 Essen)

The Advise-Me project:
The goal of the Erasmus+ Advise-Me project (Automatic Diagnostics with Intermediate Steps in Mathematics Education) is to develop an approach for using ICT for assessing mathematics achievement of pupils using learning environments for mathematics. In particular, we look at fine-grained cognitive assessment of free-form answers to math story problems, which requires determining the steps a pupil takes towards a solution, together with the high-level solution approach used by the pupil. We recognise steps and solution approaches in free-form answers and use this information to update a user model of mathematical competencies. We use the user model to find out for which student competencies we need more evidence of mastery, and determine which next problem to offer to a pupil. We have tested our work on a large dataset for one problem, and have performed multiple pilot and evaluation studies in different European countries.

The workshop:
In the pre-ICTMT workshop we share the results of the Advise-Me project with a mathematics education audience. We present the project goals and its results, including a domain reasoner that collects evidence for mastery of mathematical competencies from free-form input, and Bayesian Networks for updating the user model based on the evidence from student data. We demonstrate the detailed diagnostics that were realised in this project, and show the mathematics environments that use the feedback and assessment services: Numworx’ Digital Math Environment and the Pépite system. Participants of the workshops will get the opportunity to experiment with the feedback and assessment services by trying the mathematics environments. In the workshop, we present preliminary results of the evaluation studies.

Structure of the workshop:

• Project overview: math story problems, evidence collection, user models (Paul Drijvers)
• Preliminary results of the project’s evaluation study (Jesse Koops)
• Experiments in the learning environments (try it yourself)
• Discussion and closing (Bastiaan Heeren)

Workshop organizers:
Bastiaan Heeren (Open University of the Netherlands), Marieke Bor-de Vries (Utrecht University), Paul Drijvers (Utrecht University), Brigitte Grugeon-Allys (Université Paris-Est Créteil), Johan Jeuring (Utrecht University), Jesse Koops (Cito), Sergey Sosnovsky (Utrecht University), Armin Weinberger (Saarland University).

Registration:
Please register by email to secretariaat.mst@ou.nl (provide your name and organization). Attendance is free of charge, but registration is required. The number of places is limited, and we register on a first-come, first-served basis.

Reference:
Heeren, B., Jeuring, J., Sosnovsky, S., Drijvers, P., Boon, P., Tacoma, S., Koops, J., Weinberger, A., Grugeon-Allys, B., Chenevotot-Quentin, F., van Wijk, J., & van Walree, F. (2018). Fine-Grained Cognitive Assessment Based on Free-Form Input for Math Story Problems. In V. Pammer-Schindler, M. Pérez-Sanagustín, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), Proceedings of EC-TEL’2018: 13th European Conference on Technology Enhanced Learning (pp. 262–276). Berlin/Heidelberg, Germany: Springer.

This project has received funding from the European Union’s ERASMUS+ Programme, Strategic Partnerships for school education for the development of innovation, under grant agreement number 2016-1-NL01-KA201-023022.

Advise-Me Multiplier Event

Friday October 19, 2018
Open University Campus, Milton Keynes Building, Lounge
Heerlen, the Netherlands

In the Advise-Me project we develop an approach to using ICT for assessing mathematics achievement of pupils using learning environments for mathematics. In particular, we look at fine-grained cognitive assessment of free-form answers to math story problems, which requires determining the steps a pupil takes towards a solution, together with the high-level solution approach used by the pupil. We recognise steps and solution approaches in free-form answers and use this information to update a user model of mathematical competencies. We use the user model to find out for which student competencies we need more evidence of mastery, and determine which next problem to offer to a pupil. We have tested our work on a large dataset for one problem, and have performed multiple pilot studies in different European countries.

On this day, we will give an overview of the results of the project, and have asked two international experts to give their view on assessment in mathematics. You are very welcome to attend.

External board