Bhutan had registered late for the Programme for International Student Assessment (PISA)-D Pre-Test III.

This gave the education ministry and the Bhutan Council for School Examination Assessment (BSCEA) limited time to prepare, adapt questions in the Bhutanese context, to train coordinators and to code test instrument.

It was for these reasons that BSCEA and education ministry requested that PISA-D Pre-Test III  be dropped as a success indictor in its Annual Performance Agreement.

At the Thimphu dzongkhag and Thromde principals’ meet with the education minister last week, BCSEA’s education monitoring officer (EMO) Sonam Lhamo said that as per the Organisation for Economic Cooperation and Development’s (OECD) protocol, the country should have registered for PISA-D test three years ago.

She said other eight participating countries had already completed Field Trial (FT) several months ago and were almost done with data analysis by the Education Testing Service (ETS).

But Bhutan due to late registration received its orientation on PISA-D test administration during the National Project Manager’s meeting in New Jersey, USA in August 2017.

ETS had developed one of these PISA-D programmes to conduct field trial test (termed as Pre-Test III in Bhutan) to familiarise and provide hands-on experience to the test administration.

This would provide experience on the procedures of collecting data in such international assessment so that the country would conduct the main survey smoothly after having acquired experiences from the FT.

“The other countries had signed with PISA-D in 2013, which gave them enough time to prepare, complete the Pre-Test III and submit the data as well,” Sonam Lhamo said. “But Bhutan didn’t get time to prepare due to late registration and Bhutanese officials had to attend the meeting in Jersey as passive listeners when other countries shared their problems.”

Sonam Lhamo said ETS had conducted several professional development programmes such as trainings and meetings on the administrative procedures for the execution of PISA-D activities, which Bhutan missed it.

“The conduct of FT was left to the discretion of the National Project Centre (NPC) of Bhutan by the ETS, the responsibility of which fell on BCSEA,” she said. “It was not only late registration that we knew about after going to New Jersey, but also found out that the test was for class VII-VIII and not class IX, as targeted.”

She added that initially, as per the National Modal Grade, only class IX students of Bhutan were introduced to PISA-D instruments to familarise them on PISA items.

However, ETS maintained that Bhutan has to abide by their standard sampling protocol of having inclusive cohort of 15-year-old students ranging from grades VII-XII.

“This completely changed the PISA-D sample of Bhutan,” she said, adding that BCSEA considered 15-year old as students from class IX in Bhutan.

The time was also taken on adaptation of questions on cognitive and occupational questionnaires on mathematical, reading and scientific literacy.

“Taking into time constraints of having to complete FT, the NPC requested education minister to assess the situation following which a decision to drop Pre-Test III was made,” she said. “But despite challenges, Bhutan has successfully conducted PISA-D main survey.”

However, EMO Kinley Dema said, Bhutan had registered with OECD to participate in PISA-D by paying a registration fee of EURO 50,000 to conduct the cognitive questionnaire.

A total of 2,457 students (class VII-XII) from 53 PISA-D sample schools, 530 teachers, and 53 principals attended the examination in November 2017.

The data has been submitted to the ETS based in New Jersey and the result is awaited.

“The sample schools are selected by the ETS based on their sampling reference and although 2,457 students appeared the exam, they took only 2,000 students’ data,” Kinley Dema said. “The students’ age group is 15 years and three months to 16 years and two months during the time of exam.”

BCSEA did not register for occupational questionnaire but decided to implement themselves.

Once the result is declared, the NCP would correlate the mean point of both the questionnaires to give the students’ performance and competency in comparison with other countries.

“With this report, we can have a complete PISA-D national report that will rate the children’s performance and also state factors that affect the performance,” Sonam Lhamo said.

To familarise students with the questionnaires, the NCP had conducted a preliminary assessment in 13 schools in Thimphu.

The result showed that although students met the baseline of two for scientific literacy, they secured below baseline for both reading and mathematical literacy.

The result for Pre-Test I that was administered nation wide was all below the baseline 2.

The objective of the PISA-D is to set a benchmark profile of the knowledge, skills, and competencies of students in Bhutan, provide evidence for educational reforms. The Bhutan Education Blueprint 2014-2024 recommends participation in an international assessment.

Yangchen C Rinzin