LONGREAD > Quality assurance in Flanders: simple, appreciative and incomparable24/01/2020
Article on quality assurance and institutional reviews in Flanders, published by The Danish Accreditation Institution.
The Accreditation Organisation of the Netherlands and Flanders (NVAO) goes to great lengths to ensure that their processes in Flanders are fully adapted to the individual institution and that they are simple and do not include unnecessary documentation. We zoom in on how they do that and what it means.
By Daniel Sebastian Larsen, Head of Communications, The Danish Accreditation Institution
‘The most important tools are probably the tools we do not have.’
This is what Pieter Caris, Policy Advisor in the Flemish Department of NVAO, replies when asked about the tools that keep the external quality assurance procedures simple and straightforward. After three decades of programme accreditation, Flanders made a shift towards institutional reviews in 2015. Not as a replacement for programme accreditation, but as an addition that enables higher education institutions with a positive outcome to internalise the external quality assurance of existing programmes. A positive outcome leads to the extension of the accreditation period for each of their already accredited programmes. In developing the new QA system, NVAO has made a few startling choices. They installed NVAO’s Appreciative Approach, they try to keep the assessment procedures simple, and tailor them to the context of the individual institution thereby omitting fixed criteria and templates.
Only one document
For the institutional review in Flanders, which is the part of the Flemish QA system that is most reminiscent of the institutional accreditations in Denmark, institutions are only asked to prepare one document: a critical self-reflection. All other information is provided by means of existing documentation. It is one of the ways to try to keep the amount of documentation - and thus the workload - low for the institutions. And even the critical reflection is limited to fifty pages.
Dialogue as evidence
Another way to reduce the amount of documents in the review process is the explicit use of dialogue, which is in the Flemish system also a valid source of information.
‘Documents cannot tell everything. Explaining your own context can be difficult. Similarly, demonstrating one's own culture to others can be hard to do. This is also true for higher education institutions. So, when the review panel visits the institution after having read all the documentation, the reviewers probably have an idea of the context. But if they want to understand it thoroughly, dialogue is needed. The original documentation forms the basis, but it is not sufficient for a full assessment,’ says Pieter Caris.
His director complements: ‘This dialogue goes both ways. While looking into a specific area, the panel also further explores the strengths of the institution. The same goes for more critical issues where the panel asks for further clarifications. If they are uncertain about results, the reviewers want to know how the institution makes sure that the intended outcomes are achieved. In this way the panel continues down a track, and along the way it also gives feedback to the institution The institution should feel it has the opportunity to communicate about the issues and obstacles it has encountered and to discuss possible ways to move forward with the panel. This whole dialogue has to be very open,’ says Axel Aerden.
In the end, the institutional review results in a report from the panel that forms the basis for a decision taken by the NVAO board. As a result, the accreditation period for each of their already accredited programmes is automatically extended by six years.
‘A report should never come as a surprise to the institution, because the institution will be part of the journey together with the panel through extensive dialogue along the way' says Pieter Caris. This is a break with the past, where programmes had to wait for the findings of the panel that carried out programme assessment procedures that could in many cases be hard to predict.
NVAO’s Appreciative Approach
A key element of NVAO's reviews of the Flemish higher education institutions is what they themselves call their Appreciative Approach.
NVAO’s Appreciative Approach involves, among other things, an overall enhancement-led perspective and a special focus on the institution's strengths and success factors. This approach also means that there is a certain kind of questions that panels will never ask.
‘We will not judge in relation to the choices made by the institution. We try to understand the context in which their decisions should work, and we look at whether the model they developed for themselves works as it should and as they had intended it. That is what we are looking for. If we look elsewhere in European quality assurance, we can see that often the organisational structure of an institution is also assessed. We will not do that,’ says Axel Aerden.
‘It is also an important way of ensuring that we do not get an overload of documentation,’ Pieter Caris adds. ‘Yes, because institutions do not have the burden of proof. We just need the necessary documentation to understand the structure, but we do not need documents that motivate or defend the choices they have made about how they want to organise their institution. And frankly, these are fully-fledged higher education institutions we are talking about. Should we question these structural decisions?’ Axel Aerden is wondering.
"The most important tools are probably
the tools we do not have"
Consistency in a system without criteria and templates?
Based on the desire to make the process simple, NVAO has also decided to move away from all kinds of fixed criteria, formats and templates. Instead, the panel can focus on the areas that seem most relevant to them in the context of the institution under review. ‘Basically, quality assurance - and also the ESGs (The Standards and Guidelines for Quality Assurance in the European Higher Education Area, ed.) - is really about three questions: “What do you intend?", “How will you implement it?”, and “How can you make sure that what you intended is also achieved in the end, as you planned it?” The three questions are in fact guiding all processes, and what we would prefer is a holistic review by the panel in order to answer those questions.’
‘In the past, panels often felt limited by standards and criteria. They might notice something interesting but were not able to include it in their discussions and final report because it was not part of the criteria. And there could also be criteria that the panel did not find particularly relevant for the institution, but which they nevertheless had to discuss in the report. Therefore, we do not use standards and set criteria anymore,’ says Pieter Caris, explaining that there is also no fixed format for the critical self-reflection of each institution, just as there is no imposed structure for the review report.
‘Well, we always tell the institutions that they will not to be discriminated against or treated unfairly by how we assess other institutions. It is not that they are all equal. When you assess based on individual context as much as we do, you cannot say they are equal. A School of Arts compared to a large comprehensive university that is ranked in the top 50 worldwide – it is a totally different context. They are not equal,’ says Axel Aerden, and he continues: ‘That does not mean that they should have reason to feel discriminated. We treat them the same, but we do not assess them as if they are the same.’
The Flemish branch of NVAO has several elements in its system that aim to ensure the transverse consistency. First and foremost, NVAO recruits panel members from a pool of experts. These experts are used repeatedly, and overlap is created among the panels that carry out the institutional reviews. ‘As a result, the findings and considerations of the panel at one institution can be taken into account in in another institution. Not necessarily to push them through the same mould, but to put the context of different institutions into perspective,’ says Pieter Caris. ‘There will also be overlap between the panels of, for example, a research university and a School of Arts. Some experts can be deployed in both contexts,’ Axel Aerden explains.
Part of ensuring consistency, is also reached by an intensive training of the panels, in which role-play is used as a technique. Besides, NVAO specifically trains and certifies a pool of external secretaries that assist the panels in a consistent manner. And finally, the NVAO staff involved in conducting the institutional reviews meet on a weekly basis to maintain a common understanding of the QA system, procedures, methodologies used and NVAO’s Appreciative Approach.
There have never been any formal complaints from the institutions about a lack of consistency. However, some institutions have expressed some reservation over their own review, after reading the reports of other institutions. ‘But it is difficult for the institutions to use that argument. The report of the panel starts from the individual context of each institution and from there the panel explains what they have looked at, why, what they have found, what impact it has made, what they found interesting and what has come up as points of attention along the way. So even after one or two pages, it becomes very difficult to compare the report with a report of another institution,’ says Axel Aerden.
‘There is also another explanation,’ Pieter Caris adds. ‘We should be aware that institutions are not as concerned about the consistency across institutions, because they know they are not being compared to each other. There is no comparison in the system, no ranking. What they really appreciate, is that we are aware of their specific context - that they feel appreciated and that they recognise themselves in a report that generates added value in their pursuit of enhancing quality. They do not mind if the approach taken for their particular institution differs from the one that was adopted at another institution,’ he says.
"A report should never come as a surprise
to the institution, because the institution will be part of the journey"
Fewer degrees of freedom
The decision of an institutional review does not result in the same degree of freedom as is known from the Danish system.
A positive outcome of the review entitles an institution to take full responsibility for assuring the quality of their programmes. As a result, the accreditation period for each of its programmes is automatically extended by six years. However, this only applies to the programmes that have already undergone both initial and first accreditation. Therefore, new programmes and programmes without first accreditation are still reviewed by an external NVAO panel, even if the institution has got a positive decision on its latest institutional review.
Institutional review - not accreditation
‘That is also our primary reason for describing the institutional assessment as an enhancement-led review and not as institutional accreditation,’ Pieter Caris explains.
‘That way we want to emphasise that NVAO still acts as a gatekeeper in relation to all new programmes,’ he says, adding that it also implies that institutions are never granted the right to freely extend their offering of programmes, regardless of the outcome of the institutional review.
If an institutional review results in a conditional positive decision, the institution is reassessed within three years. A negative decision means that in the coming six-year period, the NVAO reassesses each of the institution’s programmes separately. But the question then is whether, in a system with a focus on dialogue, an Appreciative Approach and a focus on the strengths of an institution, it is still an option to take tough decisions and have a negative outcome of the institutional review?
'Yes, it may well happen. But if it had happened when we completed the first round of institutional reviews, it would have come as a shock to us. We would have regretted to learn that institutions were not ready for the new system. We sensed that institutions were fully prepared for taking up more ownership and we always expressed that idea to our political and other stakeholders. We proposed the switch to institutional reviews, precisely because we felt that after three decades of programme accreditations, the institutions were ready for it,’ says Axel Aerden.
At present, institutional reviews in Flanders have led to a positive decision for 14 institutions; four have received a conditional positive decision and none came out with a negative result.
