Senior RACGP leaders invited to discuss recent exam issues

GPRA invited RACGP senior leaders to discuss the recent issues and concerns regarding the College’s clinical and written exams

Below are the questions posed to Dr Matthew Miles (CEO), Dr Genevieve Yates (General Manager - Education Services), as well as Dr Krystyna de Lange (Chair of GPs in Training Faculty) and a summary of their responses provided at GPRA’s recent Advisory Council meeting.

How will the RACGP accommodate trainees disadvantaged by the new exam schedule? Are there any plans for a second off-set KFP?

RACGP response:
The time required to mark the 4 December KFP and/or 5 December AKT means that these results will not be available in time for candidates to sit the February 2021 AKT and KFP written exams.  For candidates who are unsuccessful in December 2020, their next opportunity to sit the KFP will be in the July 2021.2 exams.

Candidates who pass the AKT but fail the KFP, can still sit the RCE in March 2021 without passing the KFP. Candidates who pass the RCE and AKT but fail the KFP, may need to extend their training time by an additional six months in order to sit and pass the KFP (as they would have to normally if they failed any exam).

The RACGP has no plans to run an additional KFP exam between the 2021.1 and 2021.2 KFP exams.

GPRA response:
GPRA is disappointed to hear that the RACGP is not considering a second off-set written exam. We recognise that this outcome is similarly disappointing to many trainees and will place additional stress of trainees to pass the KFP exam.

What is the rationale for the decision not to allow all candidates of the deferred KFP exam access to the compromised exam, both as a study tool and to provide fairness as some candidates have knowledge of parts of this exam?

RACGP response:
Noting that we have never released a full exam paper publicly, none of the questions will be reused in any future exams. We intend to use the questions from the compromised exam in future practice exams and in other exam preparation resources. This decision is in-line with other medical organisations who have not released previous exam papers. Additionally, considerable behind-the-scenes work goes into releasing sample questions which must be accompanied by rationale for each question, ensuring this has educational value as an exam preparation resource. There is not sufficient time to reconfigure the compromised exam for use as exam preparation materials for the December resit as we have dedicated all our resources towards delivering the upcoming exams.

There are exam preparation resources already available to all candidates to assist them in their preparation for the December KFP and AKT exams. Candidates who started the October KFP will not have any advantage over candidates who did not see any of the questions on the abandoned exam paper.

GPRA response:
GPRA understands the rationale behind not making public the entire exam, however, we believe that releasing a small number of questions would be extremely helpful for those preparing for the exam, particularly as a number of candidates were able to see questions before the KFP was aborted. Candidates who were not able to even start the compromised KFP exam did not see any of the questions and this potentially puts them at a disadvantage.

The decision to release the compromised exam would be in-line with the College of Physicians (RACP) exam failure in 2018, where the exam paper was released to all candidates. While we understand that a lot is invested into creating an exam and the questions from compromised KFP will be utilised for exam preparation resources in future, this decision does not assist the affected candidates in this current exam cycle.

Why are the College Board and GPs in Training (GPiT) Faculty the only bodies eligible to discuss and set the Terms of Reference of the RACGP’s External Audit?

RACGP response:
The RACGP obtained a root cause analysis of the exam platform failure from Genix which clearly identified a technical failure of the platform. The RACGP has already conducted an internal review which has identified recommendations for potential areas of improvement, which will be implemented. The RACGP Board will set the Terms of Reference with input from the GPiT Faculty. There will be opportunities for external stakeholders, including GPRA, to provide input into the external review process. Price Waterhouse Coopers, will be undertaking the external review which will consider a whole range of exam related matters including the internal review, and will ensure a rigorous process is used in conducting their external review. As the circumstances surrounding the failure of the Genix exam platform is the subject of legal action, the RACGP is not in a position to publicly release the report at this time.

GPRA response:
The RACGP needs to commit to publishing the outcome of this review publicly before the investigation begins, to ensure trust and confidence in this review process. We urge the RACGP Board to commit to this important decision to rebuild trust and a sense of transparency amongst its GP trainee cohort.

How does the RACGP plan to improve transparency around exam fees and its exam workshops? Will the RACGP deliver on clearer and transparent communication strategies to exam candidates moving forward?

RACGP response:
Exams are very expensive to run. There is much work which goes into providing an exam, and significant costs in exam development, assessor training, exam delivery, marking, quality assurance, and censor review. Our 2019—20 annual report, shows that the RACGP made a loss providing exams. While there are ongoing internal discussions and financial evaluation of potential efficiencies to reduce exam costs and fees charged to candidates, there are no immediate plans to reduce fees as the RACGP has to run the exams in a financially sustainable manner.

We understand that the additional fee to sit Day 4 of the RCE was not well communicated and so came as a shock to some candidates. We have taken on board feedback from candidates and intend to be more effective and timely in our communication to candidates in future. All costs will be made known prior to enrolment. Opportunities to reduce the cost of the mock RCE are also being explored.

GPRA response:
The financial loss in delivery of exams in 2019/20 was partly due to only holding one clinical exam in this period (due to the coronavirus pandemic). Fees and transparency continue to be a point of contention amongst GP trainees which is indicative that the RACGP needs to improve in this area. GPRA will continue to urge the RACGP to commit to better transparency, affordability, and clear and timely communication.

Will there be compensation for affected exam candidates who have exhausted all their leave preparing for their exams?

RACGP response:
The RACGP is providing a refund on exam fees to candidates collectively worth $5.9 million. It is also providing a free resit of the KFP and AKT candidates, which is worth $4,660 to a candidate sitting both exams and $2,330 to a candidate sitting one exam. We do not consider that leave taken by candidates to study for their exams has been wasted. This effort still goes toward their overall readiness to sit the KFP and AKT. Individual candidates with acute financial concerns as a result of the cancelled October exams are asked to contact membership services on 1800 472 247. The RACGP is in the process of taking legal action and pending the outcome of legal action the RACGP is unable to provide further comment. The RACGP will keep candidates informed of the progress of the legal action. There have been a few delays with processing the exam refunds. As at 25 November, we had processed 96% of refunds, so almost all candidates should have received their refund.

Will the RACGP waive the extra fee for those who had to sit day-4 of the RCE? In the future, if this tiered system is to continue and, considering that the RACGP profits from sending a candidate to day-4, how will the RACGP address the ethical and fairness dilemmas?

RACGP response:
There will always be significant costs associated in delivering RACGP exams because they are so labour intensive. The associated costs for running the RCE have been substantial. We modelled the RCE to be cost-neutral. We also sought to ensure appropriate and sufficient contingencies were available to reduce the risk of candidates being unduly delayed or adversely affected by technical issues. These contingency measures were enacted and ensured that those candidates who had issues with one or more of their sessions were able to continue and complete their RCE with minimal disruption. The RACGP was delighted to have delivered such a smooth candidate experience.

Of 1,107 candidates who sat the RCE, 19% were invited to Day 4 of the RCE. Instead of spreading the costs to run Day 4 to all candidates, we decided that it was fairer that the cost to sit Day 4 should be paid by those who required Day 4. The alternative would have been to charge an exam fee to all candidates that included the cost of running Day 4.

An external consultancy, with specialist expertise in vocational assessment, provided significant input to development of the exam methodology and competency rubric.

The College budgeted Day 4 to be cost neutral, not revenue raising.  It was offered to provide a greater opportunity for marginal candidates to demonstrate their competency and pass their exam.  These candidates would not otherwise have had this opportunity and would have to wait to resit the RCE in the next exam cycle.  The RACGP felt that given the significant disruption caused by the COVID-19 pandemic that offering an additional day was in the best interests of the exam candidates.

See GPRA's latest news.

Become a GPRA member — it's free and takes one minute to join.

The NTCER covers the terms and conditions of GP Registrars' employment.