The design of Proficiency Testing schemes is based upon a standardised database structure that is common to all schemes.
Notes
3.04.1
Hierarchical Data Design
QAPonline has a Hierarchical data design around which all schemes operate. A EQA Series table define that nature of the scheme and identifies the Key EQA question that forms the basis of all analysis. The hierarchy is as follows:
Updated: 30/10/2012
3.04.1.1
DISCIPLINE. Examples may be Andrology or Embryology. Each discipline has a link to species. While all EQA schemes are designed for human topics, the database allows EQA schemes for other species if the nee arises.
3.04.1.2
SERIES. A Series contains the key information that is carried over from year to year. There is a 3-4 character Code to identify each series e.g. HAB= Human Sperm Motility. It contains a general description of the aim of the series and the key question upon which all primary statistical analysis is based. The same question is used for long term performance analysis.
3.04.1.3
SCHEME. Each year an EQA scheme is made available using the key question as the primary analytical tool. All enrolments use the scheme ID and all subscriptions contain a link the the scheme ID. enrolments in a scheme entitles the participant to submit replies to any question in any release under the scheme ID. All performance reviews are made against the scheme ID. The identification of each scheme is usually by combining the Series ID +the year. e.g. the Sperm motility Scheme for 2008 would be HAB2008.
3.04.1.4
RELEASE (or INSTALLMENT, older term). Each scheme contains a number of releases or units of analysis. In general there is one release per month. This encourages participants to return monthly to complete the current questions. In some months there may be more than one release and in others there may be a release every quarter. Wherever possible, QAPonline attempts to provide 1 release/month/scheme. Each release contains a number of quizes one of which will be the QAP question. The other quizes may contain questions that contribute to the QAP question or may be of an educational or supportive or general interest nature. QAPonline anticipates a reply to the QAP question in each release and many summaries document the performance/release. Each release is identified by the scheme ID Plus a date ordered number e.g. the first release for the 2008 sperm motility EQA is identified by the label: HAB2008.01 Importantly, each release also contains other information as described below.
3.04.1.5
QUIZ. Each release as one or more quiz attached to the release ID. Each release contains a question relating to the release scheme information. When a participant replies to a quiz, the quiz ID is one piece of identifying information collected. The question ID carries all the information about the nature of the questions, how it is be displayed and processed and the options that are presented and available to participants to select. All statistical summary information is arranged by the Quiz ID. It is possible to submit a NULL reply to a quiz which signifies that a reply has been recorded but the participant did not wish to record a answer.
3.04.2
One Key EQA Question.
In all Series and schemes, there will be one question that is carried over between each annual scheme
Updated: 30/10/2012
3.04.2.1
Questions may be of a number of formats including NUMERIC, PERCENTAGE, OPTIONS, MEASUREMENT, CALCULATED and TEXT.
3.04.2.1
All KEY QAP Questions are by definition NUMERIC including percentages.
3.04.3
Multiple Question with unique features.
In any release, there are an unlimited number of questions. Some may contribute to the Key Questions while others are non-compulsory and are of education and supportive nature.
Updated: 30/10/2012
3.04.3.1
NUMERIC. A numeric question allows any number including ZERO and factions but within a defined reasonable upper limit. Any submission above the upper limit is not accepted and the participant asked to resubmit a modified reply. Each numeric question has a unit of measurement attached to it appropriate for the EQA Series.
3.04.3.2
PERCENTAGE. This is a common KEY QAP question. It is a Numeric format with an upper limit of 100 and a unit of '%'.
3.04.3.3
OPTIONS. A question with an Options format contains a number of alternative answers from which to select a reply. The option ID is included in the reply when a submission is made.
3.04.3.4
MEASUREMENT. A measurement format is designed to accept a submission where the distance between two marks is calculated. This format is used for the Ultrasound EQA schemes (Follicle diameter and Endometrial thickness). It entails several stages. These are one a calibration of a yellow bar which is allocated a length (usually 10mm). The length of the bar is achieved by clicking the mouse once at one end and a second time at the other end. The difference between the 2 pixel screen coordinates is used as a calibration tool to "calibrate" each screen i.e. 10mm= xyz pixels. Then the user repeats this process on the image ( e.g. a follicle). The difference between the pixel coordinates / calibrated distance= the distance in mm. When a question format is defined as Measurement, the program will pass to a specific measurement routine.
3.04.3.6
CALCULATED. A calculated question was developed to allow conversion of a series of descriptions (options) e.g. for an embryo, into a numeric formation to allow for statistical analysis. Essentially each option is given a weight or value e.g 1-5. A calculated question is one of a number of questions (quizes) attached to a release. All the questions required to calculate a value must be included in the correct order in order for the calculation to sum or process the value attached to the option for each question. The calculated question returns a numeric value almost universally expressed as a percentage. The limitations of using a calculated question is expanded below.
3.04.3.6
TEXT. QAPonline also provides an opportunity for each participant to add qualifying information to the release. For instance, a release may prompt the participant to explain the reasons for ranking embryos or ask them to describe in their own terms how an embryo may be described, etc.
3.04.4
Limitations of using a calculated question for Statistical analysis.
No additional information.
Updated: 30/10/2012
3.04.4.1
Calculated questions were developed to allow the database driven analysis to be applied to EQA scheme that are not naturally numeric. In a general way the values attached to each option imply quality of the parameter being assessed e.g. the size of a blastocoel cavity. The summation of values from several options from several questions implies "quality" in so far as a high score is associate with the selection of a number of options that have a high value. A low score conversely implies many options selected reflected a poor or immature quality and the summation of these option gives a score that reflects this.
3.04.4.2
The submission of a number of calculated numeric replies will form a normal distribution around a mean figure. The more options available to contribute to the calculation, the more normal the distribution of the replies will be.
3.04.4.3
Calculated questions exist where there is no recognised other way to evaluate an object in the domain of the EQA scheme such as an embryology laboratory. The number is unique to QAPonline and other laboratories may have other, more simplified means to describe the quality of the embryo or other entity.
3.04.5
Types of Information provided in each release for assessment.
The material being presented for assessment is largely image based for delivery over the Internet. The format includes IMAGES, SLIDE SHOWS, VIDEOS, ZIPPED DOWNLOAD FILES. There are a small number of EQA schemes that are sample based which require a number of samples to be dispatched. These are only for local participants and not for international distribution.
Updated: 30/10/2012
3.04.5.1
The image details are housed in an IMAGE table and contains a link to the actual image location.
3.04.5.2
INDIVIDUAL IMAGES. All images used for EQA evaluation are JPG format usually 25-50kB in size.
3.04.5.3
IMAGE SLIDESHOW contains a unique Image ID with a number of images (also JPG, 25-50kB)in an associated file. The images were multiple images from the same sample e.g. a number of slides through an embryo or multiple images from a single stained slide of sperm morphology assessment. The rule is that the total number of cells (sperm) in a slideshow should contain more than the minimum number of cells required by the standard to be counted. When the release is displayed, the first image in the slide show is displayed with progression tags to allow the participant to move progressively forward or backwards through each slide
3.04.5.4
IMAGE VIDEOS. Current images are primarily FLASH swf files of ~25 MB in size and 15-30 seconds in duration. Where possible, multiple files from the same sample can be provided. A Flag above the video provides a list of all the formats and ID is provided. In some releases, there is only 1 swf file, while in others, a swf and a YouTube version is available.
3.04.5.5
OLD VIDEO formats. Originally band width was a limiting factor as was browser types. Accordingly 1 video was provided in avi, mepg and mov (for Apple users). There was only 1 15 sec, 2-3 MB files but in 3 different versions. from 2011 and progressively in 2012 forward, the swf format is been standardised. Bandwidth does not seem to be an issue.
3.04.5.6
ZIPPED Files. IN some schemes ( eg the CASA sperm motility/morphology schemes) the video is displayed but also ZIPPED as a downloaded file. In this scheme, the file is intended to be run on the participants own CASA system and the results posted to QAPonline.
3.04.6
Submission of Results and data storage of summaried data.
QAPonline is designed to be used by each and every scientist in a laboratory. The results from each participants for every question/quiz is stored in a REPLY table (now over 1.2 million records). The reply table contains ID to the participant, the release (redundant data), the quiz and the enrolment. The table captures the reply and the selected option ID together with the date/time, the company and the laboratory ID.
Updated: 30/10/2012
3.04.6.1
The reply table is the single entry point for all replies. However to improve traffic flow, a copy of each reply for a QAP question is made in a replicate file. This may also provide some backup should the reply table be corrupted.
3.04.6.2
However, to avoid excessive demands on the Reply table, key information is in several other tables. The Release summary table, the quiz summary table, the company summary table and the laboratory summary table.
3.04.6.3
Individual Summary table contains a summary of the activity of each participant and the reply data for each release/scheme/participant. It is created when the participant enrols and is updated after each reply has been submitted. This table keep track of how many questions have been answered and the current performance of their key Reply from the group mean. A regular function of QAPonline is to update these tables whenever a reply has been submitted. A flag of date of last update and last reply flags which records need updating.
3.04.6.4
Quiz Summary basic statistics table contains the summarised data for each quiz. For numeric data, including the KEY QAP question, the number, mean SD, %CV, Sum, SquaredSum, Mode, Minimum and Maximium are calculated for all professional data, trimmed professional data (where all replies over 2 SD from overall data is excluded and the data recalculate), peer data (see later). The data for option based questions store the number of replies for each option (in a sub table) and the number and ID of the most popular option. This data is updated daily and automatically after 5 new replies have been received. A flag for the date of last review and last submission identifies which records need updating.
3.04.6.6
A Company (and laboratory) summary is also updated for all professional replies from participants in each company (and laboratory) for each quiz is also maintained. This is updated when a new reply has been received for a company member. The number, mean, SD , %CV, Sum, Squared Sum, Minimum and Maximium are calculated. This data is used for company and laboratory based reporting (see later). The QAP Supervisor can call a review/ update at any time.
3.04.6
Definition of Skill status at Enrolment and PEER participants
QAPonline has several levels of enrolment into all EQA Schemes. The allocation of a skill level is largely made by the QAP Supervisor in each QAP group but can be modified by each participant between student to professional but only the Supervisor and Administrator can upgrade a professional to PEER status
Updated: 3/11/2012
3.04.6.1
GRANDFATHER status (excuse the gender bias). Each participant is asked to identify the year when they were approved to work independently. Not every participant will complete this but for those who have, QAPonline can calculate at the time of enrolment the number of skilled years and use this in one calculation as a grandfather comparison (see later).
3.04.6.2
PROFESSIONAL Skill Level is the basic level for all statistical calculations. It is meant to represent a participant who has been approved to work unsupervised in the area of the EQA Scheme.
3.04.6.3
PEER skill level represents a professional who in the opinion of the QAP Supervisor is a peer worked in the laboratory. Their level is marked 4. In addition to each QAP supervisor marking some staff as Peer level, the administrator also reviews the performance of all participants in a EQA series who have more than 5 years state experience and whose performance indicated that more than 75% of replies have been within 1 SD of the trimmed mean and have replied to more than 25 releases. If they are from a recognised laboratory, the administrator may also mark them as a Peer enrolment in the current scheme. These are marked level 5.
3.04.6.4
STUDENT enrolment is deigned to allow unskilled students to participate in the QAPonline EQA schemes and gain experience and certification WITHOUT their replies being used in any calculation or statistics. They are marked as Level 1. Their replies can be seen by the QAP Supervisor when performing data review but their contributions are ignored.
3.04.6.5
IN TRAINING enrolment status is designed to allow staff skilled in one area gain experience and certification in another area without their submission being used in any calculation. This is an important process for laboratory staff to use as a tool to assess the performance of a new staff member when they join a laboratory of move to a new area in the same laboratory. This is especially so since QAPonline provides the tools for a Supervisor to asses the performance of a staff member both against all other participants Or only against those in their own clinic. More on External and Internal QC below.
3.04.7
Calculation of Basic and Peer Statistics and how they are used.
QAPonline applies a standardised approach to calculation of the basic statistic.
Updated: 3/11/2012
3.04.7.1
TRIMMED Basic Statistics are the standard scheme statistics. The replies are reviewed monthly for severe outliers (>4 SD from the mean) where administrator has the option to mark these replies as INVALID. This renders the reply inaccessible to the routine computations. Any quiz that has received a reply is marked as update pending and routinely (nightly) updated. The updating involves the calculation of the MEAN and SD all replies from professionals and peers. The Mean and SD are then recalculated as a trimmed dataset by excluding all replies that lie outside of the 2SD range of all valid replies. The complete statistics include number, Mean, SD, %CV, Maximum, Minimum and Mode.
3.04.7.2
PEER statistics are calculated at the same time and includes ALL valid peer replies for the quiz. The complete PEER statistics include number, Mean, SD, %CV, Maximum, Minimum and Mode.
3.04.7.3
PROBLEMS with defining a PEER or reference Class. One issue faced by QAPonline is defining a reference or peer population. One was that is it necessary?. While WHO standards exist, there are no training programmes or other courses to provide certification from which to attribute Peer status. In other areas such as embryology and ultrasound, there are no even standards. ESHRE Andrology EQA has a pool of laboratories that they have defined as good and from their replies, it is understood replies are pruned to created a data set with low %CV often with few data points. EQASRM has taken all IVF clinics as their reference population in what may be seen as somewhat elitist. For several years, QAPonline actively sought the participation of primary published authors and their laboratory staff to act as a peer subgroup but NOT ONE was prepared to commit to such as role. In other words, while they may have published and developed guides that have been adopted as a default standard, they were not prepared to support they default training of others to that 'standard'. QAPonline has adopted a different approach in the development of a Peer or Reference population. It sought to identify participants who have demonstrated experience (>5 years work place activity) together with a proven record of performance ( More than 75% of replies have been within one SD of the trimmed mean). The aim was to provide a dynamic and large pool of participants such that the peer data set has meaningful numbers.
3.04.7.4
FREQUENCY OF UPDATING Statistics. As indicated above, the statistics are marked for updating every time a reply has been received. Usually, the statistics are updated nightly or as soon as possible afterwards. It is proposed that from 2013, each quiz will be automatically updated when a minimum number of replies have been registered. The default difference is proposed to be 5 subject to review on speed limitations. Since QAPonline aims to deliver a immediate summary of a participants performance, the aim is to keep the statistics as current as possible. More effort is given to updating data at the beginning of each month. After 20+ replies have been received, the basis statistics rarely significantly changes.
3.04.7.5
EXTERNAL QA and INTERNAL QC. Unlike all other Proficiency testing programmes, QAPonline used to power of individual participation to provide a unique dataset from which to generate comparative statistics. as described above, since the laboratory and clinic (company) codes are included at the item of submission, analysis of the dataset can be against all professional+peer replies to form a dataset for external comparisons or restricted to only those from within the company or laboratory. Therefore, QAPonline provides dynamically, the capacity for the QAP Supervisor to assess the performance of each individual staff, laboratory or company (clinic) against the total population or their own company or laboratory as required.
3.04.7.6
QAP SUPERVISOR REVIEW. QAPonline has developed the concept that the QAP supervisor can determine the method for EQA comparison. To achieve this, a range of comparisons are available to choose. These include all all professional+peers replies, Peer replies, grandfather replies (>=10 years experience), replies from their own country. The automatic standard is the professionally +peer dataset and this data is stored in separate, up-datable tables.
3.04.7.7
INDIVIDUAL review of a participants own performance can be made at any time. QAPonline has provided the tools for an individual to assess their performance against the above links (Professionals, Peers, fellow country workers, experienced and their won company) as well as graphs of bias and performance over time. Recently, QAPonline has also introduced the 'worm' graphical presentation of the last 30 submissions. Plans are this this to also provide statistical appraisal according to the Westgard rules.
3.04.7.8
CLINIC PERFORMANCE can be assessed at any time. Unlike other Proficiency Testing Programmes, QAPonline does no issue a paper based report. Rather, QAPonline has empowered the QAP Supervisor to elect the frequency of reviewing and printing the online reports. In many clinics, the QAP supervisor or their representative, print out their performance every 3 months. They also can elect the group they wish to be compared. While almost all use the large professional data-set, the QAP Supervisor may elect to use an alternative subgroups (see above) for their own purposes and even for accreditation. QAPonline dynamically produces the reports using all the submitted replies.
3.04.7.9
LABORATORY PERFORMANCE. QAPonline has created the capacity for a company (clinic, pathology company) and to create an unlimited number of laboratories that operate under the main company licence. The QAP Supervisor can allocate each staff to laboratory and may update them as they move between laboratories. Since each reply is allocated to a company AND a laboratory, then the results remain with the laboratory they belong to at the time of submission. Therefore critically, QAPonline provides the capacity for each sub laboratory may produce reports and compare the laboratory to others in the QAP group.
3.04.7.10
INDIVIDUAL PERFORMANCE (Competency). Finally, QAPonline allows the QAP Supervisor to review the performance of each staff member in the QAP group. The review can look at each years activity scheme by scheme, a summary using QAPonline 'worm' or review a certificate using Monash Masters in Clinical embryology method for competency.
3.04.8
Subscriptions, Enrolments and Invoices.
Information of each companies subscription and staff enrolments and the invoice/payment for each subscription is housed in separate tables.
Updated: 3/11/2012
3.04.8.1
QAPonline creates a list of subscription options each year containing various combinations of EQA schemes available at varying subscription levels. The QAPonline model allows for clinics to subscribe at a rate appropriate for their needs and staffing levels For instance, a large clinic may wish to subscribe at a GLOBAL rate that will allow all staff access to any scheme available and appropriate for their work environment while others may only wish to have access to schemes in one discipline e.g. Andrology. A different fee is levied for each subscription combination.
3.04.8.2
A company will subscribe and pay the appropriate fee to one or more of the EQA Subscription options available. The QAP Supervisor is then in a position to select from the schemes available those of interest. The subscription and the Company/Scheme combinations are housed in separate tables. An invoice is created for all the subscriptions requested and payment in $AUD, Euros or $USD is retained in their own data files.
3.04.8.3
Each participant is then enrolled in one ore more EQA Schemes appropriate for their work environment. Enrollment can be either by the QAP Supervisor (Most common) or by the participant themselves from the list of schemes nominated by the QAP Supervisor. Each enrollment contains details of the skill level (student, training, professional or Peer. Only Professional and peer participants submissions are used by QAPOnline or statistical purposes. Once a submission has been recorded, withdrawal from the enrolment is not allowed. The year the participant was allowed to work unsupervised is also requested (but not always supplied) to allow for estimation of grandfather status ( 5-10 years).
3.04.9
Individual and Company/Laboratory Tables.
Finally, each individual is registered in a primary table. This data contains personal information including name, address and email. One registrant can created a QAP group (Company), laboratories and add new staff members to the QAP Group (Company).
