The scope of the quality manual is to provide a framework for the accreditation of QAPonline as a Proficiency Testing Program and is based on and follows closely the ISO/IEC 17043 standard.
External Proficiency Assessment
17043 is largely concerned with the external comparison between laboratories. It is largely based upon current EQA models that are sample based and usually but not exclusively orientated toward mechanical assessments.
QAPonline is built upon individuals in each laboratory completing the assessment tasks in many EQA schemes. From this the means of each individual laboratory and combined clinic are calculated to represent the clinic/laboratories submission. There is no current separate ISO document that specifically deals with individual competencies as opposed to institutionalised quality assurance.
Education and Training
In addition to QAPonline, FertAid also provides some aspects of education and training largely separate from the EQA activities. QAPOnline was the original and remains the primary focus of the companies activities. Nevertheless, it has always considered education, training and other functions to be a parallel activity.
Structure of Manual to ISO/ICE 16043 standard.
The following section describes the organisations and management system for FertAid following the structure and guidelines of the ISO/IEC 17043 standard. In the standard, these are found under section 5, while in the QAPonline Manual, they are listed under section 2.
Where appropriate and necessary, QAPonline may refer to the ISO/IES 17043 reference list.
WHO manual for the Examination of Human Semen (1999,2010)
Reference is also made in some Andrology Manuals to the WHO 4th and 5th edition on the Examination of Semen Analysis.
Alpha/ESHRE Consensus Review on Embryo Assessment.
Istanbul Consensus Workshop On Embryo Assessment: Proceedings Of An Expert Meeting [RMBonline Version]. This paper reports the proceedings of an international consensus meeting on oocyte and embryo morphology assessment. Following background presentations about current practice, the expert panel developed a set of consensus points to define the minimum criteria for oocyte and embryo morphology assessment. It is expected that the definition of common terminology and standardization of laboratory practice related to embryo morphology assessment will result in more effective comparisons of treatment outcomes. This document is intended to be referenced as a global consensus to allow standardized reporting of the minimum dataset required for the accurate description of embryo development.
AS far as QAPonline is aware, there is no consensus methodology for assessment of follicles and endometrium
Wherever possible, the terminology used in this manual is the same and has the same meaning as that of the ISO/IEC 17043 standard. The following additional definitions may also be used.
Internet Service Provider.
A nominated staff member will be called the QAP Supervisor. The QAP Supervisor will have the facility to add new staff
Key QAP Question.
In each sscheme there will be one question that will be the KEY question for the scheme and the basis for all EQA comparisons.
Management of Fertaid/QAPOnline
FertAid Pty is the parent body that owns and operates QAPonline.
Background to FertAid and QAPonline.
QAPonline was developed to provide Quality assurance to Individual Professional Development, Internal Quality Control and External Quality Assurance in a dynamic and educational manner online.
FertAid was established in 2001 and QAPonline a working subdivision and approved trading name initially to develop an alternative quality assurance program for IVF and pathology Andrology departments. Previous experience with EQASRM EQA model was disappointing for several reasons the principal ones being the disconnect between sample submission and report, the requirement to conduct Internal QC program fro individual staff members, the problem with using samples that dissociated the staff assessment skills to the sample handling.
Problems with sample based Andrology EQA models.
All EQA schemes in Andrology, except QAPonline, are based upon a sample based model whereby one or more samples are posted at regular intervals for a laboratory to process and submit a response. This is the same model as applied to biochemical type laboratories where one machine will process one sample. However Andrology is largely a VISUAL process rather than a machine based process and as such the scheme should be applied to ALL staff rather than one laboratory. Indeed, many laboratories ask all staff to process the sample and the mean value is submitted. This has the effect of reducing the range of submissions and providing an unrealistic estimate of variance.
Time between submission and reporting and educational feedback
Having participated in the External Quality Assurance in Andrology for many years, there were two issues that hindered my involvement. One was the long time between processing the sample and the receipt of the reports making responding to variations difficult to understanding why any one result varied. The second was that quality assurance and the education that was required to improve was absent. My discussion with others from overseas who also participate in sample based EQA schemes produced similar responses.
EQA schemes for Embryology and Ultrasoud.
While many pathology laboratories may perform Andrology assessments, there were no EQA schemes for Embryology and Ultrasound. For many IVF laboratories, this meant vary little since the only reason for participating and an EQA scheme was to gain accreditation and thereby Medicare benefits. QAPonline however viewed the need for an EQA scheme in Embryology and Ultrasound just as important as Andrology since these schemes target highly visual and interpretive process critical to a clinics operation. An EQA scheme focusing largely on Andrology therefore provided little support for these other functions.
Aims of QAPonline.
During the initial development phase, the basic model was to provide a vehicle whereby individual staff members were assessed on their visual interpretative skills rather than on their ability to re suspend a fixed sample. Towards this end, QAPonline was developed as an online process rather than a sample based process. The aim was to present all participants with a set of visual images, common to all participants, and then process their assessments as a common pool. This was in line with NATA guidelines which stated that "As far as practicable, all staff involved in testing patient's samples must participate in the QAP with records kept." [NATA Supplementary requirements for accreditation]. In other words, NATA require all machine processing a similar assay must each be enrolled in and EQA scheme. Likewise, QAPonline saw each scientist as an independent "machine" and expected each to individually participate in the EQA program.
Building IC and EQA from individual Competency.
The second pillar for QAPonline was to build from the individual contributions a broader dataset for comparisons. Since each reply is linked to individuals, laboratories and companies, QAPOnline is able to build Internal Quality Control Estimates from the all submissions in each laboratory and each company by using only the submissions from the laboratory or company AND build mean laboratory or company submissions for external comparisons against all participants (not a pooled mean submission). Finally from a management perspective, QAPonline allows construction of an estimate of each staff competency, bias and trending against the companies or global submissions.
Education Feedback and Continuous Improvement.
The third pillar of QAPonline was the capacity for any individual at any time to asses their performance at the time of submission. This allows each participant virtual feedback on how they assessed a "sample" relative to any other current submission. In addition, because the environment allowed for other questions to be asked, additional questions, related but not essential for the EQA assessment, to be asked that forms a second tier of education, feedback and training.
Continuous Submission and Dynamic Reporting
Unlike structured EQA schemes where end dates allowed data review, validation and reporting. QAPonline recognised that this places considerable restraints on data entry and inhibitions in late entries. QAPonline's approach is to provide a dynamic process fro both data entry and reporting. The key to facilitate this was to place the reporting in the hands of the senior scientists [termed the QAP Supervisor]. The summary of each EQA episode is held is separate master files against which all submissions are made. Each companies information is also is held in clinic master files that are also updated weekly. Reporting is made dynamical online using these two master files. In this way, reports are current.
Organisation of FertAid Pty Ltd.
Fertaid Pty Ltd is a private company with a single shareholder (JD Stanger). The ABN number is:
Structure of FertAid.
FertAid was incorporated in 2002 specifically to develop and deliver EQA for Andrology , Embryology and Ultrasound.
Currently, JDS Stanger is the sole operative for QAPonline and Education services.
No further notes.
All document control is via online functions.
No further notes.
Review of requests, Tenders and Contracts
Currently QAPonline does not have any contracts.
Requests for New EQA services
Currently any requests have been handled by personnel email correspondence. The only one to date has been for Halosperm EQA.
Provision for material for EQA schemes
Currently, material for the Sperm DNA Fragmentation has been by donation.
FertAid has a contract with its ISP [IPERA Communications] to provide database management, backup and Internet access and process and resources should assistance be required.
Request, Tender and Contract Review.
Use of Subcontractors.
Currently QAPonline does not have any subcontractors.
Origin of Initial Images
Virtually all the images used in QAPonline were obtained when JDS was employed at Hunter IVF, Newcastle, in 2003/2005 with permission from the medical director.
Origin of Sperm Samples for Sperm DNA Fragmentation.
The sperm samples used in the Sperm DNA Fragmentation EQA Scheme were derived from Queensland Fertility Centre. They were the first clinic in Australia to develop this assay and invited me to create an EQA scheme to provide some confidence in their methodology. In this reciprocal arrangement, QAPonline provides the EQA system, QFG provides the samples in lieu of payment. There is no formal arrangement where QFG is an approved sub-contractor. They have a vested interest in sample integrity and range.
Origin of serum samples for Endocrine AMF assay.
No additional information.
Use of Subcontractors
Purchasing Services and Supplies
Currently, with the exception of the dispatch of material for 2 specimen based samples, all the delivery of images and their processing is provided online.
Delivery of specimens.
This is the only physical aspect of QAPonline operation. The samples are packaged and sent by private courier for overnight delivery. The current courier is `PACK AND SEND`. Samples are checked by email that the delivery is overnight and on the one occasion where this did not occur, a second sample was sent by the same courier.
Purchasing Services and Supplies
Services to Customers
QAPonline provides for services to customers in several ways.
In a regular manner, one question asked if the image quality was accept`able and if an assessment could be completed.
A feedback form is provided at all times and all comments are responded to.
Senior participants are contacted.
QAPonline periodically asks senior participants(QAP Supervisors) for comment and feedback.
QAPonline regularly posts by email reminders to all current participants to complete their EQA commitments.
Services to Customers
Complaints and Appeals
Complaints are usually lodged by the Feedback form. These are retained and replied to immediately.
Since all the replies re lodged online, occasionally a participant will enter the incorrect answer accidental. Each participant can lodge a REQUEST FOR REVIEW Form that the QAP Supervisor should endorse. This form is then passed electronically to the Administrator who will either correct the reply to that requested by the participant OR be marked INVALID.The correct is noted in the reply table and is displayed when the answer is checked.
Complaints and Appeals
Control of Non-Conforming Work.
QAPonline is a dynamic,self reporting online concept. Each participant contributes to the Laboratory/Companies performance. All replies are checked against a Quiz Summary Master File that regulates and checks for non-conformity. The essential flow of management of non-conformity is as follows.
Each Quiz contains one question and all replies are logged to this quiz and includes reply, quiz number, subscriber number, laboratory number, company number (some redundancy here),date, correction request.
Attached to each quiz is a Quiz Master file that summarises the submission to that quiz. The file contains among other data, the number of submissions, the mean and SD of the raw data for all participants marked as proficient (see later), a trimmed Mean and SD (excludes all replies more than 2SD from raw Mean) and the Mean and SD of all replies by subscribers marked PEER status.
Daily Updating of Master File
The quiz and master file number field identifies which quiz records have received new submissions. The Quiz Master File is updated daily where new replies have been received.
Reference to master File
All reporting and reviewing of performance is linked against the data held in the master file. Therefore all performance information is referenced against a controlled data set that manages valid and conforming submissions and all non-conforming replies are excluded.
Definition of non-conformity.
Samples are defined as non-conforming if they are more than 3SD from the mean and are marked INVALID, thereby being excluded from the calculation of the Mean and SD. Samples between 2 and 3 SD from the mean are included in the raw Mean and SD but excluded from the calculation of the trimmed Mead and SD.
Control of Non-Conforming Work.
QAPonline is still considered to be under development even though the format has been in use since 2003. Improvements are focused on the design, flow and speed of the online process and on the nature of the reporting and staff management tools. Since on feature of QAPonline is its performance over time, the structure of the data tables has remained relatively constant. A development online page is available to highlight improvements to all users.
New improvements are largely groups into versions that reflect the year in which they were introduced. The version number is displayed on the Update Page.
The only corrective action that occurs in QAPonline is the modification of incorrect submissions.
As per Non-conforming Tests, replies can be modified by either marking than INVALID or the submitted answer updated and the reply marked as updated.
If a reply is more than 3SD from the raw or trimmed mean, the reply may be marked INVALID. This means the reply is still recorder but the value is excluded from all calculations. This includes both the calculation of the trimmed Mean and SD and the clinics/laboratories performance. Currently, the participant is NOT informed of this action but it is visible when accessed by the participant.
QAPonline will accept requests from individuals to correct a submission that they consider was made in error. The request to modify the submission needs to be approved by the QAP Supervisor before the Administrator makes the change. The change is logged as a modification in the Modification Log and is flagged on the participants reply table.
Justification of reply modification.
It is in the interest of QAPonline and each participant that an incorrect keystroke or misunderstanding should carry a penalty to both the participant and their companies performance. Its acts as a buffer to encourage continued participation and as a tool to genuinely reflect the companies performance. The main risk is that it will be abused to the detriment of all other participants. QAPonline is aware of repeated requests for review and is reasonably generous in accommodating such requests. Serious repeat requests may be marked INVALID to remove the erroneous keystroke error from the data set.
The primary preventative action is to improve the flow and background calculations.
see 2.10 Improvements.
Control of Records.
Control of records includes the security of the electronic records, accessibility of records, the uniqueness and traceability of records and control of modification and validity of records.
Security of Records - Database and backup of data.
The management of the database is the responsibility of the ISP. The entire database is backed up daily ( each evening ~11.0opm EST. This means that data submitted after this time is not backed up until the following evening even though some participants have entered the data in their own time which may be after EST.
Accessibility of records.
Since QAPOnline is online, access to previous records is maintained. Currently, access to the original EQA data from 2003 remains open but access to submit replies to previous schemes is limited. Data Master files are updated at all times. From 2012, the Master file will not be updated for any reply submitted after 12 months from when the release was made public.
As mentioned above, each record has a tag linking the reply to the quiz/release, the participant, the current company and laboratory links. The data for the key question is copied and maintained as a backup in a separate file.
Dynamic Nature of QAPonline.
A key feature of QAPonline is the dynamic nature of the submission and review process. Unlike other proficiency schemes that are sample based, QAPonline is largely image based and submission are available to review dynamically at the time the submission is made. This has advantages and disadvantages.
Advantages of dynamic Access.
A key advantage of dynamic data entry allows for staff to contribute to the data pool, permitting staff competency while still being actively employed. Having no time limitations encourages staff to catch up and remain current.
Disadvantages of dynamic access.
Clearly, the disadvantage of dynamic access is that the data is never fixed and finalised. QAPonline has set time limits not on when data may be submitted but when the master files are updated. In this way, new participants may still have access to QAPonline EQA Schemes but summary tables and reporting are fixed.
Availability for training and orientation of new staff.
One problem all clinics face is training new staff and assessing new appointments. QAPonline allows staff to access previous years EQA Schemes thereby allowing the QAP Supervisor to assess the progress of new staff training and the compatibility of new staffs visual assessment skills.
Varying skill level defeined at enrollment.
Variable access to QAPonline is accommodated by defining the status of each participant at enrolment. The updating of mean and SD in each quizzes master files facilitates this by including replies only from participants who have been marked by the QAP Supervisor as actively involved in the topic of the EQA Scheme. Replies from participants marked as in training or orientation are ignored in the calculation of the master tables.
Implication of dynamic access for Reporting
Data integrity for external quality comparisons is maintained by limiting the means and SD in the summary tables for both all VALID replies and for each company and laboratories. Reporting for a companies EQA performance is developed from information held in both master files. In this way, even though there are many options for data submissions, the data is filtered to provide only replies from valid and approved participants.
Annual Data Off site storage
Currently, the entire database (SQLServier) is downloaded annually and stored offsite.
Control of Records
FertAid regularly reviews the performance of all schemes, checking that non-conforming replies are marked INVALID and the master file updated.
QAPonline reviews the performance of all schemes. From 2011 forward, QAPonline has developed a series of audits for both accreditation, auditing and development purposes.
This occurs prior to the creation on each new years EQA Schemes. 17043 defines the nature of the review with several items to be taken into account. In late 2011, no formal managerial review as detailed by 17043 was undertaken however from 2012 forward, a review as defined will be undertaken. This should be completed by 1st December each year to ensure the EQA schemes for the following year are complete.
The suitability of policies and procedures.
The nature of the questions asked in each EQA scheme should be reviewed.
Reports from Management and Supervisory personnel.
Outcome of Recent Internal Audits.
Corrective and preventive actions
Assessment by external bodies.
Changes to the volume and type of work.
Customer, advisory and participant feedback
Complaints and appeals
Recommendations for improvements
Other relevant factors
Review any other information relevant of the operation of QAPonline including resources and training.
Link to Reviews
The list of management reviews can be found from the link below
QAPonline is a specialised Proficiency Testing Program of the Reproductive sciences. Currently one person (JDS) provides all the technical support for the operation of QAPonline BUT, the site has been designed to allow for a broader service to other areas for testing not necessarily in the IVF or even the medical sciences. Therefore this manual has been prepared in the unlikely anticipation of expansion outside its current framework.
The structure of QAPonline.
From the outset, QAPonline was designed to be an online proficiency testing program. This limits the nature of the material being assessed in so far as it should, by and large, be image or information based and not sample based.
Since it is Internet based, subscription to QAPonline is intended to be international in scope. This has issues in language, payment facilities and local conditions, etc.
Personnel, Staffing and Training.
Currently only one staff member exists.
Potential Staff Functions
The potential staff functions may include: Programmer; Database Manager (hardware); Image processor; Scheme Coordinator; Accountant and Subscription Manager; Secretary. Currently all these positions are filled by JDS. The ISP is currently organising the backup of the database.
Scheme Professional Support.
Each scheme potentially may have an independent professional who can provide advise on the content of the scheme, supply images or other material or interact with the scheme in other professional ways.
Currently only the staff at MicrOptics [Barcelona, Spain] have specifically interacted to provide video material, questions and correct replies for the CASA EQA schemes.
Two schemes - the AMH and the Sperm DNA Fragmentation schemes use samples that are dispatched at the start of each year. Under these arrangements, one laboratory may prepare the samples and dispatch under Fertaid's instructions to the participating clinics using FertAid's courier (World Courier). Under such conditions, FertAid does not charge the contributing clinic a fee for participating in the EQA Scheme.
Equipment, Accommodation and Environment.
Currently, the server is housed and maintained at the approved ISP. The office is located at a private dwelling however this is not fixed with all communication via an Internet connection.
Hardware & Internet Site.
Currently the SQLServer database and Internet software is housed at IPERA Communications (Newcastle). They backup the database every night on a 7 day cycle.
Accounting and Subscription Data
All accounting and subscription data is contained within the database.
All development to both the Internet code and the database is carried out on notebook computers and uploaded regularly after testing. Major innovations re documented in the NEW features pages.
External Accounting Services.
FertAid employs an external accountant services for taxation purposes.
Equipment, Accommodation and Environment.
Design of Proficiency Testing schemes
The design of Proficiency Testing schemes is based upon a standardised database structure that is common to all schemes.
Hierarchical Data Design
QAPonline has a Hierarchical data design around which all schemes operate. A EQA Series table define that nature of the scheme and identifies the Key EQA question that forms the basis of all analysis. The hierarchy is as follows:
DISCIPLINE. Examples may be Andrology or Embryology. Each discipline has a link to species. While all EQA schemes are designed for human topics, the database allows EQA schemes for other species if the nee arises.
SERIES. A Series contains the key information that is carried over from year to year. There is a 3-4 character Code to identify each series e.g. HAB= Human Sperm Motility. It contains a general description of the aim of the series and the key question upon which all primary statistical analysis is based. The same question is used for long term performance analysis.
SCHEME. Each year an EQA scheme is made available using the key question as the primary analytical tool. All enrolments use the scheme ID and all subscriptions contain a link the the scheme ID. enrolments in a scheme entitles the participant to submit replies to any question in any release under the scheme ID. All performance reviews are made against the scheme ID. The identification of each scheme is usually by combining the Series ID +the year. e.g. the Sperm motility Scheme for 2008 would be HAB2008.
RELEASE (or INSTALLMENT, older term). Each scheme contains a number of releases or units of analysis. In general there is one release per month. This encourages participants to return monthly to complete the current questions. In some months there may be more than one release and in others there may be a release every quarter. Wherever possible, QAPonline attempts to provide 1 release/month/scheme. Each release contains a number of quizes one of which will be the QAP question. The other quizes may contain questions that contribute to the QAP question or may be of an educational or supportive or general interest nature. QAPonline anticipates a reply to the QAP question in each release and many summaries document the performance/release. Each release is identified by the scheme ID Plus a date ordered number e.g. the first release for the 2008 sperm motility EQA is identified by the label: HAB2008.01 Importantly, each release also contains other information as described below.
QUIZ. Each release as one or more quiz attached to the release ID. Each release contains a question relating to the release scheme information. When a participant replies to a quiz, the quiz ID is one piece of identifying information collected. The question ID carries all the information about the nature of the questions, how it is be displayed and processed and the options that are presented and available to participants to select. All statistical summary information is arranged by the Quiz ID. It is possible to submit a NULL reply to a quiz which signifies that a reply has been recorded but the participant did not wish to record a answer.
One Key EQA Question.
In all Series and schemes, there will be one question that is carried over between each annual scheme
Questions may be of a number of formats including NUMERIC, PERCENTAGE, OPTIONS, MEASUREMENT, CALCULATED and TEXT.
All KEY QAP Questions are by definition NUMERIC including percentages.
Multiple Question with unique features.
In any release, there are an unlimited number of questions. Some may contribute to the Key Questions while others are non-compulsory and are of education and supportive nature.
NUMERIC. A numeric question allows any number including ZERO and factions but within a defined reasonable upper limit. Any submission above the upper limit is not accepted and the participant asked to resubmit a modified reply. Each numeric question has a unit of measurement attached to it appropriate for the EQA Series.
PERCENTAGE. This is a common KEY QAP question. It is a Numeric format with an upper limit of 100 and a unit of '%'.
OPTIONS. A question with an Options format contains a number of alternative answers from which to select a reply. The option ID is included in the reply when a submission is made.
MEASUREMENT. A measurement format is designed to accept a submission where the distance between two marks is calculated. This format is used for the Ultrasound EQA schemes (Follicle diameter and Endometrial thickness). It entails several stages. These are one a calibration of a yellow bar which is allocated a length (usually 10mm). The length of the bar is achieved by clicking the mouse once at one end and a second time at the other end. The difference between the 2 pixel screen coordinates is used as a calibration tool to "calibrate" each screen i.e. 10mm= xyz pixels. Then the user repeats this process on the image ( e.g. a follicle). The difference between the pixel coordinates / calibrated distance= the distance in mm. When a question format is defined as Measurement, the program will pass to a specific measurement routine.
CALCULATED. A calculated question was developed to allow conversion of a series of descriptions (options) e.g. for an embryo, into a numeric formation to allow for statistical analysis. Essentially each option is given a weight or value e.g 1-5. A calculated question is one of a number of questions (quizes) attached to a release. All the questions required to calculate a value must be included in the correct order in order for the calculation to sum or process the value attached to the option for each question. The calculated question returns a numeric value almost universally expressed as a percentage. The limitations of using a calculated question is expanded below.
TEXT. QAPonline also provides an opportunity for each participant to add qualifying information to the release. For instance, a release may prompt the participant to explain the reasons for ranking embryos or ask them to describe in their own terms how an embryo may be described, etc.
Limitations of using a calculated question for Statistical analysis.
No additional information.
Calculated questions were developed to allow the database driven analysis to be applied to EQA scheme that are not naturally numeric. In a general way the values attached to each option imply quality of the parameter being assessed e.g. the size of a blastocoel cavity. The summation of values from several options from several questions implies "quality" in so far as a high score is associate with the selection of a number of options that have a high value. A low score conversely implies many options selected reflected a poor or immature quality and the summation of these option gives a score that reflects this.
The submission of a number of calculated numeric replies will form a normal distribution around a mean figure. The more options available to contribute to the calculation, the more normal the distribution of the replies will be.
Calculated questions exist where there is no recognised other way to evaluate an object in the domain of the EQA scheme such as an embryology laboratory. The number is unique to QAPonline and other laboratories may have other, more simplified means to describe the quality of the embryo or other entity.
Types of Information provided in each release for assessment.
The material being presented for assessment is largely image based for delivery over the Internet. The format includes IMAGES, SLIDE SHOWS, VIDEOS, ZIPPED DOWNLOAD FILES. There are a small number of EQA schemes that are sample based which require a number of samples to be dispatched. These are only for local participants and not for international distribution.
The image details are housed in an IMAGE table and contains a link to the actual image location.
INDIVIDUAL IMAGES. All images used for EQA evaluation are JPG format usually 25-50kB in size.
IMAGE SLIDESHOW contains a unique Image ID with a number of images (also JPG, 25-50kB)in an associated file. The images were multiple images from the same sample e.g. a number of slides through an embryo or multiple images from a single stained slide of sperm morphology assessment. The rule is that the total number of cells (sperm) in a slideshow should contain more than the minimum number of cells required by the standard to be counted. When the release is displayed, the first image in the slide show is displayed with progression tags to allow the participant to move progressively forward or backwards through each slide
IMAGE VIDEOS. Current images are primarily FLASH swf files of ~25 MB in size and 15-30 seconds in duration. Where possible, multiple files from the same sample can be provided. A Flag above the video provides a list of all the formats and ID is provided. In some releases, there is only 1 swf file, while in others, a swf and a YouTube version is available.
OLD VIDEO formats. Originally band width was a limiting factor as was browser types. Accordingly 1 video was provided in avi, mepg and mov (for Apple users). There was only 1 15 sec, 2-3 MB files but in 3 different versions. from 2011 and progressively in 2012 forward, the swf format is been standardised. Bandwidth does not seem to be an issue.
ZIPPED Files. IN some schemes ( eg the CASA sperm motility/morphology schemes) the video is displayed but also ZIPPED as a downloaded file. In this scheme, the file is intended to be run on the participants own CASA system and the results posted to QAPonline.
Submission of Results and data storage of summaried data.
QAPonline is designed to be used by each and every scientist in a laboratory. The results from each participants for every question/quiz is stored in a REPLY table (now over 1.2 million records). The reply table contains ID to the participant, the release (redundant data), the quiz and the enrolment. The table captures the reply and the selected option ID together with the date/time, the company and the laboratory ID.
The reply table is the single entry point for all replies. However to improve traffic flow, a copy of each reply for a QAP question is made in a replicate file. This may also provide some backup should the reply table be corrupted.
However, to avoid excessive demands on the Reply table, key information is in several other tables. The Release summary table, the quiz summary table, the company summary table and the laboratory summary table.
Individual Summary table contains a summary of the activity of each participant and the reply data for each release/scheme/participant. It is created when the participant enrols and is updated after each reply has been submitted. This table keep track of how many questions have been answered and the current performance of their key Reply from the group mean. A regular function of QAPonline is to update these tables whenever a reply has been submitted. A flag of date of last update and last reply flags which records need updating.
Quiz Summary basic statistics table contains the summarised data for each quiz. For numeric data, including the KEY QAP question, the number, mean SD, %CV, Sum, SquaredSum, Mode, Minimum and Maximium are calculated for all professional data, trimmed professional data (where all replies over 2 SD from overall data is excluded and the data recalculate), peer data (see later). The data for option based questions store the number of replies for each option (in a sub table) and the number and ID of the most popular option. This data is updated daily and automatically after 5 new replies have been received. A flag for the date of last review and last submission identifies which records need updating.
A Company (and laboratory) summary is also updated for all professional replies from participants in each company (and laboratory) for each quiz is also maintained. This is updated when a new reply has been received for a company member. The number, mean, SD , %CV, Sum, Squared Sum, Minimum and Maximium are calculated. This data is used for company and laboratory based reporting (see later). The QAP Supervisor can call a review/ update at any time.
Definition of Skill status at Enrolment and PEER participants
QAPonline has several levels of enrolment into all EQA Schemes. The allocation of a skill level is largely made by the QAP Supervisor in each QAP group but can be modified by each participant between student to professional but only the Supervisor and Administrator can upgrade a professional to PEER status
GRANDFATHER status (excuse the gender bias). Each participant is asked to identify the year when they were approved to work independently. Not every participant will complete this but for those who have, QAPonline can calculate at the time of enrolment the number of skilled years and use this in one calculation as a grandfather comparison (see later).
PROFESSIONAL Skill Level is the basic level for all statistical calculations. It is meant to represent a participant who has been approved to work unsupervised in the area of the EQA Scheme.
PEER skill level represents a professional who in the opinion of the QAP Supervisor is a peer worked in the laboratory. Their level is marked 4. In addition to each QAP supervisor marking some staff as Peer level, the administrator also reviews the performance of all participants in a EQA series who have more than 5 years state experience and whose performance indicated that more than 75% of replies have been within 1 SD of the trimmed mean and have replied to more than 25 releases. If they are from a recognised laboratory, the administrator may also mark them as a Peer enrolment in the current scheme. These are marked level 5.
STUDENT enrolment is deigned to allow unskilled students to participate in the QAPonline EQA schemes and gain experience and certification WITHOUT their replies being used in any calculation or statistics. They are marked as Level 1. Their replies can be seen by the QAP Supervisor when performing data review but their contributions are ignored.
IN TRAINING enrolment status is designed to allow staff skilled in one area gain experience and certification in another area without their submission being used in any calculation. This is an important process for laboratory staff to use as a tool to assess the performance of a new staff member when they join a laboratory of move to a new area in the same laboratory. This is especially so since QAPonline provides the tools for a Supervisor to asses the performance of a staff member both against all other participants Or only against those in their own clinic. More on External and Internal QC below.
Calculation of Basic and Peer Statistics and how they are used.
QAPonline applies a standardised approach to calculation of the basic statistic.
TRIMMED Basic Statistics are the standard scheme statistics. The replies are reviewed monthly for severe outliers (>4 SD from the mean) where administrator has the option to mark these replies as INVALID. This renders the reply inaccessible to the routine computations. Any quiz that has received a reply is marked as update pending and routinely (nightly) updated. The updating involves the calculation of the MEAN and SD all replies from professionals and peers. The Mean and SD are then recalculated as a trimmed dataset by excluding all replies that lie outside of the 2SD range of all valid replies. The complete statistics include number, Mean, SD, %CV, Maximum, Minimum and Mode.
PEER statistics are calculated at the same time and includes ALL valid peer replies for the quiz. The complete PEER statistics include number, Mean, SD, %CV, Maximum, Minimum and Mode.
PROBLEMS with defining a PEER or reference Class. One issue faced by QAPonline is defining a reference or peer population. One was that is it necessary?. While WHO standards exist, there are no training programmes or other courses to provide certification from which to attribute Peer status. In other areas such as embryology and ultrasound, there are no even standards. ESHRE Andrology EQA has a pool of laboratories that they have defined as good and from their replies, it is understood replies are pruned to created a data set with low %CV often with few data points. EQASRM has taken all IVF clinics as their reference population in what may be seen as somewhat elitist. For several years, QAPonline actively sought the participation of primary published authors and their laboratory staff to act as a peer subgroup but NOT ONE was prepared to commit to such as role. In other words, while they may have published and developed guides that have been adopted as a default standard, they were not prepared to support they default training of others to that 'standard'. QAPonline has adopted a different approach in the development of a Peer or Reference population. It sought to identify participants who have demonstrated experience (>5 years work place activity) together with a proven record of performance ( More than 75% of replies have been within one SD of the trimmed mean). The aim was to provide a dynamic and large pool of participants such that the peer data set has meaningful numbers.
FREQUENCY OF UPDATING Statistics. As indicated above, the statistics are marked for updating every time a reply has been received. Usually, the statistics are updated nightly or as soon as possible afterwards. It is proposed that from 2013, each quiz will be automatically updated when a minimum number of replies have been registered. The default difference is proposed to be 5 subject to review on speed limitations. Since QAPonline aims to deliver a immediate summary of a participants performance, the aim is to keep the statistics as current as possible. More effort is given to updating data at the beginning of each month. After 20+ replies have been received, the basis statistics rarely significantly changes.
EXTERNAL QA and INTERNAL QC. Unlike all other Proficiency testing programmes, QAPonline used to power of individual participation to provide a unique dataset from which to generate comparative statistics. as described above, since the laboratory and clinic (company) codes are included at the item of submission, analysis of the dataset can be against all professional+peer replies to form a dataset for external comparisons or restricted to only those from within the company or laboratory. Therefore, QAPonline provides dynamically, the capacity for the QAP Supervisor to assess the performance of each individual staff, laboratory or company (clinic) against the total population or their own company or laboratory as required.
QAP SUPERVISOR REVIEW. QAPonline has developed the concept that the QAP supervisor can determine the method for EQA comparison. To achieve this, a range of comparisons are available to choose. These include all all professional+peers replies, Peer replies, grandfather replies (>=10 years experience), replies from their own country. The automatic standard is the professionally +peer dataset and this data is stored in separate, up-datable tables.
INDIVIDUAL review of a participants own performance can be made at any time. QAPonline has provided the tools for an individual to assess their performance against the above links (Professionals, Peers, fellow country workers, experienced and their won company) as well as graphs of bias and performance over time. Recently, QAPonline has also introduced the 'worm' graphical presentation of the last 30 submissions. Plans are this this to also provide statistical appraisal according to the Westgard rules.
CLINIC PERFORMANCE can be assessed at any time. Unlike other Proficiency Testing Programmes, QAPonline does no issue a paper based report. Rather, QAPonline has empowered the QAP Supervisor to elect the frequency of reviewing and printing the online reports. In many clinics, the QAP supervisor or their representative, print out their performance every 3 months. They also can elect the group they wish to be compared. While almost all use the large professional data-set, the QAP Supervisor may elect to use an alternative subgroups (see above) for their own purposes and even for accreditation. QAPonline dynamically produces the reports using all the submitted replies.
LABORATORY PERFORMANCE. QAPonline has created the capacity for a company (clinic, pathology company) and to create an unlimited number of laboratories that operate under the main company licence. The QAP Supervisor can allocate each staff to laboratory and may update them as they move between laboratories. Since each reply is allocated to a company AND a laboratory, then the results remain with the laboratory they belong to at the time of submission. Therefore critically, QAPonline provides the capacity for each sub laboratory may produce reports and compare the laboratory to others in the QAP group.
INDIVIDUAL PERFORMANCE (Competency). Finally, QAPonline allows the QAP Supervisor to review the performance of each staff member in the QAP group. The review can look at each years activity scheme by scheme, a summary using QAPonline 'worm' or review a certificate using Monash Masters in Clinical embryology method for competency.
Subscriptions, Enrolments and Invoices.
Information of each companies subscription and staff enrolments and the invoice/payment for each subscription is housed in separate tables.
QAPonline creates a list of subscription options each year containing various combinations of EQA schemes available at varying subscription levels. The QAPonline model allows for clinics to subscribe at a rate appropriate for their needs and staffing levels For instance, a large clinic may wish to subscribe at a GLOBAL rate that will allow all staff access to any scheme available and appropriate for their work environment while others may only wish to have access to schemes in one discipline e.g. Andrology. A different fee is levied for each subscription combination.
A company will subscribe and pay the appropriate fee to one or more of the EQA Subscription options available. The QAP Supervisor is then in a position to select from the schemes available those of interest. The subscription and the Company/Scheme combinations are housed in separate tables. An invoice is created for all the subscriptions requested and payment in $AUD, Euros or $USD is retained in their own data files.
Each participant is then enrolled in one ore more EQA Schemes appropriate for their work environment. Enrollment can be either by the QAP Supervisor (Most common) or by the participant themselves from the list of schemes nominated by the QAP Supervisor. Each enrollment contains details of the skill level (student, training, professional or Peer. Only Professional and peer participants submissions are used by QAPOnline or statistical purposes. Once a submission has been recorded, withdrawal from the enrolment is not allowed. The year the participant was allowed to work unsupervised is also requested (but not always supplied) to allow for estimation of grandfather status ( 5-10 years).
Individual and Company/Laboratory Tables.
Finally, each individual is registered in a primary table. This data contains personal information including name, address and email. One registrant can created a QAP group (Company), laboratories and add new staff members to the QAP Group (Company).
Design of Proficiency Testing schemes
Choice of methods or procedures.
The QAPonline administrator largely defines the choice of method for assessments largely based upon experience and on feedback from participants. In some EQA Schemes there is no defined numeric based method such as in Embryology or Ultrasound. In others, the manufacturer has defined the methodology.
Internet Image Based Methodology versus Sample Based Methodology.
A primary departure in methodology for QAPonline is the use of image based processes as opposed to sample based processes. For some EQA disciplines such as Embryology or Ultrasound, a sample based EQA is not possible while for others such as Sperm DNA Fragmentation or AMH it is imperative. For others such as Andrology, there is an option to utilise either process. QAPonline has decided to use the image based process rather than the sample based processes for Andrology both to follow the line developed for Embryology and because of issues with samples currently in use for Adnrology EQA schemes elsewhere.
VISUAL basis for assessment. Unlike biochemical and similar analyses, Andrology and Embryology are largely visual based assessment. Even though there are methods for the computerisation of Semen Analysis, in may laboratories and indeed as the basis for all assessment is the visual interpretation of the sample rather than the mechanical preliminary steps prior to the inspection that are critical and where the majority of the variation lies in the generation of the final result.
An argument for sample based processes is that in the laboratory, a sample is mixed, pipetted, stained, etc before examination and the EQA scheme should reflect these technical steps in addition to the visual skills of the operator. In the assessment of sperm morphology, the WHO manual argues that an initial assessment is required for the determination of the amount of dilution that is required for the accurate measurement by haemocytometer. QAPonline acknowledges that only a sample based process can monitor this step and that some aspects of sample handling cannot be replicated by image based processes.
An argument against sample based processes primarily includes the variation due to the problem of repeated re-suspension a fixed sample where there has been significant agglutination of cells. Secondly, in the vast majority of samples, a standard dilution is used and the loading of the sample is relatively simple and uncomplicated. The largest variation is thought to be the counting and dilution calculations and this can be tested by image based processes.
A further issue involves the staining of sperm for morphology where there are varying methods for staining each of which may cause varying distortion of the sperm head. One argument is that is should be included in the EQA assessment since this is what happens on the bench. WHO recommends a standard staining method which is believed to generate cells of similar proportions and it is these methods that all standards are based. Quick staining methods are not encouraged for several reasons and any clinic using such methods must be able to justify the correlation between the standard methods and their own method. Regardless, the identification of a "normal" to an "abnormal" sample is based upon the WHO recommendation of shape.
Therefore, an image bases processes cannot replicate the few mechanical steps in assessment but does focus on the visual interpretive skills of the participant. Since many argue that this the largest source of variation between scientists and clinics, focusing on this aspect provides the greater opportunity to improve the clinics performance.
Finally, since QAPonline aims to deliver EQA services to an international clientele, delivery of sample based material internationally becomes prohibitively expensive.
In 2011, QAPonline introduced a methods variable to each EQA Series. The list of current method variables available are found on the link below.
The concept for the introduction of the methods variable was developed in response to the AMH and the Sperm DNA Fragmentation Assay but was used also in the Sperm Morphology EQA Scheme.
The methods listed for the EQA Series is available at the time of enrolment and is allocated to a specific participant. The method can be changes at anytime if need be AND is allocated to the REPLY table at the time of data submission.
The primary methodology for Semen Analysis is the World Health Organisation publication - WHO laboratory manual for the Examination and processing of human semen. There are various editions to this manual and disappointingly, there are some changes between the editions especially for the definition of sperm morphology. These variations exist largely in the definition of what is the normal range and not so much on methodology.
Concentration. 5th edition of WHO recommends only the use of haemocytometer suggesting all others tools are incorrect. QAPonline provides an image based haemocytometer EQA scheme and a Makler based EQA scheme since the majority of IVF clinics use a Makler for ease and speed. SIRT/NATA recognises this and indicates that if a tool other than a haemocytometer is used then the clinic needs to show a comparative dataset.
Motility. There were initially 4 grades of motility fast, average, slow, non-progressive and non-motile. Since the motility of a spermatozoon depends on temperature and space to swim, adn there was considerable variation in how one separated fast from average speed, WHO5 (5th edition) has removed the fast option and now has 4 speeds- progressive, slowly progressive, non-progressive and non motile. QAPonline has included all these estimates of motility as an educational tool and uses the % total motility as the KEY QAP question. This is the basic assessment of motility and common for WHO4 and WHO5.
Morphology is the most difficult parameter to assess in an ejaculate. Based upon experience and training, each operator needs to decide on shape alone if the sperm has normal morphology. There have been various definition in the past but that based upon sperm in cervical mucus (Kruger Method) has become the common standard. Implementation of this can be strict (in which cases few sperm are normal) or less strict (where more sperm are normal)is variable as is the rules for the normal range. QAPonline has 2 EQA schemes one for WHO4 (HAA - less Strict) and WHO5 (HAC - more strict) to allow participants using either of these two standards to participate independently. Minor society based variations are accommodated using the methods tool.
AntiSperm Antibodies in semen is assessed by the degree of binding of IGA,IGA beads to sperm if the sperm is coated with antibodies. Some EQA schemes use serum of either positive or negative antibody presence. QWO 4/5 indicate a normal sample is one where more than 60% of motile sperm have two or more beads attached. QAPonline has focused on the reading of the sample by supplying a video with immunobeads and motile sperm and where the KEY QAP Question is what is the % binding.
Sperm DNA Fragmentation estimates the degree that ejaculated sperm DNA is fragmented. This is a recent test but is included on NATA list of test that may be accredited. There are several methods including TUNEL (several methods), Sperm Chromatin decondensation Assay (SCSA), Halosperm Assay and COMET assay. The halosperm assay is an image based EQA scheme (see below) while the others are a sample based EQA system.
Halosperm Assay for Sperm DNA Fragmentation. This is a commercial assay from Spain and involves the dispersion and staining of Sperm DNA in a gel where the measure is the diameter of the spreading of the DNA in the gel. the greater the spread, the less DNA fragmentation is present. QAPonline presents a slideshow of the stained slides and ask for the % sperm with DNA Fragmentation as per the manufacturers recommendations. The EQA does not cover the initial preparation.
There are no clear standards for describing embryos even though there have been several books illustrating what are considered to be healthy and abnormal embryos at each stage of their development. The real problem is that morphology is loosely linked to pregnancy and that the methodology is not described in publications to indicate the rules for assessment. Neither has there been a tool to convert embryo assessment into a numeric description that will allow analysis. QAPonlines appoach is to use the commonest descriptors for an embryo at each stage of it's development and using weighted options construct a value to a value between 0 and 100 and use this construct as the KEY QAP Question for each stage. QAPonline has an EQA Scheme for each stage of embryo preimplantation development.
Oocytes. QAPonline uses the ESHRE/ALPHA guidelines for the description of oocytes, using these parameters to build a calculated numeric KEY QAP value.
PRONCULEAR stage (day 1 post fertilisation). Oocytes. QAPonline uses the ESHRE/ALPHA guidelines and those described by Lyn Scott for the description of pronuclear, using these parameters to build a calculated numeric KEY QAP value.
EARLY CLEAVAGE (between days 2 and 3). QAPonline uses the ESHRE/ALPHA guidelines for the description of day 2 and day 3 cleavage stage embryos, using these parameters to build a calculated numeric KEY QAP value.
Fragmentation. There are no rules for assessing fragmentation even though it is the cornerstone for all embryo grading schemes. QAPonline asks a simple question - what is the % fragmentation as the KEY QAP Question.
Advanced Cleavage (Morula and Blastocyst days 4 - 6). QAPonline uses the ESHRE/ALPHA guidelines for the description of day 4 and day 5/6 cleavage stage embryos, using these parameters to build a calculated numeric KEY QAP value.
Ranking of Oocytes and Embryos. Introduced as a trial scheme in 2012, the Ranking EQA scheme seeks to emulate and quantitate the ranking efficiency of embryologists. The scheme covers oocytes, pronuclear, cleavage and blastocysts stages.
There are no other EQA schemes examining follicle diameter variability. QAPonline has developed a multiple image based system where individuals needs to rotate through he images to select one that best illustrates the largest diameter (similar to what an ultrasonographer does with the US probe). When selected QAPonline provides tools to measure the distance between 2 points (using a pixel calculator) and a screen standardisation tool.
There is no standard method for measuring follicle diameter nor endometrial thickness. Rather there are publications where methods are described and used in most clinics. In real time ultrasound, the technician may re-orientate the probe between several orientations to perform measurements. However, in most clinics with severe time pressures, most will select one perspective that gives what they consider to represent the largest diameter and make all their measures form the static image.
Follicle diameter EQA. QAPonline provides for up to 4 diameters to be made on a 2D image before being averaged. From experience, many scanners perform only 1 diameter and there is no data to suggest more than 2 diameters improves the estimate.
Endometrial thickness is normally measured from the two outer echogenic bands that describe the outer vascular layer on the endometrium. QAPonline provides up to 4 estimations to measure thickness even though most operators usually only perform 1 estimate. The pattern of endometrial develop follows commonly used terminology of 2-3 patterns loosely describing the number of echogenic layers and the intensity of the central layer.
The users of the Ultrasound EQA scheme are largely nurses charged with the responsibility of performing follicle and endometrial scans. Very few ultrasonographers (if any) participate , it being the administrators opinion they cannot see the value in it since they think they are always correct! Attempts to procure the participation of skilled ultrasonographers to act as peer participants have been successfully unsuccessful!!
Endocrinology (AMH online)
QAPonline was asked to set up a AMH (Anti Mullarian Hormone) assay in 2011 on a trial basis following the rapid use in IVF programmes to predict the likelihood of hyperstimulation.
The Scheme is made available only to interested parties to check how their assays compared to other clinics ON A TRIAL BASIS ONLY. It used pooled IVF human follicle phase serum samples to generate a number of samples that covered the range of normal samples expected from an IVF population. The samples came from women who have been tested for HIV and Hepatitis B & C during their routine monitoring and stabilised using the bacterostatic ProClin 150 (Sigma). The AMH of each serum sample was known and those samples of similar value were pooled to produce 10 pooled populations.
Each pooled primary sample was then aliquoted into smaller 0.8ml samples in 2l Cryovials. Sets of 10 test samples were then stored at -20C until dispatch which was usually in the first 2 weeks of January by overnight courier in special post approved sample tubes with frozen ice pack. The overnight delivery was confirmed the following days.
There were 10 monthly samples for the months February to November and were repeated in a different order between 2011 and 2012.
The Operation of the Proficiency Testing Schemes.
Since QAPonline is a database defined structure, the operation of each scheme is similar.
Creation of EQA scheme Database Infrastructure.
Towards the end of October/early November, the EQA schemes for the following year are created. Initially, each scheme is replicated in full as a mirror of the current year. Then systemically the image linkages are modified to meet the aims of the scheme.
Initially, the linkage to each image or video is modified to ensure diversity . The new linkage is selected from the library of images/videos to check if and when it was previously used and that sufficient time has elapsed between repetitions. The release is then linked to a previous display in a separate file and used for intra-assay variability studies.
Videos are check to ensure they are in the current correct format (flash format). In 2011, QAPonline moved to replace all previous MPEG, MOV and AVI files with SWF (Flash) format. In some instances, a YouTube format and link have been created but since many clinics have banned YouTube access (due to staff usage), at present, SWF is the standard format. When these are played, a FLASH player is loaded to ensure that it will always play.
Questions are checked to ensure they reflect the image displayed (e.g. number of embryos, etc) and that a record in the summary file for each question has been created and initialised.
Creation of subscription Options
Since QAPonline primarily provides bundled subscriptions, i.e. a clinic will pay one subscription to gain access to a number of EQA schemes from which they select those they need, a number of subscription options are provided.
Global Subscription. This provides access to all the Embryology, Ultrasound and image based Andrology EQA Schemes.
Andrology bundled subscription provides access to all the Image based Andrology EQA Schemes
Embryology bundled subscription provides access to all the Image based Embryology EQA Schemes
Ultrsaound bundled subscription provides access to all the Image based Ultrasound EQA Schemes
Specialised subscription modules are created for unique, unbundled schemes such as Sperm DNA Fragmentation or Endocrine(AMH).
Subscription to any of the above schemes is usually created by the QAPonline Administrator based upon their previous subscription and a notice and invoice is sent by email to the Accounts staff manager.
Registration and Completion of a QAP Group
To initiate a subscription, one staff member, usually from the senior laboratory staff team will register with QAPonline. This requires disclosure of all address and other details. There is no fee to register.
It is possible for an individual to enrol in an EQA schemes without created but any replies cannot be linked to a future EQA group.
Set Up of the QAP Group - Laboratories, Staff,
Prior to subscribing, and following registration, the subscriber will create a QAP group with themselves as the QAP Supervisor (this can be changed at any time), create a record of all the laboratories within their clinical group or pathology company. After this they need to register each staff member who may be expected to participate in the EQA schemes. The list can be modified at nay time, new staff members can be added and retired staff members withdrawn from the staff list. Each staff member is allocated to a laboratory. A default laboratory is created when the QAP group is made to ensure all staff are allocated a laboratory linkage.
Subscription, Invoice, Payments
To gain access to the EQA Schemes, the QAP supervisor needs to access the subscription modules under the Supervisors Option and select a bundle fee subscription from the list and arrange payment of the invoice that is created.
When payment has been received, the QAP supervisor can then access the list of schemes that have been bundled into the subscription option chosen. From the list, they can select those schemes of interest to them. This selection can be changed at any time. The selected list of schemes is used to limit enrolment and review of schemes of no interest to the company.
Enrolment of Staff in EQA Schemes
The supervisor is responsible for the enrolment of each staff member in each scheme.
At enrolment, the supervisor can nominate the skill level of each staff member as either in training (such that their submission are excluded from analysis), professional or as a peer within the QAP group.
Staff may be enrolled at various skill levels in different schemes. The skill status is used as part of the internal QC analysis.
Staff enrolment can be withdrawn at any time as long as no submissions have been made. Once a submission has been received, the enrolment is fixed.
An email is posted to each staff member confirming their enrolment in each scheme.
Replies contain a link to the company, laboratory and the enrolment but the enrolment table contains a history of the skill level and experience (fro grandfather data analysis).
Individual Staff Participation.
Staff may access QAPonline and complete any release in any EQA scheme at any time. Access is data limed such that release after the current month cannot be accessed (e.g. February releases cannot be seen until February) but retrospective access is possible.
The list of current schemes and those from previous years can be accessed from the participants home page. A link to the current outstanding questions is also on the home page.
After each submission, QAPonline updates the quiz register to indicate to the administrator that another reply has been added for that quiz/question. The register of each participants activity in reach release is updated and the reply performance at that time is calculated, recorded and displayed to the user real time.
The user (Participant) can update the summary tables for each Quiz/Question and readjust their performance at any time.
The capacity for a user to review their performance real time differs considerably from other EQA programs. It is a key element in the education and training potential especially where a visual assessment has been made and where review at a later time may not be the same. The user may elect to review their performance against several other data sets as outlined in section 3.04.6
Review of performance by a staff member
Each individual as a set of tools available to review and analyse their performance. In addition, QAPonline has recently included a performance certificate (see below). This can be used for performance appraisals, job applications, competency and professional development documentation.
The first is a monthly review against a number of data sets including professional, all QAP group members (internal QC), professionals in the same country and professional of 10 years or more.
Worm Analysis where the SD from the professional mean is plotted over the last 30+ submissions for the KEY QAP question of the series.
Estimates of bias over the last 24 submissions of the KEY QAP Question.
MMCE scoring and certification. This is based upon Monash University Masters in Clinical Embryology Assessment criteria. The scoring gives 3 points to a reply within 1 SD of the mean, 2 to a reply 1-2 SD from the mean and 1 point for a reply 2-3 SD from the mean. The number of tallied for all replies and divided by the maximum possible score and converted to a percentage.
Review of the clinics and staffs performance by the QAP Supervisor
The QAP Supervisor or other nominated staff can review the performance the clinic, individual laboratories or individuals.
QAPonline has two levels of supervision. The QAP Supervisor controls staff and subscription. Other designations include a Co-Supervisor who has access to the reporting modules. See next section.
Operation of Proficiency Testing Schemes
Data Analysis and evaluation of proficiency Tests
The analysis and evaluation of each proficiency scheme is largely identical. The database structure of QAPOnline has been developed to allow uniformity in data analysis.
The EQA schemes are continuously monitored. Unlike other EQA systems that involve the distribution of samples and require a deadline for data submission to allow for data review and report preparation.
The reporting of QAPonline EQA scheme is online, dynamic and self-reporting in design.
No further notes.
Communication with participants
Communication with participants is minimal and in two primary ways. The first is a regular email encouraging completion of outstanding assessments and secondly via individual emails.
No further notes.
Communication with participants
Confidentiality is provided by use of individual logon and passwords that are system generated. QAP Supervisors are able to review the performance of each participant who are a member of their QAPgroup. Individuals are able to see the names of all other members of their QAP group but not their performance.