College of Physicians and Surgeons of Ontario

Background: Risk and protective factors influencing the performance of health professionals are of significant interest to regulators and the public. We aimed to develop a predictive model to identify factors influencing optometrist performance, providing insights for improving regulatory oversight and supporting targeted interventions.Methods: In our retrospective cohort study, we analyzed data from optometrists registered between 1987 and 2019 in the Alberta College of Optometrists Continuing Competence (CC) program to develop a predictive model for CC practice review outcome. We evaluated reviews using self-assessments, onsite visits, and clinical evaluations, with pass or fail status as the primary outcome. Key covariates included sex, age, training location, and previous review scores. We used a generalized additive model with a logit link and assessed its performance using five-fold cross-validation. Sensitivity and specificity were assessed with a holdout testing set.Results: We analyzed 2,075 CC reviews of 916 optometrists. Of these reviews, 75.6% received a passing grade. Practitioners were primarily male (51.7%, 48.3% female) and trained in the United States (49.8%) or Canada (46.2%). Significant predictors of review outcome were sex, training location, previous review score, follow-up score, age (included as a nonlinear effect varying by sex), and years since last review. In developing a selection tool for future assessments, we replaced age with years since graduation and removed training location. Among the 388 practitioners selected for assessment since 2021, practitioners flagged as high risk had significantly higher failure rates (16.1%) compared with practitioners selected randomly (3.0%).Discussion: Male sex, years since graduation, and poor outcomes on previous reviews emerged as significant predictors of failing an assessment. The developed selection tool effectively identified high-risk practitioners for reassessment, supporting fair and efficient resource allocation in the CC program.Conclusions: Key factors influencing CC review outcomes were identified and a selection tool was developed to ensure fairness across subgroups defined by age and sex.

Abstract Background Clinical practice guidelines (CPGs) synthesize high-quality information to support evidence-based clinical practice. In primary care, numerous CPGs must be integrated to address the needs of patients with multiple risks and conditions. The BETTER program aims to improve prevention and screening for cancer and chronic disease in primary care by synthesizing CPGs into integrated, actionable recommendations. We describe the process used to harmonize high-quality cancer and chronic disease prevention and screening (CCDPS) CPGs to update the BETTER program. Methods A review of CPG databases, repositories, and grey literature was conducted to identify international and Canadian (national and provincial) CPGs for CCDPS in adults 40–69 years of age across 19 topic areas: cancers, cardiovascular disease, chronic obstructive pulmonary disease, diabetes, hepatitis C, obesity, osteoporosis, depression, and associated risk factors (i.e., diet, physical activity, alcohol, cannabis, drug, tobacco, and vaping/e-cigarette use). CPGs published in English between 2016 and 2021, applicable to adults, and containing CCDPS recommendations were included. Guideline quality was assessed using the Appraisal of Guidelines for Research and Evaluation (AGREE) II tool and a three-step process involving patients, health policy, content experts, primary care providers, and researchers was used to identify and synthesize recommendations. Results We identified 51 international and Canadian CPGs and 22 guidelines developed by provincial organizations that provided relevant CCDPS recommendations. Clinical recommendations were extracted and reviewed for inclusion using the following criteria: 1) pertinence to primary prevention and screening, 2) relevance to adults ages 40–69, and 3) applicability to diverse primary care settings. Recommendations were synthesized and integrated into the BETTER toolkit alongside resources to support shared decision-making and care paths for the BETTER program. Conclusions Comprehensive care requires the ability to address a person’s overall health. An approach to identify high-quality clinical guidance to comprehensively address CCDPS is described. The process used to synthesize and harmonize implementable clinical recommendations may be useful to others wanting to integrate evidence across broad content areas to provide comprehensive care. The BETTER toolkit provides resources that clearly and succinctly present a breadth of clinical evidence that providers can use to assist with implementing CCDPS guidance in primary care.

We thank Dr. Barrett and colleagues and Dr. Harter for their thoughtful Letters regarding our recent JMR article titled “Do Medical Licensing Questions on Health Conditions Pose a Barrier to Physicians Seeking Treatment? A Literature Review.”1As regulators, we regulate in the public interest. Unless we could conclude that mental health never impacted fitness to practice, it would be difficult to remove a requirement for reporting some mental health concerns.As a self-regulating profession, we must work with physicians to address health concerns in a fair and compassionate manner to ensure physicians are healthy to care for patients.The challenge remains finding the right balance.

This paper describes the use of Kane’s validity framework to redevelop a workplace-based assessment program for practicing physicians administered by the College of Physicians and Surgeons of Ontario. The developmental process is presented according to the four inferences in Kane’s model. Scoring was addressed through the creation of specialty-specific assessment criteria and global, narrative-focused reports. Generalization was addressed through standardized sampling protocols and assessor training and consensus-building. Extrapolation was addressed through the use of real-world performance data and an external review of the scoring tools by practicing physicians. Implications were theoretically supported through adherence to formative assessment principles and will be assessed through an evaluation accompanying the implementation of the redeveloped program. Kane’s framework was valuable for guiding the redevelopment process and for systematically collecting validity evidence throughout to support the use of the assessment for its intended purpose. As the use of workplace-based assessment programs for physicians continues to increase, practical examples are needed of how to develop and evaluate these programs using established frameworks. The dissemination of comprehensive validity arguments is vital for sharing knowledge about the development and evaluation of WBA programs and for understanding the effects of these assessments on physician practice improvement.

IntroductionThe professional regulation sector is moving toward risk-informed approaches that require high quality data. A key component of a corporate 2017 Data Strategy is the implementation of a data inventory and mapping project to catalogue, centralize, document and govern data assets that support regulatory decisions, programs and operations. Objectives and ApproachIn a data rich organization, the goals of the data inventory are to: enhance authoritative data that support programs; identify data duplications/gaps; identify data sources, owners and users; and, apply consistent data management and standards organizationally. Routinely used data assets outside the large enterprise workflow system (excel/word files; databases; paper collections) were catalogued. Using data governance principles and a facilitated questionnaire, departmental data stewards were interviewed about their generated data. Questions included data purpose/sources/types/formats/owners, retention rates, analytical products, gaps and visions for a desired data state. A data mapping methodology highlighted data set and variable connections within and across departments. ResultsTo date, over 40 staff members in 10 departments were identified as data content experts. In addition to data in the corporate enterprise system, over 80 unique datasets were identified. In 1 large department, over 2,000 data elements across 26 datasets were inventoried. Data mapping analysis revealed thematic data domains, including member demographics, outcomes, certifications, tracking and financial data, collected and held in multiple formats ((Microsoft Access, Excel, Word), SPSS, PDF, e-mails and paper documents). While 72% of the data elements were formatted numerically, approximately 8% were free text. Significant data redundancies across staff members and departments were revealed, as well as unstandardized variable naming conventions. Gaps analysis highlighted need for standardized, electronic data, where not available and data management training. Conclusion/ImplicationsCustomized data mapping reports to data users will facilitate the development of local, standardized departmental data hubs that will centrally link to a centralized data repository to facilitate seamless organization-wide analytics, improvements in current data management practices and greater data collaboration with the ultimate goal of supporting risk-informed approaches.