ACA Rehab Council: Your #1 source for the latest in neuromuscular rehabilitation and state of the art doctors

The Development of a Valid Role Delineation Study for the American Chiropractic Rehabilitation Board

David D. Juehring DC, DACRB
Associate Professor, Director Palmer Chiropractic Rehabilitation and Sports Injury Department, Palmer College of Chiropractic

Palmer College of Chiropractic
1000 Brady Street
Davenport, IA 52803

Corresponding author:
David D. Juehring, DC, DACRB
Juehring_d@palmer.edu
Palmer College of Chiropractic
1000 Brady Street Davenport, IA 52803
563-884-5455
(fax) 563-884-5865

Abstract:

Introduction: A role delineation process was used to develop content, validate knowledge, and skills statements needed for competencies used by the American Chiropractic Rehabilitation Board for diplomat certification examinations.

Methods: Three phases were used in the process of completion of the role delineation. The first phase contains the initial development, evaluation of domains, tasks, knowledge, and skill statements by a 7-member role delineation expert panel. In the next phase, a representative sample of professionals (in the field of chiropractic rehabilitation) utilize Cronbach’s alpha. Finally, the third phase contains the development of test specifications based on the domains, tasks, knowledge, and skills statements.

Results: This role delineation was shown to be a valid study with all its’ reliability coefficients scores which are strong within the domains; all domains exceed the critical value of 0.7 for importance, criticality and frequency.

Conclusion: Compared to the DACRBs sample, the work of the expert panel is shown to be valid. This valid role delineation study allows a new, defensible, and content outline with testing percentages to be used by the ACRB. This is used for implementation to Rehabilitation Diplomat programs and Rehabilitation Residencies.

Keywords: role delineation, validation, domain

Introduction:

Before a content-valid examination is developed, knowledge and skills must be determined by professionals in a respective field. The process for identifying these competencies is through a role delineation, which serves as a blueprint for examination development. A role delineation is helpful in determining the content, and type of examination along with the development of a percentage breakdown in order to assess competency. Certain logically sound and legally defensible procedures for developing examinations must be followed (Standards for Educational and Psychological Testing 1999). The critical reason for conducting a role delineation study is to ensure that an examination is content-valid. Content validity is the most commonly applied and accepted validation strategy utilized in establishing certification programs (Rykiel 1996). In psychometric terms, validation is the way a test developer documents the competence inferred from a test score, which is actually measured by the examination. A content-valid examination appropriately evaluates knowledge, or skills required to function as a competent practitioner in the field. A content-valid examination contains a representative sample of items that measure the knowledge, or skills contained in the profession/group being tested. Currently, the American Chiropractic Rehabilitation Board (ACRB) is the major credentialing organization in the Chiropractic profession; they are responsible for overseeing programs and quality assurance by testing competence for the Diplomat American Chiropractic Rehabilitation Board (DACRB) credentials. The Board ensures competence, professionalism in the field of chiropractic rehabilitation, and provides assurance that the certified practitioners have met the specific criteria designed; a set standard on examinations is used to affirm their adequacy. Thus, the role delineation study is an integral part of ensuring that an examination is content-valid. In essence, the aspects of the profession covered on the examination, reflect the tasks performed in practice settings. For both broad content areas and tasks, the study identifies their importance, criticality, and frequency. These ratings play an important role in determining the content of the examination. This role delineation was performed to verify that knowledge, and skill statement (along with percentage breaks of these competencies) was content-valid for the ACRB.

Methods:

This role delineation study consisted of the following 3 phases, which are the focus of this report:

  1. Initial Development and Evaluation. The 7-member role delineation panel was assembled by a project chair. Then the panel was given the assignment to identify the domains, tasks, knowledge, and skills essential for a chiropractor to perform the most effective clinical rehabilitation. A Delphi approach was used to gather data from the panel (Rowe 1999). This material was then evaluated by the group based on importance, criticality and frequency.
  2. Validation Study. A representative sample of professionals in the field of chiropractic rehabilitation (attendees of the General Assembly meeting at the 2008 National Chiropractic Rehabilitation Convention) reviewed the above domains, tasks, knowledge, and skills statements; they validated the work of the panel again based on importance, criticality and frequency. Reliability was measured utilizing Cronbach’s alpha.
  3. Development of Test Specifications. Based on the ratings gathered from the above mentioned representative sample, the test specifications for the certification examination were developed.

Results:

Phase One – Initial Development and Evaluation:

The initial steps in the role delineation were the identification of first, the major content areas or domains, secondly, the listing of tasks performed under each domain, and finally, the identification of the knowledge and skills associated with each task. This phase was accomplished by the work by an expert panel. An ACRB elected project chair assembled the panel based on the recommendations of chiropractic rehabilitation diplomat administrators from Southern California University of Health Sciences (SCU), the Chiropractic Rehabilitation Association (CRA), National University of Health Sciences (NUH), American Chiropractic Educational Systems (ACES) and the recommendations of the ACRB along with various major authors within chiropractic rehabilitation. The panel members represented a variety of practice settings, geographic regions and gender as well as being assembled of ACRB members, SCU and CRA instructors along with a layman familiar with rehabilitation in a chiropractic setting. The following steps were undertaken to complete Phase I:

  1. The above panel was posed with the question, “What knowledge and skills does it take for a chiropractor to perform the most effective clinical rehabilitation?” 1 The panel through numerous blinded e-mail correspondences determined that the topic area of chiropractic rehabilitation could be divided into 2 major content areas or performance domains. These performance domains were as followed:
    1. Assessment
    2. Patient Care/Management
  2. Next, the panel broke down each of the domains into task statements. Thirteen tasks were developed for the Assessment domain, and 21 tasks were developed for the Patient Care/Management domain. The panel subsequently took each task statement; they broke the statements into knowledge, and skills requirements needed to perform each task. Varying numbers of knowledge and skill requirements were developed for each task statement within the respective domain.
  3. Last, the panel rated each domain and task statement within the domain, on importance, criticality, and frequency. They defined Importance as the degree to which knowledge in the domain and each task is essential in the overall job performance as a chiropractor performing rehabilitation. Criticality was defined as the degree to which the inability to perform the domain and each task in each domain would be seen as causing harm to a patient, the rehabilitation DC, the public, etc. “Harm” may be physical, emotional, financial, etc. Frequency was defined as the percent of total case time within the specific domain, and task within the domain, spent performing rehabilitative duties associated with an average rehabilitative case. With all three ratings, domains were first rated relative to the other domains, and then on completion of this requirement each task was rated relative to other tasks exclusive to the specific domain2.
    This phase of the role delineation was accomplished via numerous guided e-mails by the project chair. Based on this phase of the role delineation panel, the project chair developed an 18-page survey 3.

Phase Two – Validation Study:

The 18-page questionnaire was distributed to and completed by a representative sample of professionals in the field of chiropractic rehabilitation for evaluation. The questionnaire gave feedback on the role delineation expert panel’s domain, and task statements. Next, the sample group rated each aspect of the domains and tasks based on importance, criticality, and frequency. The survey was distributed to the attendees of the General Assembly meeting at the 2008 National Chiropractic Rehabilitation Convention in Las Vegas NV. The survey was distributed to 52 attendees with the requirement for a return of the survey within at least thirty days. Of the 52 questionnaires distributed, 34 (65.4%) usable responses were returned within the cut-off period. Biographical information was also solicited on the survey in order to ensure a representative response and completion by appropriately qualified individuals. The majority of respondents (29 or 85.3%) are male, with 5 (14.7%) female. The average age was 46.5 with a range of 34 to 64. The most common state represented in the survey was Pennsylvania with 13 (38.2%) respondents followed by Ohio with 4 (11.8%), Minnesota 3 (8.8%), California 3 (8.8%), Washington 2 (5.9%) and 1 (2.9%) for New Jersey, Maryland, Texas, Nevada, North Carolina, Missouri, South Dakota, Oregon and Arkansas. Note: due to rounding error the state total is 99.6%. Continuing education program sponsors for DACRB certification programs represented in the study was SCU, CRA, ACES, Palmer Chiropractic College and Canadian Memorial Chiropractic College4.

The reliability of the respondents’ results of the survey was assessed relative to the panel’s results in order to determine how well the tasks consistently measured the domain of question. Reliability refers to the degree to which tests or surveys are free from measurement error. Reliability was measured by internal consistency via Cronbach’s alpha (Reynaldo 1999) using the respondent’s ratings of importance, criticality, and frequency for each domain in order to draw defensible conclusions. Cronbach’s alpha can be written as a function of the number of test items, and the average inter-correlation among the items. Below, for conceptual purposes, we show the formula for the standardized Cronbach’s alpha:

Here it states that N is equal to the number of items, c-bar is the average inter-item covariance among the items, and v-bar equals the average variance. Calculations were performed on an Excel spreadsheet for all domains and tasks within the domains. Only the calculation of the Assessment Domain Frequency was shown5 and all other calculation results are listed in the below Reliability Table. Reliability coefficients can be low, when the pattern of respondents to a particular task in a domain is different from the pattern of respondents to the other tasks in the domain. This calculates the extent to which each task rating within each domain consistently measures what other tasks within that domain measure. Reliability coefficients range from 0 to 1 and should be above 0.7 to be judged as adequate (Reynaldo 1999). Reliability values below 0.7 indicate an unacceptable amount of measurement error (Reynaldo 1999). As shown below, the reliability statistics are strong since all domains exceed the critical value of 0.7.

RELIABILITY TABLE
Domain Importance Criticality Frequency
I. Assessment 0.79 0.83 0.89
II. Patient Care/Management 0.93 0.95 0.96

Phase Three – Development of Test Specifications:

The final phase of the role delineation study was the development of test specifications. These identify the proportion of questions from each domain and task and appear on the DACRB examinations. Test specifications are developed by combining the overall evaluations of importance, criticality, frequency, and the converting of the results into percentages. These percentages are listed below; they are used to determine the number of questions related to each domain and task that should appear on examinations.

Domain Test Blueprint % of Test
I. Assessment 19 % II. Patient Care/Management 81 %

The results of the Test Percentage Specific Document6 were forward to the ACRB to ultimately be distributed to the diplomat certification programs as content guidelines for their education programs.

Conclusion:

The expert panels’ content and outline are valid compared to the sample DACRBs, including the testing breakdowns. The results of this role delineation study were determined to be valid. The new content outline and testing percentages were forwarded to the ACRB for implementation to Rehabilitation Diplomat programs and Rehabilitation Residencies. Each reliability rating for importance, criticality, and frequency were all based on Cronbach’s alpha. Therefore the domains, tasks, knowledge, and skills developed by the role delineation panel constitute an accurate definition of the work of the credentialed DACRB.

References:

  1. Reynaldo J, Santos A 1999 Cronbach’s Alpha: a tool for assessing the reliability of scales. Journal of Extension 37(2)
  2. Rowe G, Wright G 1999 The Delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting 15(4):353-375
  3. Rykiel EJ 1996 Testing ecological models: meaning of validation. Ecological Modeling, 90:229-244
  4. Standards for Educational and Psychological Testing published by the American Educational Research Association, 1999

 

© Copyright 2015
American Chiropractic Association Rehab Council