1	CLEAR Pre-Conference Workshop Testing Essentials Job Analysis- Reed A. Castle, PhD Item Writing- Steven S. Nettles, EdD Test Development- Julia M. Leahy, PhD Standard Setting- Paul D. Naylor, PhD Scaling/Scoring-Lauren J. Wood- PhD, LP 5 topics, 20 minutes and 20 minutes Q&A
2	Job Analysis Reed A. Castle, Ph.D. Schroeder Measurement Technologies, Inc.
3	What is a Job Analysis? An investigation of the ability requirements that go with a particular job (Credentialing Exam Context). It is the study that helps establish a link between test scores and the content of the profession. The Joint Technical Standards14.14 “The content domain to be covered by a credentialing test should be defined clearly and justified in terms of importance of the content for the credential-worthy performance in an occupation or profession. A rationale should be provided to support a claim that the knowledge or skills being assessed are required for credential-worthy performance in an occupation and are consistent with the purpose for which the licensing or certification program was instituted.”
4	Why Conduct a Job Analysis? Need to establish a validity link. Need to articulate a rationale for examination content. Need to reduce the threat of legal challenges. Need to determine what is relatively important practice. Need to understand the profession before we assess it.
5	Types of Job Analyses Focus Group Traditional Survey-Based Electronic Survey-Based Transportability
6	Focus Group Need to Identify the best group of SMEs possible Areas of Practice Geographic representation Demographically Balanced 8 to 12
7	Focus Group Prior to Meeting- Comprehensive review of profession Job Descriptions Performance Appraisals Curriculum Other job-related documents Create a Master Task List Send list to SMEs prior to meeting to give them chance to review
8	Focus Group At Meeting- Review Comprehensive Task List Determine which tasks are important Determine which tasks are performed with an appropriate level of frequency Determine which tasks are duplicative Identify and add missing tasks Organize into coherent outline
9	Focus Group Advantages- May be only solution for new/emerging professions Relatively quick Less expensive
10	Focus Group Disadvantages Based on one group (Results may not generalize) May be considered a weaker model when considering validation. May result in complaints from constituents about the content of the test.
11	Traditional Survey-Based First steps are similar to the focus group (i.e., task list is generated in same manner) After the task list is created, three more issues must be addressed to complete the first survey development meeting.
12	Traditional Survey-Based First, demographic questions must be developed with two goals in mind. Questions should help describe the sample of respondents Some Question will be used for analyses help generalize across groups (e.g., geographic regions)
13	Traditional Survey-Based Second, rating scale(s) should be developed. Minimally, two pieces of information should be collected Importance or significance Frequency of performance Additional scales can be added but may take away from response rate. Shorter is sometimes better.
14	Traditional Survey-Based Sample Scale combining Importance and Frequency High correlation b/w Freq and Imp Ratings (.95 and higher) Considering both the importance and frequency, how important is this task in relation to the safe, effective, and competent performance of a Testing Professional? If you believe the task is never performed by a Testing Professional, please select the 'Not performed' rating. 0 = Not performed 1 = Minimal importance 2 = Below average or low importance 3 = Average or medium importance 4 = Above average or high importance 5 = Extreme or critical importance
15	Traditional Survey-Based Sampling- One of the more important considerations is the sampling model employed. Surveys should be distributed to a sample that is reflective of the entire population. Demographic questions help describe the sample. One should anticipate a low response rate (20%) when planning for an appropriate number of responses.
16	Traditional Survey-Based Mailing Surveys Enclose a postage paid return envelope. Plan well in advance for international mailings (can be logistically painful with different countries). When bulk mailed, plan extra time. Keep daily track of return volume.
17	Electronic Survey-Based Identical to traditional, but delivery and return are different. Need Email addresses. Need profession with ready access to Internet.
18	Electronic Survey-Based Advantages Faster response time. Data entry is no longer needed. Reduced processing time on R & D side. Possibly less expense (less admin costs). Can modify sampling and survey on the fly if needed Sample can be the population with little additional cost.
19	Electronic Survey-Based Disadvantages Need Email addresses High rate of “bounce-back” Control for ballot stuffing Data compatibility
20	Transportability Using the results of other job analysis Determine compatibility or transportability Similar to Focus Group
21	Four Types Review Focus Group Traditional Survey-Based Electronic Survey-Based Transportability
22	Data Demographics Importance Ratings Frequency Ratings Composite Sub group Analyses Decision Rules Reliability Raters Instrument Survey Adequacy
23	Primary Demographics Geographic Region Years Experience Work Setting Position Role/Function Percent Time in certain activities
24	Mean Importance Ratings- 3.0 criterion
25	% Not Performed Ratings, Criterion 25% (75% perform)
26	Composite Ratings Composite ratings using rating scale Natural Logs (when multiple scales are used) can be calculated and combined based on some weighting scheme. For example, if you want to weight frequency 33.33% and importance 66.66%, you can adjust for this in the composite rating equation. Personal opinion is that you will likely end up in a very similar place if establishing decision criteria on each scale individually. In addition, multiple decision rules is more conservative
27	Mean Importance Sub-group Analyses
28	Assessment Type SMEs are asked to determine which assessment type will best measure a given task Multiple choice Performance Essay/short answer
29	Cognitive Levels Each task on the content outline requires some level of cognition to perform 3 basic levels exist (from Bloom’s Taxonomy) Knowledge/Recall Application Analysis Steve will discuss in next presentation
30	Cognitive Levels Of the remaining tasks-post inclusion decision criteria, SMEs are asked to rate them on a 3 point scale For each major content area, an average rating is calculated The average is applied to specific criteria to determine the number of items by cognitive level for each content area
31	Weighting Weighting is usually done with SME’s based on some type of data For example, average importance or composite rating for a given content area Applied to assessment type and cognitive levels.
32	Test Specifications/Weights Standard Exclusion/Inclusion criteria Test Specifications Assessment type/Cognitive levels Weights based on rational approach Reflect test-type Statistical Consensus