CMF Clearinghouse >> Star Quality Rating
Mục lục
About the Star Quality Rating
The star quality rating indicates the quality or confidence in the results of the study producing the CMF. While the reviewers applied an objective as possible set of criteria, the star quality rating still results from an exercise in judgment and a degree of subjectivity. The star rating is based on a scale (1 to 5), where a 5 indicates the highest or most reliable rating. The review process to determine the star rating judges the accuracy and precision as well as the general applicability of the study results. Reviewers considered various factors for each study (depending on the study type used to develop the CMFs) — study design, sample size, statistical methodology, statistical significance, etc. — and judged each CMF according to its performance in the various factors (including multiple subcategories within each factor). The chart below provides a summary of the factors used for the various study types. The total score for a CMF is calculated by adding the points for the different factors, with a maximum possible score of 150.
BEFORE-AFTER AND CROSS-SECTIONAL STUDIES
FACTORS
SUBCATEGORIES
TOTAL POINTS
Data
Number of miles/sites and number of crashes for reference/comparison and treatment groups
55
Study Design andStatistical Methodology
Potential biases, accounting for trends, and appropriateness of statistical procedures
75
Statistical Significance
Statistical significance of CMF
20
META-ANALYSIS STUDIES
FACTORS
SUBCATEGORIES
TOTAL POINTS
Methodology and Data
Consistent methodology and crash type and severity definitions across included studies
55
Individual CMF Quality
Quality of studies used and standard errors of the individual CMFs.
35
AppropriatenessofCombining
Use of appropriate methods to estimate the combined CMF
40
Statistical Significance
Overall statistical significance of the combined CMF
20
META-REGRESSION STUDIES
FACTORS
SUBCATEGORIES
TOTAL POINTS
Methodology and Data
Consistent methodology and crash type and severity definitions across included studies
55
Individual CMF Quality
Quality of studies used and standard errors of the individual CMFs.
35
Appropriateness of Statistical Analysis
Use of appropriate statistical methods to estimate crash modification function including use of appropriate model and functional forms
60
The star rating is assigned based on the score and the ranges in the table below. It should be noted that information may be missing from a study report for specific characteristics such as sample size. In these cases, the rating is based on available information and the CMF will likely receive a lower rating due to the lack of information.
Score
Star Rating
135-150
5 Star
110-134
4 Star
75-109
3 Star
35-74
2 Star
0-34
1 Star
DETAILED THRESHOLDS FOR CMF CLEARINGHOUSE STAR QUALITY RATING PROCESS
The following tables provide a description of the thresholds used by CMF Clearinghouse reviewers in determining the rating for a CMF. These thresholds are consistent with the rating system developed by NCHRP Project 17-72 “Update of Crash Modification Factors for the Highway Safety Manual”. These are general “rules of thumb” followed by the reviewers to ensure consistency. However, there may be certain aspects of a study that cause it to be rated higher or lower than these general rules would indicate (for e.g., a study may report sample size for total crashes but not for the disaggregate crashes used to develop a CMF, in such a case, a reviewer can make a judgement on the sample size and allocate partial or full points). This determination is made by the reviewer based on his or her experience in statistical analysis, research experiment design, and highway safety knowledge.
RATING CRITERIA FOR BEFORE-AFTER STUDIES
Factors
Description/Subcategories
Levels[1]
Points
Total Points
Data
Number of miles/sites of reference/comparison group for SPF estimation and trend analysis (for analysis based on “segments”, number of miles should be used; for analysis based on intersections or similar units, sites should be used)
50 or more miles/sites
10
55
25 to 49 miles/sites
5
Less than 25 miles/sites, or cannot be determined
0
Number of crashes in reference/comparison sites for SPF estimation and trend analysis
500 or more crashes and 100 or more crashes per year on average
10
500 or more crashes and less than 100 crashes per year on average
7
250 – 499 crashes
5
Less than 250 crashes, or cannot be determined
0
Reference/comparison group is appropriate to account for any spillover/crash migration
Yes
5
No, or cannot be determined
0
Number of miles/sites for treatment group (for analysis based on “segments”, number of miles should be used; for analysis based on intersections or similar units, sites should be used)
50 or more miles/sites
10
25 to 49 miles/sites
5
Less than 25 miles/sites, or cannot be determined
0
Number of crashes in the before period + crashes expected in the after period without the treatment, for treatment group
200 or more
10
100 to 200
5
Less than 100, or cannot be determined
0
At least one traffic volume count in the before period and one traffic volume count in the after period
Yes
10
No, or cannot be determined
0
Confounding and Appropriateness of Statistical Analysis
Possible bias due to RTM is addressed or the treatment was a systemwide implementation
Yes
25
75
No, or cannot be determined
0
Accounts for changes in traffic volume during the study period
Yes
10
No, or cannot be determined
0
Accounts for time trends and other changes during the study period
Yes
10
No, or cannot be determined
0
Reference/comparison group is similar to treatment group in terms of AADT, i.e., the AADT range for the reference group overlaps the AADT range for the treatment group, and the mean AADT for the reference and treatment groups are similar to each other.
Yes
10
No, or cannot be determined
0
The reference/comparison group and treatment groups belong to the same roadway type (e.g., rural two lane roads) and site type (e.g., horizontal curve).In addition, the reference/comparison groups are similar to treatment group in terms of other important site characteristics.
Yes
10
No, or cannot be determined
0
The SPFs were estimated using appropriate statistical procedures and functional form is reasonable
Yes
10
No, or cannot be determined
0
Statistical Significance
CMF is statistically significant at 0.05, 0.10, or 0.15 levels.
Significant at 0.05 level
20
20
Significant at 0.10 level
10
Significant at the 0.15 level
5
Not significant at 0.15 level, or cannot be determined
0
[1] Some journal/articles do not provide the necessary information for a reviewer to rate the CMF. In such cases, the original report (if available) may need to be reviewed.
RATING CRITERIA FOR CROSS-SECTIONAL STUDIES
Factors
Description/Subcategories
Levels
Points
Total Points
Data
Number of miles/sites. Depending on the treatment being evaluated, this may include sites with and without the treatment.In other cases, the range of each independent variable of interest should be adequate (for analysis based on “segments”, number of miles should be used; for analysis based on intersections or similar units, sites should be used)
100 or more miles/sites
20
55
50 to 99 miles/sites
10
Less than 50 miles/sites, or cannot be determined
0
Number of crashes for all sites combined
700 or more
25
500 to 699
15
250 to 499
10
Less than 250, or cannot be determined
0
At least two years with actual/estimated traffic volume counts in the study period
Yes
10
No, or cannot be determined
0
Confounding and Appropriateness of Statistical Analysis
Selection bias (similarity of sites with and without the treatment)
Very similar (based on appropriate matching procedures) or other simultaneous equation procedures
25
75
Treated and untreated sites are similar in some of the characteristics (e.g., AADT or other critical site characteristics)
10
Treated and untreated sites are not similar, or cannot be determined
0
Appropriate model form (including error terms)
Yes
10
No, or cannot be determined
0
Appropriate functional form (including possibility of non-traditional non-GLM[2]forms)
Yes
10
No, or cannot be determined
0
Appropriate consideration of omitted variable bias (i.e., variables known to influence safety were considered as terms in the model or controlled through study design).
Yes
10
No, or cannot be determined
0
Appropriate consideration of correlation between independent variables (with significant correlation, the coefficient of variables may have the wrong sign)
Yes
10
No, or cannot be determined
0
Appropriate consideration of spatial and temporal correlation
Yes
10
No, or cannot be determined
0
Statistical Significance
CMF is statistically significant at 0.05, 0.10, or 0.15 levels.
Significant at 0.05 level
20
20
Significant at 0.10 level
10
Significant at the 0.15 level
5
Not significant at 0.15 level, or cannot be determined
0
[2] GLM refers to generalized linear models
Rating Criteria for Meta-Analysis Studies
Factors
Description/Subcategories
Levels
Points
Total Points
Methodology and Data
Included studies applied the same methodology and accounted for the same confounding factors, including RTM, traffic volume changes, time trends, and crash migration/spillover effects (if applicable)
Study methodologies and confounding factors accounted for identical; All confounding factors controlled for
20
55
Majority of studies applied the same methodology and accounted for the same confounding factors; All or most of confounding factors controlled for
10
Substantial variation in the applied methodology and/or confounding factors accounted for
0
Crash type and severity definitions consistent between outcome measures of studies
Yes
20
No, or cannot be determined
0
Individual estimates exhibit consistency in the direction of effect
Yes
10
No, or cannot be determined
0
Publication bias was tested for and addressed if present
Yes
5
No, or cannot be determined
0
Individual CMF Quality
A majority of studies used are deemed acceptable by the CMF Clearinghouse rating scheme
Yes
25
35
No, or cannot be determined
0
The standard error of at least one of the CMFs is less than or equal to 0.10
Yes
10
No, or cannot be determined
0
Appropriateness of Combining
A test of homogeneity indicates that the CMF estimates can be combined
Yes
20
40
No, or cannot be determined
0
Appropriate method used to estimate the combined CMF
The recommended method from 17-63 guidance followed
20
Estimates were weighted by the inverse of the standard errors or other appropriate measure but CMF values and not the natural log of the CMFs were used
10
Another method was used to estimate the overall effect
0
Statistical Significance
CMF is statistically significant at 0.05, 0.10, or 0.15 levels.
Significant at 0.05 level
20
20
Significant at 0.10 level
10
Significant at the 0.15 level
5
Not significant at 0.15 level, or cannot be determined
0
Rating Criteria for Meta-Regression Studies
Factors
Description/Subcategories
Levels
Points
Total Points
Methodology and Data
Included studies applied the same methodology and accounted for the same confounding factors, including RTM, traffic volume changes, time trends, and crash migration/spillover effects (if applicable)
Study methodologies and confounding factors accounted for identical; All confounding factors controlled for
20
55
Majority of studies applied the same methodology and accounted for the same confounding factors; All or most of confounding factors controlled for
10
Substantial variation in the applied methodology and/or confounding factors accounted for
0
Crash type and severity definitions consistent between outcome measures of studies
Yes
20
No, or cannot be determined
0
Treatment was applied similarly between locations or accounted for in the model
Yes
10
No, or cannot be determined
0
Publication bias was tested for and addressed if present
Yes
5
No, or cannot be determined
0
Individual CMF Quality
A majority of studies used are deemed acceptable by the CMF Clearinghouse rating scheme
Yes
25
35
No, or cannot be determined
0
The standard error of at least one of the CMFs is less than or equal to 0.10
Yes
10
No, or cannot be determined
0
Appropriateness of developing a CMFunction
A test of homogeneity indicates that the CMF estimates cannot be combined and exhibit more variation than expected due to random variation
Yes
10
10
No, or cannot be determined
0
Appropriateness of Combining
Appropriate model form including error terms applied
Yes
10
50
No, or cannot be determined
0
Appropriate functional form applied
Yes
10
No, or cannot be determined
0
Consideration of omitted variable bias, considered through design or included in model
Yes
10
No, or cannot be determined
0
Consideration of correlation between independent variables
Yes
10
No, or cannot be determined
0
Considered the possible impacts of country of study origin and year
Yes
10
No, or cannot be determined
0
Some journal/articles do not provide the necessary information for a reviewer to rate the CMF. In such cases, the original report (if available) may need to be reviewed.GLM refers to generalized linear models
