How to Evaluate a Custom Model

Make sense of your data.

giphy

A model rating should be the highest level where the model meets all of the criteria for that level. 

Example: If a model had a 1.2X-3.5X predictive power, AUC-ROC of 0.65, all of the ROC is above the line, and PR shape has 30% of PR averaging 0.9 precision, then the overall rating should be “Acceptable.”

ELEMENT

DO NOT USE

CAUTION

ACCEPTABLE

GOOD

EXCELLENT

Low End Predictive Power

<0.75

<1

1

1

1

High End Predictive Power

1

1.01-1.50

1.51-2.00

2.01-3.00

3.01+

AUC-ROC

<0.5

.0.5-0.6

0.6-0.7

0.7-8

0.8+

ROC Shape

50% or more of ROC is under the line

Any part of ROC goes under the line

All of ROC is above the line

All of ROC is above the line

All of ROC is above the line

PR Shape

No part of PR averages above .7 precision

1-20% of PR averages 0.7 precision

20%+ of PR average 0.7 precision

20%+ of PR average 0.8 precision

20%+ of PR average 0.9 precision

General Notes for Low Ratings:

  • Low End Predictive Power: “Model's lowest predictive power needs to be improved.”
  • High End Predictive Power: “Model's highest predictive power can be improved.”
  • AUC-ROC: “Model's predictive accuracy can be improved.”
  • ROC Shape: “Model's predictive accuracy can be more consistent.”
  • PR Shape: “Model's precision can be more consistent.”