Best Model Binary Outcome Continuous Predictor
A team of researchers collects and publishes detailed information about factors that affect heart disease. Variables include age, sex, cholesterol levels, maximum heart rate, and more. This example is based on a public data set that gives detailed information about heart disease. The original data are from archive.ics.uci.edu.
The researchers want to find a model that makes the most accurate predictions that are possible. The researchers use Discover Best Model (Binary Response) to compare the predictive performance of 4 types of models: binary logistic regression, TreeNet®, Random Forests® and CART®. The researchers plan to further explore the type of model with the best predictive performance.
- Open the sample data, HeartDiseaseBinaryBestModel.MTW.
- Choose .
- In Response, enter 'Heart Disease'.
- In Continuous predictors, enter Age, 'Rest Blood Pressure', Cholesterol, 'Max Heart Rate', and ' Old Peak'.
- In Categorical predictors, enter Sex, ' Chest Pain Type', 'Fasting Blood Sugar', 'Rest ECG', 'Exercise Angina', Slope, 'Major Vessels', and Thal.
- Click OK.
Interpret the results
The Model Selection table compares the performance of the different types of models. The Random Forests® model has the minimum value of the average –loglikelihood. The results that follow are for the best Random Forests® model.
The Misclassification Rate vs Number of Trees Plot shows the entire curve over the number of trees grown. The misclassification rate is approximately 0.16.
The Model summary table shows that the average negative loglikelihood is approximately 0.39.
The Relative Variable Importance graph plots the predictors in order of their effect on model improvement when splits are made on a predictor over the sequence of trees. The most important predictor variable is Thal. If the contribution of the top predictor variable, Thal, is 100%, then the next important variable, Major Vessels, has a contribution of 98.9%. This means Major Vessels is 98.9% as important as Thal in this classification model.
The confusion matrix shows how well the model correctly separates the classes. In this example, the probability that an event is predicted correctly is approximately 87%. The probability that a nonevent is predicted correctly is approximately 81%.
The misclassification rate helps indicate whether the model will accurately predict new observations. For prediction of events, the out-of-bag misclassification error is approximately 13%. For the prediction of nonevents, the misclassification error is approximately 19%. Overall, the misclassification error for the test data is approximately 16%.
The area under the ROC curve for the Random Forests® model is approximately 0.90 for the out-of-bag data.
Discover Best Model (Binary Response): Heart Diseas vs Age, Rest Blood P, ...
Method Fit a stepwise logistic regression model with linear terms and terms of order 2. Fit 6 TreeNet® Classification model(s). Fit 3 Random Forests® Classification model(s) with bootstrap sample size same as training data size of 303. Fit an optimal CART® Classification model. Select the model with maximum loglikelihood from 5-fold cross-valuation. Total number of rows: 303 Rows used for logistic regression model: 303 Rows used for tree-based models: 303
Binary Response Information Variable Class Count % Heart Disease 1 (Event) 165 54.46 0 138 45.54 All 303 100.00
Model Selection
Average Area Under Misclassification Best Model within Type -Loglikelihood ROC Curve Rate Random Forests®* 0.3904 0.9048 0.1584 TreeNet® 0.3907 0.9032 0.1520 Logistic Regression 0.4671 0.9142 0.1518 CART® 1.8072 0.7991 0.2080 * Best model across all model types with minimum average -loglikelihood. Output for the best model follows.
Hyperparameters for Best Random Forests® Model Number of bootstrap samples 300 Sample size Same as training data size of 303 Number of predictors selected for node splitting Square root of the total number of predictors = 3 Minimum internal node size 8
Misclassification Rate vs Number of Trees Plot
Model Summary Total predictors 13 Important predictors 13 Statistics Out-of-Bag Average -loglikelihood 0.3904 Area under ROC curve 0.9048 95% CI (0.8706, 0.9389) Lift 1.7758 Misclassification rate 0.1584
Confusion Matrix Predicted Class (Out-of-Bag) Actual Class Count 1 0 % Correct 1 (Event) 165 143 22 86.67 0 138 26 112 81.16 All 303 169 134 84.16 Out-of-Bag Statistics (%) True positive rate (sensitivity or power) 86.67 False positive rate (type I error) 18.84 False negative rate (type II error) 13.33 True negative rate (specificity) 81.16
Misclassification Out-of-Bag Actual Class Count Misclassed % Error 1 (Event) 165 22 13.33 0 138 26 18.84 All 303 48 15.84
Source: https://support.minitab.com/en-us/minitab/21/help-and-how-to/statistical-modeling/predictive-analytics/how-to/discover-best-model-binary-response/before-you-start/example/