The difference between the R-squared statistics measuring goodness of fit in both directions: $$C_{R2} = R_y^2 - R_x^2$$

University of Michigan - Ann Arbor

Google

R-Squared ($R^2$)—the proportion of variance explained—measures how well a prediction model for "regression" fits the given data. It is independent of the scale of $Y$.
$$0 \leq R^2 \leq 1$$
The larger the R-Squared value, the closer it is to 1, the better the regression model explains the variation of observations around its mean.
$$ R^2 = \frac{TSS-RSS}{TSS} = 1 - \frac{RSS}{TSS} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2} $$

R-Squared (Coefficient of Determination)

Several cause-effect pair filters (not involving any learning) have been proposed as causation coefficients, based on various data generative assumptions. Here are some examples: - R-squared $$C_{R2}$$ - Input/Residual Independence $$C_{IR}$$ - Entropy $$C_H$$ - Slope $$C_S$$ - Conditional Distribution Standard Deviation $$C_{CDS}$$

Examples of Cause-Effect Pair Filters as Causation Coefficients

 Isabelle Guyon, Alexander Statnikov, Berna Bakir Batu (2019)
Cause Effect Pairs in Machine Learning (The Springer Series on Challenges in Machine Learning)
https://www.amazon.com/Effect-Machine-Learning-Springer-Challenges/dp/3030218090

Cause Effect Pairs in Machine Learning

What does $R^2 = 1$ Mean?

What does $R^2 = 0$ Mean?

Adjusted R-Squared is a version of R-Squared ($$R^2$$) that is adjusted for the number of predictors (independent variables) in a model. Its advantage over $$R^2$$ is that it accounts for statistical shrinkage. The normal $$R^2$$ statistic tends to increase when more independent variables occur in the system; i.e., we cannot use it to compare models with different numbers of predictors. The adjusted R-Squared can be negative, but it is usually not. It is always lower than or equal to the $$R^2$$.

$$\bar{R}^2 = 1 - (1 - R^2)\frac{n - 1}{n - p - 1}$$

Where $$p$$ is the number of predictors and $$n$$ is the sample size.

Adjusted R-Squared

The difference between statistics of independence between input and residual: $$C_{IR} = I(R_y, X) - I(R_x, Y)$$

Input/Residual Independence (Cause-Effect Pair Filter)

The difference in (discretized) entropy of the marginal distributions: $$C_H = H_x - H_y$$, with $$H_x = \sum_{k=1}^m p_k \log p_k$$ using $$m = 20$$ bins.

Entropy (Cause-Effect Pair Filter)

The difference between the conditional distributions' average standard deviation in both directions: $$C_{CDS} = CDS(X|Y) - CDS(Y|X)$$

Conditional Distribution Standard Deviation (Cause-Effect Pair Filter)

The difference in average slope in both directions: $$C_S = S_y - S_x$$, with $$S_y = \sum_{k=1}^{m-1} p_k \frac{\Delta y_k}{\Delta x_k}$$ using $$m = 20$$ intervals.

Learn Before

Related