Classification and logistic regression

Section 36.1 Classification and logistic regression

We focus on the case of two categories, represented by the binary outcome variable \(z\) attaining two possible values, \(0\) and \(1\text{.}\) The model may involve one or more explanatory variables \(x_k\text{,}\) and the goal is to predict the outcome based on these variables. Some examples:

Explanatory variables: temperature, humidity, atmospheric pressure. Outcome: rain (1) or no rain (0).

🔗
Explanatory variables: patient’s weight, height, age, activity level. Outcome: has diabetes (1) or not (0).

🔗

In the examples such as these, it is understood that the prediction may be far from certain. A way to think of such a model as assigning a probability of outcome (1), based on the data available. This leads to the idea of a model being a function that takes values between \(0\) and \(1\text{.}\) The standard logistic function (Section 29.2) has this property:

\begin{equation*} \sigma(x) = \frac{1}{1+\exp(-x)} \end{equation*}

In order to include several explanatory variables \(x_1, \dots, x_m\) in this model, we use the composite function

\begin{equation} h_{\mathbf c}(\mathbf x) = \sigma\left(\sum_{k=1}^m c_k x_k\right) = \frac{1}{1+\exp\left(-\sum_{k=1}^m c_k x_k\right)}\tag{36.1.1} \end{equation}

where the coefficients \(c_1, \dots, c_m\) are to be determined by training the model on some data set with known outcomes.

🔗

In Chapter 29 our training method was to minimize the sum of squares

\begin{equation*} S(\mathbf c) = \sum_{k=1}^n (z_k - h_{\mathbf c}(\mathbf x_k))^2 \end{equation*}

However, in the context of probability there is a more natural method: maximizing the likelihood function. Think of \(h_{\mathbf c}(\mathbf x_k)\) as the probability that \(z_k=1\text{,}\) and therefore \(1 - h_{\mathbf c}(\mathbf x_k)\) is the probability that \(z_k=0\text{.}\) Assuming the independence of outcomes, the probability of observing the data that was actually observed is

\begin{equation} L(\mathbf c) = \prod_{z_k = 1} h_{\mathbf c}(\mathbf x_k) \cdot \prod_{z_k = 0} (1-h_{\mathbf c}(\mathbf x_k))\tag{36.1.2} \end{equation}

Since the outcomes \(z_k\) already occurred, this expression is not really the probability of a random event; it is the likelihood of the parameters being \(\mathbf c\text{.}\) We want to maximize this likelihood. Since the product may be very small when there are many observations, it is easier to maximize the log-likelihood, the logarithm of (36.1.2):

\begin{equation} \log L(\mathbf c) = \sum_{z_k = 1} \log h_{\mathbf c}(\mathbf x_k) + \sum_{z_k = 0} \log (1-h_{\mathbf c}(\mathbf x_k))\tag{36.1.3} \end{equation}

🔗

Once we find \(\mathbf c\) which maximizes \(\log L\text{,}\) the model can then be used to make predictions about unobserved outcomes on the basis of explanatory variables \(\mathbf x\text{.}\) If \(h_{\mathbf c}(\mathbf x)\) is close to \(1\text{,}\) we expect the outcome \(z=1\text{;}\) it is close to \(0\text{,}\) we expect the outcome \(z=0\text{.}\) The quantity \(h_{\mathbf c}(\mathbf x)\) may be interpreted as the probability of \(z=1\text{,}\) predicted by the model.

🔗

Unless the explanatory variables \(\mathbf x\) are centered (have zero mean by design), the exponential should include a constant term, as shown in the following example.

🔗

Example 36.1.1. Smoking, age and blood pressure.

The following data, taken from Matlab’s built-in patients data sample (reference), records the ages and systolic pressure of 10 smokers (\(z=1\)) and 12 nonsmokers (\(z=0\)). Train a logistic regression model on this data.

🔗

age1 = [38 33 39 48 32 27 44 28 30 45];
sys1 = [124 130 130 130 124 123 128 129 127 134];
age0 = [43 38 40 49 46 40 28 31 45 42 25 36];
sys0 = [109 125 117 122 121 115 115 118 114 115 127 114];

Then test the model on a separate data set, generating predictions for the people based on their age and systolic pressure:

🔗

age = [38 45 30 48 48 25 44 49 45 48];
sys = [138 124 130 123 129 128 124 119 136 114];

Finally, compare the model’s prediction with actual smoker status in the test dataset.

🔗

actual = [1 0 0 0 0 1 1 0 1 0];

Solution.

We set up the log-likelihood function LogL, maximize it, and use the optimal parameters cc to predict the status of the 10 people who were not a part of the training set.

🔗

LogL = @(c) sum(log(1./(1+exp(-c(1)*age1-c(2)*sys1-c(3))))) + sum(log(1 - 1./(1+exp(-c(1)*age0-c(2)*sys0-c(3)))));
cc = fminsearch(@(c) -LogL(c), [0; 0; 0]);

age = [38 45 30 48 48 25 44 49 45 48];
sys = [138 124 130 123 129 128 124 119 136 114];
prediction = 1./(1+exp(-cc(1)*age-cc(2)*sys-cc(3)));
actual = [1 0 0 0 0 1 1 0 1 0];
disp([prediction' actual']);

Excluding the cases where prediction is a number close to \(0.5\) (which should be considered a “don’t know” answer), the model got 6 out of 8 right.

🔗

Prev Top Next