Google

In machine learning, a distinction is made between a loss function and a cost function. A **loss function** (or error function) calculates the penalty for a single training example. For a given sample consisting of input features $$x$$ and a true label $$y$$, the loss function $$L(\hat{y}, y)$$ measures the discrepancy between the model's prediction $$\hat{y}$$ and the true label $$y$$. In contrast, a **cost function** (or objective function) is typically the average of the loss function values over the entire training set, providing an aggregate measure of the model's performance.

Loss Function vs. Cost Function

When evaluating a model on a specific training instance, the loss function is calculated solely over the target sub-sequence, $$\mathbf{y}_{\mathrm{sample}}$$, instead of the full sequence. For a model defined by parameters $$\hat{\theta}^+$$, this loss is expressed as the negative log-likelihood of the probability of generating the output sub-sequence, given the input sub-sequence $$\mathbf{x}_{\mathrm{sample}}$$. The formula is:

$$\mathcal{L}_{\hat{\theta}^+}(\mathrm{sample}) = -\log \mathrm{Pr}_{\hat{\theta}^+}(\mathbf{y}_{\mathrm{sample}}|\mathbf{x}_{\mathrm{sample}})$$

Sample-wise Negative Log-Likelihood Loss for a Sub-sequence

A sequence-level loss is a type of loss function that computes the total error over an entire data sequence. Instead of evaluating each element of the sequence independently, it aggregates the individual losses from each step—often through summation or averaging—to provide a single measure of the model's performance on the whole sequence.

Sequence-Level Loss

A machine learning practitioner is debugging a training process. They notice that the error value printed after every single data point fluctuates wildly, but the error value printed at the end of each complete pass through the dataset is steadily decreasing. Explain why these two error values behave differently, using the correct terminology for each.

Interpreting Training Metrics

Match each term to its most accurate description regarding how a model's performance is measured during training.

An engineer is training a model on a large dataset. They are monitoring two metrics:

*   **Metric A:** A value calculated for each individual data sample. This value fluctuates significantly from one sample to the next.
*   **Metric B:** A single, aggregate value calculated after the model has processed the entire training dataset. This value shows a steady, downward trend over multiple passes through the dataset.

Based on the standard terminology for measuring a model's performance, what is t

Learn Before

Related