- View
When a “good model” fails in production
A payments team ships a fraud model that looked excellent offline. Two weeks later, the alerts are noisy, reviewers complain, and the model seems “unstable”—small data changes produce different decisions. When you dig in, the features and labels are fine (no leakage, correct example unit), but training behaves unpredictably across runs and environments.
This is often where beginners discover a quiet truth: you don’t just choose data—you choose “knobs.” Some knobs are learned from the data (the model’s internal settings), and other knobs are chosen by you (the rules of the learning process). Mixing those up makes it hard to debug, reproduce results, or explain why your model changed.
This lesson gives you a clean mental model for that split: parameters vs hyperparameters—what the algorithm learns, what you set, and how each affects performance and reliability.
The two kinds of “knobs” in a model
A machine learning model is a function that maps features → prediction. During training, the algorithm adjusts internal values so predictions match labels as well as possible on the training examples. Those internal values are parameters.
Parameters are the values the learning algorithm fits from data. They are what the model “knows” after training—numbers stored inside the trained artifact that you ship to production. In linear regression, parameters are the weights for each feature and an intercept. In a decision tree, parameters are the split choices (which feature, what threshold) and the leaf outputs. In a neural network, parameters are the weights and biases across layers. If you retrain on new data, parameters typically change, because the model is re-learning from examples.
Hyperparameters are the values you choose that shape how training happens or what family of models is even allowed. They’re not learned directly from the training data in the same way parameters are; instead they control the learning process (or model capacity). Examples include learning rate, regularization strength, tree depth, number of neighbors in k-NN, and the choice of kernel in an SVM. If you keep hyperparameters fixed and retrain, you’re asking the algorithm to learn new parameters under the same rules.
A helpful connection to your dataset contract (examples/features/labels) is this: features and labels define the learning task, while hyperparameters define the learning strategy for that task, and parameters are the learned result of applying that strategy to your examples.
What changes when you change parameters vs hyperparameters
It’s tempting to think “hyperparameters are advanced tuning,” but at beginner level they’re really about control and clarity. If parameters are wrong, training didn’t fit the data well (or didn’t fit in the right way). If hyperparameters are wrong, you may be asking the algorithm to solve the task with the wrong constraints—too flexible, too rigid, too sensitive, or too slow to learn.
A key misconception is: “Hyperparameters are just preferences; the model will figure it out.” In practice, hyperparameters often decide whether the model overfits, underfits, or behaves unreliably. For example, a very deep decision tree can memorize quirks of the training set; a very shallow tree may miss real signal. A very high learning rate in gradient descent can bounce around and never settle; a very low learning rate can take forever or get stuck looking like it “works” but underperforms. These are not cosmetic differences—they change the kind of solution the model can reach.
Another pitfall is treating hyperparameters like “free accuracy.” You can often dial up capacity (deeper trees, more layers, weaker regularization) and watch training performance improve. But if evaluation doesn’t reflect reality—say, your splits are wrong in time or your examples are correlated—those hyperparameters will reward memorization. You’ll ship a model whose parameters look confident on paper and collapse in production. Hyperparameters don’t fix a dataset definition problem; they amplify it.
To keep the concepts crisp, use this test: If you can write it into the saved model file and it changes when you retrain, it’s probably a parameter. If it’s something you set before training (or choose during model selection), and it defines the training behavior or model capacity, it’s probably a hyperparameter.
| Dimension | Parameters | Hyperparameters |
|---|---|---|
| What it is | Values learned from data during training | Values chosen by you that control learning or model capacity |
| When decided | Determined by the optimizer while fitting | Determined before/during model selection (not directly “fit”) |
| Where they live | Stored inside the trained model artifact | Stored in experiment config/training code and sometimes recorded with the model |
| What they affect | The exact mapping from features to predictions | The “shape” of models you can learn and how training behaves |
| Common failure mode | Poor fit because training didn’t converge or data isn’t informative | Overfitting/underfitting or instability because capacity/constraints are wrong |
How training actually “learns” parameters (and where hyperparameters show up)
Most supervised training can be described as: choose a loss, then minimize it. The loss function measures how wrong predictions are compared to labels (for your example unit and label definition). The training algorithm searches for parameter values that make that loss small on the training set.
For many models, that search is done with some form of gradient-based optimization: compute how a small change to each parameter would change the loss, then update parameters in a direction that reduces it. This is where hyperparameters quietly govern the dynamics. The learning rate controls the step size of each update; too large, and you overshoot; too small, and you crawl. The number of training iterations/epochs controls how long you keep updating; stop too early, and you underfit; train too long (especially with a flexible model), and you risk overfitting. Even if you never touch a neural network, this pattern appears across ML: iterative fitting with update rules whose behavior depends on chosen hyperparameters.
Another major set of hyperparameters control regularization, which is a way to discourage overly complex solutions. One common approach adds a penalty term to the loss for large parameter values (often called L1 or L2 regularization). Intuitively, regularization says: “Prefer simpler parameter settings unless the data strongly supports complexity.” The hyperparameter is the strength of that penalty. Turn it up, and parameters are pushed toward smaller magnitudes (or sparsity); turn it down, and the model is freer to fit the training data closely. This matters because real datasets contain noise, proxies, and occasional leakage-like shortcuts; regularization can reduce the chance that parameters become a perfect encoder of quirks rather than general patterns.
A practical best practice is to treat hyperparameters as part of the experiment’s “contract,” just like your feature definition. Record them every time you train. If a model changes, you want to know whether it’s because data changed (parameters changed under the same hyperparameters) or because the rules changed (hyperparameters changed). That separation is the foundation of reproducibility and debugging.
[[flowchart-placeholder]]
A simple capacity lens: why some hyperparameters matter more than others
Not all hyperparameters are equally important. A useful way to prioritize is to ask: does this hyperparameter mostly control capacity, mostly control optimization, or mostly control data processing? Capacity hyperparameters change what kinds of patterns the model can represent; optimization hyperparameters change whether you can find good parameters efficiently; data processing choices (often treated like hyperparameters in practice) change what the model sees as input, even if you call them “preprocessing settings.”
Capacity is the most conceptually important for beginners because it ties directly to overfitting and underfitting. For a decision tree, maximum depth and minimum samples per leaf are capacity controls. For k-NN, the number of neighbors controls smoothness: small k fits local quirks; large k averages more broadly. For linear models, regularization strength is a capacity control: stronger regularization restricts the space of parameter values the model can comfortably take. In each case, you’re deciding how “wiggly” the learned function is allowed to be.
Optimization hyperparameters include learning rate, batch size, solver choice, and number of iterations. These do not necessarily change the best possible model in theory, but they heavily affect whether training reaches a good solution in practice. A misconception is that optimization hyperparameters are only about speed; they also affect stability and final performance. If optimization fails to converge, your parameters are not “the model’s truth,” they’re just “where training happened to stop.”
Finally, treat some preprocessing decisions as hyperparameters because they’re choices you set and they strongly shape results: scaling numeric features, encoding categories, and the exact definition of “as-of time” in feature computation. Even though preprocessing is not a model parameter, it’s part of the system that generates parameters. If preprocessing differs between training and production, you can get failures that look like “bad parameters,” when the real issue is inconsistent inputs.
Applied example 1: House price prediction and regularization strength
Return to house price prediction: each example is a sold property, features are known-at-listing attributes (square footage, neighborhood, bedrooms), and the label is sale price. Suppose you start with a linear regression model. The parameters are the per-feature weights—how many dollars (or log-dollars) each feature contributes—and the intercept.
Now consider a hyperparameter such as regularization strength. With weak regularization, the model can assign large positive and negative weights to fit the training data closely. If your dataset includes correlated features (square footage and number of bedrooms often move together), the model may “explain” price by pushing one weight up and the other down, producing unstable coefficients. You might see this when retraining: average error changes only slightly, but weights flip in surprising ways, making the model hard to explain and brittle when the data distribution shifts.
With stronger regularization, the model is encouraged to keep weights smaller and avoid extreme tradeoffs between correlated predictors. The benefit is often stability and better out-of-sample performance, especially when the dataset isn’t huge or has noise. The limitation is that too much regularization can oversimplify: the model underfits and fails to capture genuine relationships (for example, neighborhood effects that really do matter may be shrunk too aggressively). Notice the roles: the hyperparameter sets the “simplicity preference,” and the parameters are the specific weights produced under that preference.
Operationally, this matters because the model’s downstream users may care about interpretability and consistency. If an agent-facing tool shows “bedrooms add $40k” one week and “bedrooms subtract $10k” the next, trust erodes—even if aggregate error metrics look similar. Regularization is a hyperparameter lever that can make parameters more stable and explanations more consistent.
Applied example 2: Churn prediction and tree depth (capacity) vs training randomness
For churn prediction with customer-month examples, features come from behavior up to an as-of date, and the label is whether the customer cancels in the next window. Suppose you choose a decision tree–based model. In that family, the parameters are the learned split rules and leaf predictions.
A major hyperparameter is maximum depth (or related constraints like minimum samples per leaf). A deep tree can carve the feature space into many tiny regions, which can look great on training data. But churn labels are often noisy (users stop using the product but don’t cancel immediately, cancellations are delayed, and labels can be imperfect proxies). Deep trees can end up learning the noise: “If plan=Pro and tickets_last_30_days=3 and usage_drop=17% and … then churn.” Those precise rules may not survive new months of data. A shallower tree (or more constrained tree) forces broader patterns: “Recent payment failures and sustained usage decline signal risk,” which may generalize better.
There’s another real-world issue that confuses beginners: training the “same” tree twice can produce different trees. Even with identical hyperparameters, stochastic elements (like tie-breaking, sampling, or slight data ordering differences) can make a different set of splits win. When that happens, it’s tempting to say “the parameters are unstable,” but the more actionable view is: the model family has enough flexibility that small perturbations change the learned parameters. Controlling depth (and related constraints) can reduce that sensitivity, making retrains more predictable and easier to validate.
The impact is workflow-level. A retention team often wants consistent targeting rules and stable segment sizes. If each retrain changes who is flagged dramatically, the business process becomes chaotic. Constraining capacity with hyperparameters can produce parameters that change more smoothly over time—still adapting to new behavior, but not whiplashing.
Closing: how to talk about models like an engineer
The cleanest way to stay oriented is to keep this sentence in mind: hyperparameters define the search space and the learning behavior; parameters are the specific solution found inside that space. When a model result changes, ask which side changed: the data (examples/features/labels), the hyperparameters (the rules), or the parameters (the learned values under those rules).
If you document hyperparameters, you get reproducibility. If you understand which hyperparameters control capacity, you can reason about overfitting vs underfitting without guessing. And if you treat parameters as “the learned artifact,” you can explain what retraining does: it updates the internal numbers so the model reflects the newest evidence in your examples.
Next, we’ll build on this by exploring Generalization & evaluation basics [20 minutes].