Supervised Learning Models

Linear models assume the output is a weighted sum of the input features. For regression, that's a direct linear combination. For classification, a linear boundary separates the classes. This is a strong assumption, and it pays off when the data actually has that structure: fast training, fast inference, and coefficients you can hand to a stakeholder and explain.

The catch is that real-world relationships are rarely linear. You can extend these models with polynomial features or kernel tricks (SVM), but at that point you're adding complexity that a tree-based model handles more naturally. That said, linear models make excellent baselines. If a logistic regression already solves your problem, adding a gradient booster is just overhead.

Regularization (Ridge, Lasso, ElasticNet) is critical in practice. Without it, linear models overfit whenever features outnumber samples or when features are correlated. Lasso also does implicit feature selection by driving irrelevant coefficients to exactly zero.

Instance-based models are the odd one out: there is no training phase in the traditional sense. The model just stores the training data. When a new input arrives, it finds the most similar stored examples (by distance) and uses their labels to make a prediction. All the work happens at inference time, not training time.

K-Nearest Neighbors is the canonical example. You pick K, and at prediction time it finds the K closest points in the training set (by Euclidean distance, cosine similarity, or whatever metric fits the problem) and takes a majority vote for classification or an average for regression.

The tradeoffs are real. No training cost, but inference cost scales linearly with training set size. Predictions are fully interpretable since you can point to exactly which training examples drove the output. But the model is extremely sensitive to feature scale (feature scaling is mandatory) and degrades fast in high-dimensional spaces because distance metrics stop being meaningful. This is the curse of dimensionality hitting this family the hardest.

Probabilistic models build an explicit statistical model of the data and use Bayes' theorem to compute class probabilities directly. Instead of learning a decision boundary, they model the probability of each class given the input features.

Naive Bayes is the most common member of this family. It's called "naive" because it assumes all features are conditionally independent given the class label. That assumption is almost never literally true, but the model works surprisingly well in practice, especially on text. For spam filtering and document classification it's been competitive with much more complex models for decades.

The appeal is speed and interpretability. Naive Bayes trains in a single pass through the data (just computing frequencies), handles high-dimensional feature spaces well, and works with very small datasets. The failure mode is when feature independence is badly violated and the probability estimates become unreliable, though the classification decision is often still correct even when the probabilities themselves are off.

This is the most practical family for tabular data. Tree-based models make no assumptions about feature distributions, handle missing values natively in some implementations, work on mixed numeric and categorical data, and capture non-linear interactions between features without any feature engineering.

The family splits into two strategies: bagging and boosting. Bagging (Random Forest) trains many trees independently on random data subsets and averages their predictions. Errors are reduced by variance cancellation. Boosting (XGBoost, LightGBM, CatBoost) trains trees sequentially, with each new tree correcting the errors of the previous ensemble. Boosting tends to hit higher accuracy but requires more tuning and is easier to overfit.

Gradient boosting methods dominate Kaggle competitions on tabular data for a reason: they're robust, well-regularized, handle feature interactions automatically, and scale to datasets too large for neural networks to be practical without massive compute.

Neural networks learn by stacking layers of transformations, where each layer extracts increasingly abstract features from the input. A feedforward network is the baseline: inputs flow forward through weighted layers, a loss is computed at the output, and gradients flow backward to update the weights. That's backpropagation.

From that foundation, specialized architectures handle specific data types. Convolutional networks (CNNs) exploit spatial structure in images. Recurrent networks (RNNs, LSTMs) handle sequences. Transformers replaced both of those for most tasks by 2021 and are now the architecture behind basically every large model you've heard of.

On tabular data, neural networks historically underperform tree-based models without significant effort. They need more data, more tuning, and more compute for equivalent results. Architectures like TabNet and FT-Transformer have closed the gap somewhat, but gradient boosting is still the first thing to reach for on structured data.