We apply it to the abalone dataset, predicting sex from physical measurements. Unlike the logistic regression post, which dropped infants to get a binary problem, here we keep all three categories — M, F, and I — because majority voting handles any number of classes without changing anything. That makes this the first genuinely multi-class problem in the series.

→ open the interactive demo

the model

The idea is simple. To classify a new abalone, we find the \(k\) training abalone most similar to it, and let them vote — the majority class among those \(k\) neighbors is the prediction.

“Most similar” means closest in feature space. We measure closeness with Euclidean distance — the straight-line distance between two points across all \(n\) features:

where \(\mathbf{x}\) is the point we want to classify and \(\mathbf{x}_i\) is one training sample. We compute this distance to every training point, sort ascending, take the \(k\) smallest, and return the most common label among them:

where \(y_{(1)}, \ldots, y_{(k)}\) are the labels of the \(k\) nearest neighbors. Because it is just a count, this generalizes to three classes — or thirty — with no extra machinery, which is why we can keep infants in the problem.

No training, no parameters. This is the defining feature of KNN. There is no gradient descent, no loss function, nothing to minimize. fit() only stores the (normalized) training data; all the actual work happens at prediction time, when the distances are computed. This style is called lazy or instance-based learning: the model is the training set. It is the mirror image of the regression models, which spent all their effort in fit() and then predicted instantly.

The decision boundary is jagged. KNN never writes down a boundary, but one exists implicitly. If you colored every point in feature space by the class KNN would assign it, you would see regions — one per class — and the borders between them are not straight. They bend and wander around individual training points. At \(k=1\) the regions are the Voronoi cells around each training sample, maximally irregular; as \(k\) grows they smooth out. This is the sharp contrast with logistic regression, whose boundary is always a single flat hyperplane. KNN can carve arbitrary, curved regions — it trades the linear model’s rigidity for flexibility. As we will see, that flexibility is also its weakness.

regression, too. KNN is not only a classifier. To predict a continuous target — ring count, say — you swap the majority vote for the mean of the \(k\) neighbors’ values. Everything else stays identical. This post stays on classification, but the same code is one line away from the regression setting.

distance and normalization

Euclidean distance sums contributions from every feature, which means a feature’s scale decides how much it matters. The abalone measurements live on very different scales — weights in grams, lengths in millimetres, ring counts as integers up to 29 — so without rescaling, whichever feature has the largest numeric range would dominate the distance and effectively pick the neighbors on its own. The others would barely register.

KNN is therefore even more sensitive to feature scaling than the gradient-descent models, where scaling only affected convergence speed. Here it changes the answer. So we standardize every feature to zero mean and unit variance before computing any distance:

Here \(\mu\) is the feature’s mean and \(\sigma\) its standard deviation, both measured across the training samples. Subtracting \(\mu\) recenters the feature on zero; dividing by \(\sigma\) rescales it so one unit equals one standard deviation. The transformed value \(z\) then reads as “how many standard deviations above or below average this measurement is” — a common language every feature speaks, whether it started out as grams, millimetres, or ring counts.

data leakage. as in the regression posts, \(\mu\) and \(\sigma\) are computed from the training set only and reapplied to the test data without recomputing — see linear regression. Computing them from the full dataset would leak information about the held-out points into the model.

With every feature on the same scale, a millimetre of length and a gram of shell weight contribute comparably to the distance, and the neighbors are chosen on overall similarity rather than on whichever column happened to have the biggest numbers.

choosing k

\(k\) — the number of neighbors — is the model’s only knob, and it controls the bias–variance tradeoff directly and visibly.

Small \(k\) makes the model sensitive to individual points. At \(k=1\), each prediction is simply the label of the single closest training sample; the decision regions are the jagged Voronoi cells described above. This fits the training data perfectly — including its noise — which is high variance: move one training point and the boundary shifts. On unseen data it is brittle.

Large \(k\) averages over a wide neighborhood. The boundary smooths, small local structures wash out, and in the limit \(k \to m\) every query just returns the global majority class. That is high bias: the model is too rigid to follow the real structure.

The right \(k\) sits between these extremes, and we find it by cross-validation rather than by guessing.

Ties. With two classes an odd \(k\) can never tie, but with three (M, F, I) it still can — \(k=5\) might split 2–2–1. We break such ties toward the nearest neighbor: among the classes with the top count, the one whose closest member is nearest wins. This keeps predictions deterministic and gently favors the more similar class.

k-fold cross-validation

We cannot choose \(k\) by training accuracy, and KNN makes the reason especially stark: because the model is the training set, predicting on the training data lets every point find itself as its own nearest neighbor (distance zero). At \(k=1\) that gives 100% training accuracy — a number that says nothing about how the model generalizes.

So we evaluate on held-out data. The mechanism is the same k-fold cross-validation used in the regression posts: split the data into 5 folds, and for each candidate neighbor count, train on the other 4 folds and score on the held-out one, then average the 5 scores. That average is a single held-out estimate of how well that value of \(k\) generalizes.

We compute that estimate for every candidate \(k\) and keep the one with the highest average validation accuracy — that is how \(k\) is chosen. The sweep across all candidates is exactly what the grid search (next section) plots, and the winning \(k\) is the value used for every figure that follows.

One naming collision is worth flagging: the \(k\) in “k-nearest neighbors” (the number of neighbors) is not the \(k\) in “k-fold” (the number of folds). We use 5 folds to choose the best number of neighbors.

grid search

To pick the neighbor count automatically, we sweep a range of values, score each by its average validation accuracy across the folds, and keep the best. It is the same procedure as the \(\lambda\) grid search in the earlier posts — only the hyperparameter has changed: we are searching over the number of neighbors instead of a regularization strength, and selecting the value with the highest validation accuracy.

a note on running time. KNN flips the cost structure of the series. There is no training loop — fit() just stores the data — but every prediction compares the query against all \(m\) training points, so scoring a whole set is \(O(m^2)\). That makes the grid search, which re-scores across folds and neighbor counts, the expensive part. Reduce the k max or cv folds sliders if you want faster results; raise them for a finer search.

interpreting this run

k = 39 found by grid search · 5-fold cross-validation

The chosen model gets a little over half of the held-out abalone right — about 54% overall — but that single number averages three very different outcomes, so it is worth reading class by class.

Confusion matrix

Rows are the actual class, columns the predicted class; each row sums to 100%, and a darker diagonal means better classification.

One row behaves; two do not. Infants land on the diagonal 80.3% of the time, leaking only slightly to M (12.0%) and F (7.7%). The male and female rows tell the opposite story: just 48.7% of males and 34.6% of females are labeled correctly, and the misses pile up between M and F rather than spilling into I. The most revealing cell is the top middle — actual females are called male 49.4% of the time, more often than they are called female. And the F and M rows are near-copies of each other (34.6 / 49.4 / 16.0 versus 31.2 / 48.7 / 20.1), exactly what you would expect if the two sexes sit in the same region of feature space: a point’s neighbors are a mix of both, and the vote tips toward whichever is locally denser.

Per-class accuracy

The fraction of samples correctly classified within each sex category — this is the diagonal of the confusion matrix, read one class at a time.

The gap is the whole result. Infants (~80%) are easy: smaller and lighter across every measurement, so their nearest neighbors are almost all other infants. Adults are not — males (~49%) and females (~35%) overlap so heavily that even 39 neighbors cannot pull them apart, and the model is barely better than a coin flip between the two. The asymmetry matters too: females do worse than males because M is the more common adult class, so wherever the neighborhood is mixed the majority vote leans male and the females pay for it.

Prediction distribution

For each actual class, where the model’s predictions land — the off-diagonal dots are where confusion happens.

The same story, as dots instead of percentages. The bottom row — actual infants — collects into the I column with only a thin scatter elsewhere. The top two rows spread almost evenly across the F and M columns, with just a light tail into I. There is no tight diagonal for the adults; the off-diagonal cells are nearly as full as the on-diagonal ones. That is the signature of an honest failure, not a buggy one: the model makes the same structured mistake everywhere — confusing the two adult sexes — because the measurements genuinely do not separate them.

Grid search — k vs validation accuracy

Cross-validation accuracy across the candidate neighbor counts, with the best \(k\) highlighted.

This is the bias–variance tradeoff in a single curve. At \(k=1\) accuracy sits around 49% — each prediction just copies its single nearest neighbor, noise included. (Note it is ~49%, not the 100% you would get by scoring on the training data; the held-out split keeps it honest.) Accuracy rises steeply as \(k\) grows and the vote begins averaging out that noise, then flattens into a broad plateau from about \(k=20\) on, peaking at \(k=39\) near 55.7%. The plateau is nearly flat — \(k=39\) is barely ahead of \(k=25\) or \(k=55\) — so the exact winner is not meaningful; what matters is that the model needs a large neighborhood to do its best. That is itself a finding: local structure is too noisy to trust, so the model does better polling dozens of neighbors than a few. Push \(k\) higher still and the curve drifts back down, as the neighborhood grows wide enough to wash out real structure — the underfitting end of the tradeoff.

where KNN struggles

Every feature counts equally. Distance treats all features the same, so an irrelevant or noisy feature adds noise to every comparison. A linear model can drive a useless coefficient to zero; KNN has no such mechanism — it cannot down-weight a feature, only be dragged around by it. Careful feature selection and scaling matter more here than in any model so far.

No model, no explanation. KNN returns predictions but learns no structure. There are no weights to interpret, no equation, nothing to tell you why two sexes are hard to separate — only that the neighbors disagreed. The regression posts at least handed us coefficients to argue over; KNN hands us nothing but the data itself.

The cost lives at prediction time. Lazy learning has a price: there is no compact model to ship, and every prediction scans the whole training set. Cost grows with the data, exactly where the regression models stayed constant.

None of this makes KNN a bad choice — on the right problem, with few, well-scaled, relevant features, a model that simply asks “what do the nearest examples look like?” is hard to beat. It just asks more of the data, and of us, than a model that compresses everything into a handful of weights.

references

• Russell, S. and Norvig, P. Artificial Intelligence: A Modern Approach, 4th ed. Pearson, 2021. §12.6 (nearest-neighbor models), §19.7.1 (nonparametric classification).

• James, G., Witten, D., Hastie, T., Tibshirani, R., and Taylor, J. An Introduction to Statistical Learning with Applications in Python. Springer, 2023. §3.5 (KNN regression), §4.4.4 (KNN classification).

• K-Nearest Neighbor (KNN) Algorithm — GeeksForGeeks

• The Bias-Variance Tradeoff — Towards Data Science

→ open the interactive demo

source code: github.com/jjginga/ferrolearn

We apply it to the abalone dataset, predicting sex (M or F) from physical measurements. Infants are dropped — they form a third category and logistic regression is binary. That leaves ~2,835 samples: 54% male, 46% female.

→ open the interactive demo

the model

The starting point is the same as linear regression: compute a weighted sum of the input features.

This is the linear part — the same bias-augmented matrix-vector product, where \(\tilde{X} \in \mathbb{R}^{m \times (n+1)}\) has a leading ones column and \(\tilde{w} \in \mathbb{R}^{n+1}\) holds the bias as its first element. The result \(z \in \mathbb{R}^m\) is one score per sample.

The problem: \(z\) is unbounded — it can be any real number. A probability must live in \([0, 1]\). We need a function that squashes \(\mathbb{R}\) into \((0, 1)\) without losing the ordering (a higher score should still mean a higher probability). The sigmoid function does exactly this:

It has an S-shaped curve: as \(z \to +\infty\), \(\sigma(z) \to 1\); as \(z \to -\infty\), \(\sigma(z) \to 0\); and \(\sigma(0) = 0.5\) exactly. It is smooth, monotonic, and its derivative has a clean form — \(\sigma(z)(1 - \sigma(z))\) — which will matter when we compute the gradient.

The full model is then:

where \(\hat{y}_i \in (0, 1)\) is the estimated probability that sample \(i\) is male.

Decision boundary. To produce a hard class label, we threshold at 0.5:

Since \(\sigma(z) = 0.5\) when \(z = 0\), the decision boundary is the set of points where \(\tilde{X}\tilde{w} = 0\) — a hyperplane in feature space. Every point on one side gets classified as male, every point on the other as female. This is what makes logistic regression a linear classifier: the boundary it can draw is always a straight line (or flat hyperplane in higher dimensions). It cannot draw a curve.

The threshold of 0.5 is the natural default — predict male when male is more probable than not. A different threshold would make sense if false positives and false negatives had asymmetric costs (medical diagnosis, fraud detection), but here we have no reason to prefer one error over the other.

Why “regression”? Despite being a classifier, the model is linear on the log-odds scale:

We are regressing on the log-odds (the logit). The sigmoid is the inverse logit — it maps the linear output back to a probability.

the loss function

With the model defined, we need a way to measure how wrong it is. The natural candidate is the same mean squared error we used for linear regression — but MSE is a poor fit for classification.

The problem is the sigmoid. When the model makes a confidently wrong prediction — say it outputs \(\hat{y} = 0.001\) for a true male — the sigmoid is deep in its flat region and its gradient is nearly zero. MSE combined with the sigmoid produces a loss surface with near-zero gradients for the worst predictions: the errors we most want to correct get the weakest signal. There is a second problem: mean squared error wrapped around a sigmoid is non-convex in the weights, so gradient descent can settle into a local minimum that isn’t the best fit. Binary cross-entropy paired with the sigmoid is convex — a single global minimum — which is why the loss curves descend cleanly to one place no matter where the weights start.

Instead we use binary cross-entropy (also called log loss):

The formula has two terms that never fire at the same time:

• When \(y_i = 1\) (male): loss is \(-\log \hat{y}_i\). This is large when \(\hat{y}_i\) is close to 0 and approaches zero as \(\hat{y}_i \to 1\).
• When \(y_i = 0\) (female): loss is \(-\log(1 - \hat{y}_i)\). This is large when \(\hat{y}_i\) is close to 1 and approaches zero as \(\hat{y}_i \to 0\).

Why “cross-entropy”? The name comes from information theory. The cross-entropy between a true distribution \(p\) and a predicted distribution \(q\) is \(H(p, q) = -\sum p \log q\) — the average number of bits needed to encode outcomes drawn from \(p\) using a code optimized for \(q\). For a single sample the true distribution is one-hot: all the probability mass sits on the actual class. The sum over the two classes collapses to a single term — \(-\log \hat{y}_i\) when the label is male, \(-\log(1 - \hat{y}_i)\) when female — which is exactly the per-sample loss above. Minimizing binary cross-entropy means minimizing the bits wasted by predicting \(\hat{y}\) when the truth is \(y\); it reaches zero only when the predicted distribution matches the labels exactly. Equivalently, it minimizes the KL divergence from the true distribution to the predicted one.

The key property is that the penalty grows without bound as the model becomes more confidently wrong. If the model assigns \(\hat{y} = 0.001\) to a true male, the loss is \(-\log(0.001) \approx 6.9\). If it assigns \(\hat{y} = 0.5\), the loss is \(-\log(0.5) \approx 0.69\) — the starting point you see in the loss evolution chart.

In practice, \(\hat{y}\) is clamped away from 0 and 1 by a small epsilon (here \(10^{-15}\)) to avoid \(\log(0) = -\infty\).

Where it comes from. Binary cross-entropy is not an arbitrary choice — it falls out of maximum likelihood. The model treats each label as a Bernoulli draw with probability \(\hat{y}_i\):

The exponents act as a switch: the expression is \(\hat{y}_i\) when \(y_i = 1\) and \(1 - \hat{y}_i\) when \(y_i = 0\). Assuming the samples are independent, the likelihood of the whole dataset is the product \(\prod_{i=1}^{m} P(y_i \mid x_i)\). Taking the log turns the product into a sum, then negating and averaging gives:

which is exactly the loss above. Minimizing it is the same as finding the weights that make the observed labels most probable — maximum likelihood estimation.

gradient descent

To minimize the loss we need its gradient with respect to the weights — the direction of steepest increase, which we step against. Starting from the chain rule:

The sigmoid derivative is \(\frac{\partial \sigma}{\partial z} = \sigma(z)(1 - \sigma(z)) = \hat{y}_i(1 - \hat{y}_i)\), so:

Substituting and simplifying:

In matrix form:

This is identical in form to the linear regression gradient — residuals projected back through the feature matrix. The sigmoid derivative \(\hat{y}(1 - \hat{y})\) cancels exactly with the denominator from the log inthe loss. This is not a coincidence: binary cross-entropy was designed to pair with the sigmoid precisely because of this cancellation. It gives clean gradients even when the model is confidently wrong.

The update rule is the same as before:

One epoch is one full pass through the training data, computing this gradientand updating the weights. We repeat for thousands of epochs until the loss stops decreasing.

normalizing features

Same approach as in linear regression — means and standard deviations are computed from the training set only and reapplied without recomputing during predict(). Computing them from the full dataset before splitting leaks validation information into training.

metrics

Logistic regression outputs a probability, not a ring count — RMSE and R² measure distance from a continuous target and are meaningless here.

Binary cross-entropy is the training loss, covered in the loss function section above.

Accuracy is the evaluation metric — the fraction of samples classified correctly after thresholding at 0.5:

Accuracy has a known weakness on imbalanced datasets: a model that always predicts the majority class achieves high accuracy without learning anything. The abalone sex split is 54% male / 46% female — close enough to balanced that accuracy is a fair metric here. A model stuck at 54% is not learning; it is predicting the prior.

k-fold cross-validation

Same mechanism as in linear regression, with \(k = 5\). The only change is the metric: instead of minimizing RMSE, we maximize accuracy. The cross-validation function is generic over the scoring metric via a closure — no structural change was needed to support classification.

regularization

Same L1 and L2 penalties as in linear regression, with one difference: the regularization term is scaled by \(\frac{\lambda}{m}\) rather than bare \(\lambda\):

The gradient already divides by \(m\), so the penalty must be on the same scale. Without this, the effective regularization strength would grow with dataset size.

grid search

Same procedure as in linear regression, searching 30 log-spaced \(\lambda\) values from \(10^{-6}\) to \(10^{1}\) via 5-fold cross-validation. The only difference: we select the \(\lambda\) with the highest average validation accuracy rather than the lowest RMSE — the optimization direction flips for classification.

a note on running time. with \(\alpha = 0.01\) and 5,000 epochs, a single training run converges in a few seconds in the browser. grid search trains 5 models per \(\lambda\) across 30 values — use the cv epochs slider (default: 5,000) to control the cost. fewer iterations are enough to rank \(\lambda\) values reliably without full convergence.

interpreting this run

Learning rate α = 0.01 · 5,000 epochs · L2 regularization · best λ = 0.356 found by grid search

Loss evolution

The loss starts at 0.693 — exactly \(\ln 2\), the binary cross-entropy of a model that assigns 0.5 to every sample regardless of its features. This is the random-guess baseline for a balanced binary problem, and it is where logistic regression begins: all weights at zero, all predictions at 0.5.

Both curves drop steeply in the first few hundred epochs, then flatten. By epoch 5,000 training loss sits around 0.680 and validation loss around 0.675 — a reduction of only ~0.013 from the baseline. The model is learning something, but not much. The validation loss ending slightly below training loss is not a sign of exceptional generalization; it reflects the 80/20 split landing slightly easier samples in the validation set.

Accuracy evolution

Accuracy starts around 53% — the majority-class prior — and climbs slowly to ~57% (train) and ~58% (val) by epoch 5,000. The curves track each other closely throughout: there is no overfitting gap. When train and val accuracy are nearly identical, it means the model has not memorized the training data — it simply has not found much signal to memorize.

Both curves are still slowly rising at epoch 5,000, so the model has not fully converged. But the trajectory makes clear that more training would bring marginal gains — the ceiling is in the data, not the iteration count.

Predicted probabilities

Each dot is one abalone. The x-axis is P(male); the two rows separate actual males (blue, top) from actual females (red, bottom). The dashed line at 0.5 is the decision boundary.

The separation is real but weak. The male distribution is centered around 0.55 and the female distribution around 0.45 — correctly shifted to opposite sides of the threshold, but with massive overlap. Most predictions fall in the 0.4–0.6 band; the model is rarely confident. This is what a ~58% accuracy ceiling looks like in probability space: not a failure to learn, but an honest reflection of how much the two classes overlap in the feature space.

Feature weights

Each bar shows the weight assigned to that feature after training. Positive weights increase P(male); negative weights decrease it.

shucked_weight is the dominant positive predictor (~+0.38), with the other size and weight features carrying smaller negative weights. These signs are correlational, not causal: the physical measurements are highly collinear, so individual weights can shift or flip with small changes in the data and shouldn’t be read as biological mechanisms. The pattern is consistent with sex being weakly and diffusely encoded across overall body size — no single measurement isolates it. rings is near zero — age contributes almost nothing to sex prediction once size is accounted for.

Grid search — λ vs validation accuracy

The curve is completely flat: every λ from \(10^{-6}\) to \(10^{1}\) gives the same ~56% validation accuracy. The best λ = 0.356 is selected by the grid search but the margin over any other value is negligible.

A flat grid search is an honest result: regularization only helps when the model is overfitting — fitting noise in the training data that does not generalize. Here the model is underfitting. It has not found enough signal to overfit in the first place, so there is nothing for regularization to correct. Any λ gives the same result.

Weight stability across folds

Each row shows the distribution of a feature’s weight across the 5 CV folds.

These weights come from the 5 cross-validation refits, so their magnitudes differ slightly from the single-split run above (e.g. shucked_weight ~+0.43 here vs ~+0.38 there). Most features are remarkably stable: shucked_weight clusters tightly around +0.43, diameter and length around −0.2, whole_weight around +0.13. height shows the most spread — its whiskers extend from around −0.25 to near zero, making it the least reliable signal. This is not multicollinearity in the same dramatic sense as the linear regression weights — no feature is being pushed to opposite extremes across folds. The model is consistent; it is just consistently uncertain about the target.

where logistic regression falls short

58% accuracy on a near-balanced binary problem means the model is correctly classifying only 4 out of 100 more samples than the majority-class baseline — always predicting male — would. Three factors explain this:

The boundary is linear. Logistic regression draws a single flat hyperplane through the feature space. If the true relationship between physical measurements and sex is nonlinear — and biology suggests it is — no choice of weights can capture it.

The features overlap. Males and females share nearly identical distributions across all eight measurements. shucked_weight is the strongest signal, but even it shows heavy overlap. The classes are not linearly separable, and may not be separable at all with these features.

Sex is hard to predict from morphology alone. The abalone dataset was collected for age estimation, not sex classification. The features it contains — size, weight, rings — are proxies for age, not sex. A dedicated study would collect gonad measurements, spawning observations, or genetic markers.

This is not a failure of the implementation. It is an honest result: the model correctly tells us that sex cannot be reliably inferred from these physical measurements with a linear classifier. That is useful to know.

references

• James, G., Witten, D., Hastie, T., Tibshirani, R., and Taylor, J. An Introduction to Statistical Learning with Applications in Python. Springer, 2023. §4.3 (logistic regression), §6.2 and §6.4 (regularization).

• Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistical Learning, 2nd ed. Springer, 2009. §4.4 (logistic regression), §3.4 (regularization).

• Géron, A. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 3rd ed. O’Reilly, 2022. Chapter 4 (training models).

• Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016. Chapter 3 (information theory and cross-entropy).

• Logistic Regression — GeeksForGeeks

• Binary Cross Entropy — Towards Data Science

→ open the interactive demo

source code: github.com/jjginga/ferrolearn

This post covers the implementation I wrote in Rust (compiled to WASM) for the ferrolearn series, applied to the abalone dataset — predicting the number of rings (a proxy for age) from physical measurements.

→ open the interactive demo

the model

Linear regression assumes the target is a linear combination of the input features plus a bias term:

To keep the math clean, we absorb the bias into the weight vector by prepending a column of ones to the feature matrix — a technique called bias augmentation. Without it, we have two separate objects to optimize: a weight matrix \(W \in \mathbb{R}^{m \times n}\) and a bias vector \(b \in \mathbb{R}^m\). With it, we stack them into one:

\(\tilde{X}\) is a matrix with \(m\) rows (one per sample) and \(n+1\) columns (one per feature, plus the leading ones column). \(\tilde{w}\) is a vector of \(n+1\) weights, where \(w_0\) is the bias and \(w_1, \ldots, w_n\) are the feature weights. The prediction for all samples at once is then the matrix-vector product \(\hat{y} = \tilde{X}\tilde{w} \in \mathbb{R}^m\) — one number per sample. This gives us the augmented form:

where \(\tilde{X} \in \mathbb{R}^{m \times (n+1)}\) has a leading ones column and \(\tilde{w} \in \mathbb{R}^{n+1}\) holds the bias as its first element. There is now only one vector to optimize — no separate bias term to track.

We measure fit with the mean squared error, carrying a factor of ½ that cancels the 2 from differentiating the square and keeps the gradient clean:

A geometric view. Minimizing squared error has a clean geometric meaning. Picture the target vector \(y\) and every prediction the model could make — all of \(\tilde{X}\tilde{w}\) as \(\tilde{w}\) ranges over weight space. Those reachable predictions form a flat subspace: the column space of \(\tilde{X}\). Least squares finds the point in that subspace closest to \(y\) — the orthogonal projection of \(y\) onto it. The residual \(y - \hat{y}\) is what’s left over, sticking out perpendicular to the subspace: it is everything about \(y\) that no linear combination of the features can reach. Gradient descent walks toward that projection one step at a time; the normal equation jumps straight to it.

gradient descent

To minimize \(\mathcal{L}\) we need to know which direction to move the weights to make the error smaller. The gradient tells us exactly that — it is a vector that points in the direction of steepest increase of the loss, so we step in the opposite direction. Concretely, for each weight \(\tilde{w}_j\), the gradient tells us: “if you increase this weight slightly, does the error go up or down, and by how much?”

The gradient of the loss with respect to the weights is:

We divide by \(m\) (the number of samples) so the gradient magnitude does not grow with dataset size — without this, a dataset twice as large would produce steps twice as large, requiring a different learning rate for every dataset.

The update rule is applied once per epoch — one full pass through the training data:

An epoch is one complete cycle where every training sample has contributed to a gradient update. We repeat this process for hundreds or thousands of epochs, nudging the weights a little each time until the loss stops improving.

\(\alpha\) is the learning rate — a small positive number (typically between 0.0001 and 0.1) that controls how large each step is. Too large and the weights overshoot the minimum, causing the loss to oscillate or diverge. Too small and convergence is needlessly slow. It is a hyperparameter: we set it before training, not learned from data.

Why this works at all. Mean squared error is a convex function of the weights — its loss surface is a single smooth bowl with one lowest point and no others. That is what makes gradient descent safe here: wherever the weights start, following the slope downhill leads to the same global minimum, with no local minima to get trapped in and no dependence on initialization. It is why we can set every weight to zero at the start and still converge. Many models later in this series — neural networks especially — give this up: their loss surfaces are riddled with local minima, and where you start genuinely matters.

Note that we never compute \((\tilde{X}^\top \tilde{X})^{-1} \tilde{X}^\top y\) — the closed-form normal equation. It gives the exact optimum in one step but requires inverting an \((n+1) \times (n+1)\) matrix, which is \(O(n^3)\) and numerically unstable for correlated features. Gradient descent scales to large \(n\) and adapts naturally to regularization.

Two features are correlated when they carry redundant information — knowing one tells you a lot about the other. In the abalone dataset, length, diameter, and height are all measurements of physical size and move together: a longer shell is almost always also wider and taller. Similarly, whole_weight, shucked_weight, viscera_weight, and shell_weight are all mass measurements of the same animal split different ways — they are nearly interchangeable.

When features are correlated, the normal equation \((\tilde{X}^\top \tilde{X})^{-1} \tilde{X}^\top y\) becomes unreliable because \(\tilde{X}^\top \tilde{X}\) is nearly singular — it has no unique inverse. Intuitively: if two columns of \(\tilde{X}\) are nearly identical, the matrix cannot tell them apart, and there are infinitely many weight combinations that produce the same predictions. The inversion amplifies tiny numerical errors into wildly different weight values. Gradient descent sidesteps this entirely — it never inverts anything, it just follows the slope.

normalizing features

Raw features in the abalone dataset span very different scales: lengths are in millimetres (0–1), weights are in grams (0–3), rings are integers (1–29). Without normalization, gradient descent would take enormous steps along the weight dimension and tiny steps along the length dimension — convergence would be slow or divergent.

We standardize each feature to zero mean and unit variance:

But there is a subtle trap here.

data leakage. \(\mu\) and \(\sigma\) must be computed from the training set only. If you compute them from the full dataset before splitting, information from the validation set bleeds into the training process — the model appears to generalize better than it does. In the Rust implementation, column_means and column_stds are called inside fit() on the training data only, stored on the model struct, and reapplied (without recomputing) during predict().

encoding sex

The abalone dataset includes a categorical feature: sex (M, F, I for infant). Linear models require numerical input, so we need to encode it.

A naive approach assigns integers — M=0, F=1, I=2 — but this implies a false ordering: the model would treat “infant” as twice “female”, which is meaningless for a nominal category.

Instead we use one-hot encoding with female as the reference (dropped) category:

This gives each category an independent weight. Female is the reference — its effect is captured by the intercept. Dropping one column avoids the dummy variable trap: if we kept all three, they would sum to 1 for every row, creating perfect multicollinearity with the bias column and making the system underdetermined.

metrics: R² and RMSE

The sum of squared residuals is the loss we optimize, but it is hard to interpret — SSR of 22800 in what units? Rings squared? We use two additional metrics for reporting:

Root mean squared error (RMSE) brings the error back to the original unit:

An RMSE of 2.3 rings means the model is off by about 2.3 rings on average — much easier to reason about.

R² (coefficient of determination) measures the fraction of variance in the target that the model explains:

\(R^2 = 1\) means perfect prediction. \(R^2 = 0\) means the model does no better than predicting the mean. Negative values mean it is actively worse than the mean — this happens in early training when weights are all zero.

k-fold cross-validation

A single train/test split can be lucky or unlucky depending on which samples end up where. K-fold cross-validation gives a more robust estimate:

1. Divide the data into \(k\) equal parts (folds)
2. For each fold: train on the remaining \(k-1\) folds, evaluate on the held-out fold
3. Average the \(k\) validation scores

We use \(k=5\), which is the standard choice — it balances variance (more folds = more stable estimate) against computational cost (\(k\) times more training runs). With 4177 samples each fold holds ~835 validation samples, which is large enough to be representative.

In the Rust implementation, cross-validation is generic over any SupervisedModel via a factory closure — the same function is to be reused across several models: linear regression, logistic regression, and future models without modification.

regularization

When features are correlated or the model has many parameters relative to samples, the weights can grow large to fit noise. Regularization adds a penalty to the loss that discourages this:

\(\lambda \geq 0\) is the regularization strength — a hyperparameter that controls how much the penalty influences the weights. \(\lambda = 0\) means no regularization; larger values impose a stronger penalty.

L2 (Ridge) penalizes the sum of squared weights:

L2 shrinks all weights towards zero smoothly. It handles correlated features by distributing weight among them rather than arbitrarily choosing one.

L1 (Lasso) penalizes the sum of absolute weights:

L1 produces sparse solutions — it drives irrelevant weights exactly to zero, effectively performing feature selection. The non-smooth absolute value means the gradient is constant regardless of weight magnitude, so small weights get pushed to zero rather than just shrunk.

Both regularizers skip the bias term (\(j=0\)) — regularizing the intercept would shift predictions towards zero, which is not what we want.

The tradeoff is the bias-variance tradeoff: increasing \(\lambda\) introduces bias (the model can no longer fit the data as closely) but reduces variance (the model is less sensitive to noise in the training set). The right \(\lambda\) is the one that minimizes validation error, found via grid search.

grid search

\(\lambda\) controls how strongly regularization penalizes large weights — but the right value depends on the data. Too small and it has no effect; too large and it shrinks all weights towards zero regardless of their predictive value. Grid search finds the best \(\lambda\) automatically: we evaluate a range of values using 5-fold cross-validation and pick the one with the lowest average validation RMSE. The search space spans from \(10^{-6}\) to \(10^{1}\) in 30 log-spaced steps — logarithmic because the effect of regularization scales multiplicatively, not additively — plus \(\lambda = 0\) prepended as a no-regularization baseline, for 31 candidates in total.

a note on running time. with α = 0.0005 and 25,000 iterations, the final training run takes around 10–20 seconds in the browser. grid search is more expensive — it trains 5 models per λ value across 31 values, which would take several minutes at 25,000 iterations each. to keep it interactive, the demo uses a separate cv epochs slider (default: 5,000) for the cross-validation runs: fewer iterations are enough to compare λ values reliably, since CV only needs to rank them, not fully converge. if you want faster results, reduce cv epochs or the number of grid steps; if you want a more thorough search, increase them.

interpreting this run

Learning rate α = 0.0005 · 25,000 iterations · L2 regularization · best λ found by grid search

α = 0.0005 is deliberately small so gradient steps are cautious and stable; 25,000 iterations gives the model enough passes through the data to fully converge even at that slow pace.

R² evolution

This chart shows how well the model fits the data as training progresses — one point per epoch for both the training set (solid blue) and the validation set (dashed orange).

Both lines start pinned at the bottom of the chart, around −0.5 — but that is a display floor, not the real value. The y-axis is clamped at −0.5, because the first epochs sit so far below the rest of the curve that showing them would flatten everything informative. With all weights at zero, every prediction is zero, so the true starting R² is much lower — roughly −9. A negative R² means the model is worse than simply predicting the average, which makes sense at epoch 0: predicting zero for a target that averages ~10 rings is far worse than predicting the mean. A negative R² means the model is actively worse than just predicting the average, which makes sense at epoch 0: all weights are zero, so every prediction is zero — far below the ~10-ring average. As gradient descent adjusts the weights, R² climbs rapidly in the first ~3,000–5,000 iterations, then flattens. By 25,000 iterations the curves are completely flat — the model has converged and more iterations would change nothing.

The final values: training R² ≈ 0.55, validation R² ≈ 0.43. The gap between them (~0.12) is mild but real — the the model fits the training data slightly better than it fits data it has never seen — this is expected, since the model was optimized on the training set and had no information about the validation set during training. With only 9 features and 4,177 samples this is not severe overfitting; it reflects the inherent noise in ring counting more than model complexity.

RMSE evolution

RMSE measures the average prediction error in the same unit as the target — rings. It starts around 11 rings at epoch 0: with all weights at zero, every prediction is zero, so each one misses by roughly a full target value — the error is enormous. (If the model instead predicted the mean, RMSE would be the ring standard deviation, about 3.3.). It drops steeply in the first 5,000 iterations as gradient descent finds the main signal in the data, then flattens. By 25,000 iterations both curves are flat — the model has converged.

Final values: train RMSE ≈ 2.2 rings, val RMSE ≈ 2.1 rings. Notably, validation RMSE ends up slightly below training RMSE — the opposite of what you might expect. The cause is the split: it is taken contiguously (first 80% / last 20%, no shuffle), and the last fifth of the file happens to have a narrower spread of ring counts — standard deviation ≈ 2.7 versus ≈ 3.3 for the training portion. A lower-variance target is easier to predict in absolute terms, so RMSE comes out lower even though validation R² is worse (0.43 vs 0.55). It is not a sign the model generalises perfectly — shuffling before splitting would make the two sets match.

Grid search — λ vs validation RMSE

Before the final model was trained, we searched for the best regularization strength λ using 5-fold cross-validation. For each candidate λ, five models were trained on different subsets of the data and evaluated on the held-out fold. The chart shows the average validation RMSE (error in rings) across the five folds for each λ.

The curve is flat from λ = 10⁻⁶ to λ ≈ 10⁻², with RMSE staying around 2.50 rings. Above 10⁻², RMSE rises steeply — at λ = 10¹ it reaches ~3.0 rings. The best λ is 0 (no regularization). This is an honest result: with 9 features and ~4,000 samples the model has far more data than parameters and is not overfit, so there is nothing for regularization to fix. At high λ the penalty dominates and drives all weights towards zero regardless of their predictive value — predictions collapse towards the mean, RMSE rises.

Note that grid search used 5,000 iterations per fold rather than 25,000 — enough for the models to separate meaningfully across λ values, which is all CV needs to do.

Predicted vs actual

Each dot is one abalone. A perfect model would place every dot exactly on the diagonal dashed line.

The model does well in the 5–15 ring range, where most of the dataset lives — dots cluster tightly around the diagonal. Above 15 rings the model systematically underpredicts: the diagonal keeps rising but predictions plateau around 13–15. This is not a bug; it is a fundamental limitation of linear models. Ring count grows nonlinearly with age and physical size — older abalones add rings at a slower rate relative to their size — and no linear combination of the features can capture that curve.

The vertical striping is also characteristic: ring counts are integers (3, 4, 5, …) but the model outputs continuous predictions, which cluster around the most common values in the training set.

Feature weights

Each bar shows how much a one-unit increase in that feature (after standardization) changes the predicted ring count, holding all other features constant.

The most striking result: shell_weight has a large positive weight (~+2.3) and shucked_weight has a large negative weight (~−2.2). Both measure mass — shell_weight is the dried shell, shucked_weight is the meat. They are highly correlated with each other and with ring count. When two features carry nearly the same information, the model cannot distinguish their individual contributions — it compensates by assigning them large weights of opposite sign that partially cancel. This is called multicollinearity: the individual weights are not interpretable in isolation, but their combined effect is stable. L2 regularization would shrink both towards zero and spread the weight more evenly — but as the grid search shows, it does not improve generalization on this dataset.

sex_M is near zero — being male versus female has almost no predictive power for ring count once physical measurements are included. sex_I (infant) is negative — infants have fewer rings, as expected, since ring count is a proxy for age.

Weight stability across folds

Each row shows how much a feature’s weight varies across the 5 CV folds. A tight cluster means the model assigns that feature a consistent weight regardless of which samples it trained on — a reliable signal. A wide spread means the weight is sensitive to the specific training data — a sign of instability, often caused by multicollinearity.

Most features are stable: shell_weight sits around +0.85, height around +0.5, diameter around +0.4, sex_I around −0.4, and shucked_weight around −0.5; the rest hug zero. The widest spreads belong to height, shell_weight, and shucked_weight — each varying by about ±0.1 across folds (shucked_weight, for instance, ranges from roughly −0.4 to −0.6, never near zero). That shell_weight and shucked_weight — the two opposite-sign mass measurements — are among the least stable is the multicollinearity effect in action: they carry overlapping information, so different training splits divide the credit between them differently.

Note that these weights come from models trained for 5,000 iterations (the CV epochs setting), not 25,000 — they are partially converged. The multicollinearity signature is already visible but less extreme than in the fully trained model above.

where linear regression breaks

Validation R² ≈ 0.43 means the model explains about 43% of the variance in ring count. The remaining 57% comes from three sources:

Nonlinearity. Physical measurements scale nonlinearly with age. A linear model draws a flat hyperplane through the data — it cannot bend to follow a curve.

Multicollinearity. The four weight measurements (whole, shucked, viscera, shell) carry heavily overlapping information. The model artificially splits them into positive and negative contributions rather than extracting a clean signal.

Biological noise. Ring counting is done by hand under a microscope and has known inter-rater variability. Some of the variance in the target is irreducible — no model can predict it because it is measurement error, not signal.

This is exactly the motivation for the next models in the series. Tree-based methods can capture nonlinearity without any manual feature engineering. Ensemble methods reduce variance by combining many models. But before going there, it is worth sitting with this result — a simple model honestly evaluated tells you more about your data than a complex model you do not understand.

references

• James, G., Witten, D., Hastie, T., Tibshirani, R., and Taylor, J. An Introduction to Statistical Learning with Applications in Python. Springer, 2023. §3.1 (linear regression), §6.2 and §6.4 (regularization).

• Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistical Learning, 2nd ed. Springer, 2009. §3.1 and §3.2 (linear regression and least squares), §3.4 (regularization).

• Géron, A. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 3rd ed. O’Reilly, 2022. Chapter 4 (training models).

• Linear Regression in Machine Learning — GeeksForGeeks

• The Bias-Variance Tradeoff — Towards Data Science

→ open the interactive demo

source code: github.com/jjginga/ferrolearn

that’s where this series comes from. i decided to reimplement everything from the course in rust, compile it to wasm, and build interactive browser demos so you can actually watch the algorithms do their thing. the project is called ferrolearn. the code lives at github.com/jjginga/ferrolearn.

the plan is to cover these algorithms, roughly in course order:

each post pairs with a live demo you can interact with directly in the browser. the rust code compiles to wasm via wasm-pack and the demos are built with vanilla js and d3. no framework, no bundler — just a script tag and a canvas.

i’m also learning rust as i go, so expect the posts to spend time on the language itself alongside the maths.

source code: github.com/jjginga/ferrolearn

before training a single model, you have to look at the data. not glance at it — actually look. every assumption you make about what algorithm to use, what features matter, what preprocessing is needed, lives or dies on what the data actually looks like. this first post is about that: exploratory data analysis on the abalone dataset, with all the computation running in rust compiled to webassembly.

the dataset

abalone are marine molluscs. they live on rocky coasts, graze on algae, and happen to have been the subject of a 1994 population biology study out of tasmania by nash, sellers, talbot, cawthorn and ford. their data has since become one of the classic regression benchmarks on the uci machine learning repository.

the problem is this: how old is an abalone? you can find out exactly by cutting the shell through the cone, staining a cross-section, and counting the growth rings under a microscope — the same way you’d age a tree. rings + 1.5 gives the age in years. but that process is slow, destructive, and requires lab equipment. the question is whether you can get a good estimate just from physical measurements taken in the field.

the dataset has 4177 samples and 8 features: sex (male, female, or infant), longest shell length, diameter perpendicular to length, height with meat in shell, whole weight, shucked weight (meat only), viscera weight (gut after bleeding), and shell weight after drying. the target is the ring count.

why eda first

a few things you can only learn by looking. is the target normally distributed or heavily skewed? are there outliers that will distort training? do features have enough variance to carry signal, or are some nearly constant? are any features so correlated that they’ll cause problems for certain models?

these aren’t rhetorical questions. the answers change what you build.

interactive demo

the eda runs entirely in rust compiled to wasm — the rust code parses the csv, computes summary statistics, and calculates the correlation matrix client-side. d3 handles the visualisation.

→ open the interactive demo

what the data is telling us

the target is well-behaved.

the rings histogram is roughly bell-shaped, centred around 9–10 rings (10.5–11.5 years), with a modest right tail out to 29. this is good news for linear regression — we’re not fighting a heavily skewed or bimodal target. it won’t be trivial either: there’s genuine spread, and the upper tail thins out fast, which means the model will see very few examples of old abalone.

the physical measurements are spread but not exotic.

most features follow a roughly bell-shaped distribution centred on mid-range values. height stands out: there are a handful of samples with values near zero or suspiciously large. the uci page notes that missing-value examples were removed, but doesn’t mention measurement errors in height specifically. for now i’ve kept them — the rust parser validates field count and type but doesn’t filter by range — but it’s worth knowing they’re there.

the physical measurements are highly collinear.

length, diameter, whole weight, shucked weight, viscera weight, and shell weight are all correlated with each other at 0.9 or higher. this is expected — bigger shells are heavier in every dimension. but it has a direct consequence for linear regression: when predictors move together this tightly, coefficient estimates become unstable. a small change in the data can produce wildly different weights. this is exactly the problem regularisation (l1 and l2) exists to solve, and it’s why we’ll spend time on it in the next post before fitting anything. shell weight ends up being the strongest individual predictor of rings — biologically sensible, since shell mass accumulates continuously over an abalone’s life.

the infant category is an age effect, not a sex effect.

infants have systematically fewer rings than males and females. this makes biological sense — they haven’t grown as long. but it matters for modelling: sex isn’t a clean categorical feature on equal footing with the others. “I” encodes age information directly. if you one-hot encode sex naively and throw it into a regression alongside the physical measurements, you’re doubling up on age signal in a way that’s hard to reason about.

the scatter coloured by sex makes this concrete — infants cluster at the lower end of both size and ring count, while males and females overlap considerably across the full range.

what this sets up

the collinearity issue is the key takeaway going into linear regression. when predictors move together, the model can’t isolate the contribution of each one — there are infinitely many weight combinations that produce the same predictions on the training set, and tiny perturbations send the weights in different directions. l2 regularisation (ridge) addresses this by penalising large weights, shrinking correlated coefficients toward each other rather than letting them cancel out. we’ll implement gradient descent with both l1 and l2 penalties, and use cross-validation to pick the regularisation strength.

next post: linear regression, from scratch, with a gradient descent animation.

source code: github.com/jjginga/ferrolearn

We’re constantly coining new terms for the same underlying issue: “quiet quitting,” “boreout,” and a host of others. These aren’t just fads; they’re symptoms of a deeper problem: anomie.

It’s tempting to blame external factors or specific environments, and they certainly play a role. However, the root cause is often the profound social shifts we’re navigating. For decades, work was a core pillar of identity. You were a doctor, an engineer, a factory worker. Your career path was often linear, offering a clear connection between education, profession, and self-definition.

Today, that stability is gone. Social structures are fluid, change is constant, and professional identities are no longer fixed. The result? A significant loss of clarity and stability. This leads directly to that pervasive feeling of being adrift—what sociology calls anomie: a lack of clear direction, purpose, or defined rules.

Consider the recent discussions about Gen Z’s approach to leadership, sparked by a recent newspaper article. This debate, while important, risks staying superficial. The real issue isn’t generational; it’s structural. The “liquid identities” of today’s workforce clash with outdated organizational frameworks. We’ve moved past fixed careers and clear, vertical hierarchical structures, yet many companies operate as if nothing has changed.

This is where the recruitment process itself often falls short. Many companies still approach hiring by seeking “parts for a machine that no longer exists,” rather than focusing on the agility and transversal skills that are truly needed. They persist in crystallized job profiles, disconnected from the current and future reality of work. This often reflects a lack of strategic thinking about competencies, leading to CV screening processes that are reductive and blind to the very transferable skills that could bring immense value. Without adequate training or strategic vision, those screening CVs often lack the capacity to identify these “liquid identity” talents — precisely the agility and innovation capacity that organizations so desperately need in this new era.

The arrival of Artificial Intelligence is absolutely crucial here. AI can be a brutal accelerator for the need for reformulation. However, in companies that haven’t reflected on their own architecture, the advent of AI risks widening the gap and increasing anomie, instead of becoming a “companion.” Its adoption will demand a profound effort of reformulation and rethinking from companies — not just at the process level, but also in the very definition of roles, responsibilities, and collaboration models. For many organizations, especially SMEs and even larger ones, this is a major challenge, as many still lack a structured reflection on their organizational architecture or strategic competency development. In practice, they are light-years away from the maturity needed to effectively integrate these new technologies as true “companions” in their daily business.

The challenge is clear: most organizational structures haven’t adapted to this new social reality. The old paradigm—where individuals were expected to conform to the organization—is obsolete. The path forward demands an inversion: organizations must adapt to individuals, embracing their evolving identities and aspirations.

True readiness for the future means building systems where people and organizations collaborate, challenging traditional leadership hierarchies, and collectively pursuing objectives. Only then can we bridge the gap created by this modern anomie.

Articles that Prompted this Reflection (in Portuguese)

[1] - Expresso - Z Generation does not want to lead [2] - CNN - There is a new risk associated to work: the boreout

Introduction

The rise of advanced AI, particularly models like OpenAI?s O3, is reshaping the software development landscape. OpenAI’s O3 model, announced on December 20, 2024, is designed to enhance reasoning capabilities, enabling it to tackle complex tasks in coding, mathematics, and science. Reportedly, it can perform 72% of tasks a software engineer faces daily, including writing efficient code snippets, debugging, and optimizing algorithms. Early case studies have demonstrated that developers using O3 have seen productivity boosts of up to 40%, especially in repetitive or time-intensive tasks[1].

However, such groundbreaking advancements come at a cost. Tools like OpenAI?s O3, while incredibly powerful, are currently resource-intensive and expensive to operate - costing as much as $1,000 per task[2]. This limits their accessibility to large organizations or well-funded teams for now.

Yet, as with most technologies, history has shown that costs tend to decrease over time as adoption scales and efficiencies improve. A prime example is cloud storage: over the past decade, costs have dropped by more than 80%, making it widely accessible[3]. It?s reasonable to expect AI tools like O3 to follow a similar trajectory, eventually democratizing access and enabling businesses of all sizes to leverage their potential.

Adapting to AI

AI is becoming a powerful ally for developers. When used as a tool, models like O3 have the potential to increase the productivity of a single developer to levels that could surpass that of an entire team. These systems can generate, test, and optimize code at a pace humans cannot match. However, they are far from self-sufficient.

Developers remain critical for understanding processes, optimization, and requirements?the strategic elements that ensure solutions meet business and user needs. AI might handle the heavy lifting of generating code, but developers act as directors, testers, and validators, ensuring the AI?s output aligns with the goals of the project.

Transforming Roles

This shift doesn?t eliminate the role of developers?it reconfigures it. The future developer will not just write code but will leverage AI to accelerate workflows, reduce repetitive tasks, and focus on higher-order problem-solving. The goal is not merely to perform tasks that AI can handle but to achieve outcomes that would be impossible without it.

Much like the role of online tools in education, AI offers immense opportunities to enhance productivity but carries the risk of becoming a crutch. Developers must avoid letting AI do all the work and instead focus on becoming individuals capable of using AI to push the boundaries of innovation. The question is not what AI can do for you, but how you can leverage it to achieve what was once impossible.

The Path Forward

To thrive, developers will need to:

• Understand Requirements: Translating business needs into actionable tasks for AI.
• Validate and Test: Ensuring AI-generated solutions are secure, efficient, and practical.
• Optimize Solutions: Fine-tuning AI outputs for performance, scalability, and alignment with the bigger picture.

The key is to use AI as an amplifier of human potential. Those who use it effectively will multiply their impact and unlock new possibilities, while those who rely on it passively may find themselves left behind.

A Symbiotic Relationship

AI is a tool, not a replacement. It is there to complement developers’ skills and enable them to reach new heights, not to render them obsolete. This symbiotic relationship requires both adaptability and critical thinking. As developers, we must learn to embrace AI as a partner in creativity and problem-solving, using it to complete our abilities and achieve what was previously impossible.

As the field evolves, one thing is clear: developers who embrace AI will thrive. Those who resist might find themselves outpaced?not because AI eliminates their role, but because it redefines it.

Sources

[1] - Ars Technica
[2] - TechCrunch
[3] - Wasabi Blog

Further Reading

• Wikipedia on OpenAI O3
• Why OpenAI?s O3 Won?t Replace You Yet

Image Source

Image by Freepik.

The Railway Station Puzzle aims to optimize the placement of railway stations to minimize the number of stations and the average travel cost for families. This problem is inspired by historical railway expansion, requiring efficient computational solutions. To understand the intricacies and rules of this puzzle, explore the problem statement on GitHub.

The problem as described in the README of the repository and in an attached PDF is a practical component of a course in my university, heard some collegues talking about it and decided to try.

Problem Definition

The task involves an NxM grid representing a map where each cell contains a number of families. The goal is to minimize the number of railway stations (A) and the average travel cost (B) to the nearest station. The travel cost is defined by distance with specified unit costs. The overall cost function is:

Graph Representation

The problem is represented as a grid where each cell is a zone containing a number of families. The nodes are the zones, and edges represent the possible placements of stations.

This table succinctly encapsulates the problem’s components, providing a clear framework for the “Land Permutation Problem.” The constraints ensure that the number of borders in a successor state must not exceed that of the current state, guiding the search towards the objective.

Approach and Algorithms

Informed Search Algorithms

Informed search algorithms use heuristics to guide the search process, making them more efficient than uninformed search methods. These algorithms prioritize exploring paths that are more likely to lead to the goal, thus reducing the search space and time.

What is a Heuristic?

A heuristic is a technique used to estimate the cost of reaching the goal from a given state. It provides an educated guess based on available information, helping to prioritize certain paths over others.

Why Use Heuristics Instead of Cost?

Heuristics allow algorithms to make decisions based on estimated future costs, leading to more efficient searches. By using heuristics, informed search algorithms can quickly discard less promising paths, focusing computational resources on more likely solutions.

A* Algorithm

A* combines the actual cost to reach a node and the heuristic estimated cost to the goal, ensuring that the path found is both optimal and efficient.

How A* Works:

1. Initialization: Starts from the initial node, using a priority queue to manage the frontier.
2. Cost Function: Uses the evaluation function \(( f(n) = g(n) + h(n) )\), where \(( g(n) )\) is the actual cost and \(( h(n) )\) is the heuristic estimate.
3. Node Expansion: Expands the node with the lowest \(( f(n) )\) value.
4. Path Finding: Continues until the goal node is reached, ensuring the path with the minimum cost is found.

Best-First Search

Best-First Search uses only the heuristic estimate to guide the search, always expanding the most promising node based on the heuristic value.

How Best-First Search Works:

1. Initialization: Begins at the start node, with a priority queue to handle the frontier.
2. Heuristic Evaluation: Uses a heuristic function \(( h(n) )\) to estimate the cost to the goal.
3. Node Expansion: Prioritizes nodes with the lowest heuristic value.
4. Efficiency: Can quickly find a solution, but may not always find the optimal path compared to A*.

Algorithm Selection

Informed search algorithms are chosen for their efficiency because they use heuristics to guide the search process. Unlike blind search methods, which explore all possible paths without direction, informed search algorithms focus on the most promising paths based on heuristic estimates. This reduces the search space and time required to find a solution. For the Railway Station Puzzle, using an informed search like Best-First Search ensures that both the number of stations and the average travel cost are minimized effectively, providing optimal results more efficiently.

Why Use Best-First Search for the Railway Station Puzzle

Best-First Search (BFS) is used in your Railway Station Puzzle implementation for several reasons:

1. Heuristic-Driven Efficiency

• Heuristic Focus: Best-First Search uses a heuristic to guide the search process, prioritizing nodes that are estimated to be closer to the goal. This can be particularly effective in problems where a good heuristic can significantly reduce the search space.
• Problem-Specific Heuristic: In this implementation, the heuristic is based on the distance to the nearest station and the number of families affected, which directly aligns with the goal of minimizing the total cost.

2. Memory Management

• Simplified Cost Handling: Best-First Search avoids the need to manage and update the actual cost (g(n)) for each node, focusing solely on the heuristic (h(n)). This reduces the complexity of state management and can lead to better performance in terms of memory usage.
• Priority Queue Efficiency: Using a priority queue based on heuristic values allows the algorithm to efficiently manage and retrieve the most promising nodes without the overhead of combining costs.

3. Implementation Suitability

• Straightforward Heuristic Application: The heuristic calculation in this implementation is straightforward and effectively guides the search towards solutions that minimize travel costs for families.
• Focused Search: By prioritizing nodes with the lowest heuristic values, Best-First Search can quickly hone in on promising areas of the search space, which is suitable for the structure of the Railway Station Puzzle.

Why Best-First Search Is Better for This Problem

1. Effective Heuristic Utilization

• Quality of Heuristic: The heuristic used in this implementation is effective in estimating the cost of reaching the goal. It accurately reflects the problem constraints by considering both the distance to the nearest station and the number of families affected, leading to efficient pathfinding.

2. Reduced Computational Overhead

• Simpler Calculations: Best-First Search requires fewer calculations per node compared to A*, as it only uses heuristic values for prioritization. This can result in faster execution, particularly for large search spaces.

3. Flexibility in State Expansion

• State Expansion Based on Heuristics: The algorithm expands nodes based on heuristic values, which can be particularly useful in problems where the heuristic provides a strong indication of the optimal path. This aligns well with the goal of minimizing station placements and travel costs.

Conclusion

Best-First Search is a suitable choice for the Railway Station Puzzle because it leverages an effective heuristic to guide the search process, reducing the search space and computational overhead. By focusing on heuristic values, the algorithm can efficiently navigate towards solutions that minimize the total cost, making it a practical approach for this optimization problem.

Heuristic Selection

The heuristic used in this problem balances the number of stations and the travel cost:

This ensures that the search process focuses on minimizing both the number of stations and the average travel cost, making it well-suited for the A* algorithm.

Implementation

To solve the Railway Station Puzzle, the problem was implemented in a structured manner using a Best-First Search algorithm. The solution includes:

• A RailwayStation class that defines the state space of the problem, including the map layout, station placements, sucessor state generation, heuristc calculation and cost calculations.
• A BestFirst class that implements the Best-First Search algorithm to find the optimal placement of stations.
• A DistanceMapViewer class to visualize the results using JavaFX.

Problem Modeling

The RailwayStation class represents the state space, defining the layout of the map, the placement of stations, and the cost calculations. The cost calculation is done using the formula provided in the problem statement, the heuristic is calculated using the formula provided above and a set is used to prevent states that are already in the queue of being added again (and having the cost of calculating the heuristic again). These are all fundamental for the algorithm.

Algorithms

In the implementation of the search algorithms we used the tools provided to us by java, we use a PriorityQueue to store the generated states after each interaction, and for this we also overriden the compareTo method in the state to use the heuristic. We also use a HashSet to keep track of the states already explored and since it has the states we overriden the equals and hashCode.

So on every iteration, we get the state with the best heuristic from the Priority Queue, we calculate its cost, check if it is a valid solution, if it is, we compare it with the best solution we have so far. Generate its children and add them to the priority queue. The algorithm goes on until all the state space is explored or until a minute or 100000 evaluations (of the cost), have been met.

Result Presentation

The DistanceMapViewer class visualizes the results using JavaFX. This class is responsible for presenting the map, station placements, and various statistics such as average cost, total cost, evaluations, and generations. The visualization provides a clear and interactive way to analyze the performance of the algorithm and the effectiveness of the station placements.

Color Coding of the Map:

• Green: Distance 0 (station location)
• Light Blue: Distance 1
• Yellow: Distance 2
• Orange: Distance 3
• Light Coral: Distance 4
• Red: Distance 5
• Light Gray: Distance 6

Results

The table below presents the outcomes of applying the Best-First Search algorithm to the Railway Station Puzzle across 20 different instances. Each entry includes the number of stations placed, the average travel cost, the total cost, the number of evaluations, the number of states generated, and the execution time.

Discussion

The results of the Best-First Search algorithm on the Railway Station Puzzle reveal several key points about its performance:

1. Evaluation Consistency: Across all instances, the algorithm consistently performed 100,000 evaluations, indicating that it fully utilized the allowed computation budget in most cases.
2. Execution Time: The execution times varied significantly, from 0 seconds to 60 seconds. The instances that reached the 60-second mark did not find an optimal solution within the time limit, suggesting increased complexity and larger state spaces in these cases.
3. Generation of States: The number of generated states varied widely, from around 100,000 to over 1,250,000. This highlights the varying complexity of different instances and the corresponding impact on search space exploration.
4. Solution Quality: The average travel costs ranged from 1.42 to 2.97, and the total costs ranged from 1197 to 7291. Instances with more stations generally had higher total costs but sometimes lower average costs, indicating a trade-off between the number of stations and travel efficiency.

The table provides a comprehensive overview of the algorithm’s performance, reflecting its strengths in efficiently exploring large search spaces and generating numerous potential solutions. However, the instances that hit the 60-second limit reveal areas where the algorithm struggles with time complexity, suggesting that further optimizations or alternative approaches may be needed for more complex scenarios.

Further Information

For readers interested in learning more about heuristic search algorithms and their applications, the following resources provide comprehensive explanations and examples:

• Geeks for Geeks - A* Algorithm: An in-depth article on the A* search algorithm, explaining its methodology, implementation techniques, and practical applications.
• Geeks for Geeks - Best-First Search: A detailed guide on Best-First Search, including its algorithmic approach, implementation, and use cases.
• Javatpoint - Heuristic Techniques: An informative piece on heuristic techniques, covering various heuristic search algorithms and their applications.
• SpringerLink - An Overview of Heuristics and Metaheuristics: A comprehensive overview of heuristics and metaheuristics, including traditional local search methods and advanced metaheuristic algorithms.
• Russell, S., & Norvig, P. (2001). Artificial Intelligence: A Modern Approach (3rd ed). AIMA: A seminal textbook in the field of artificial intelligence, providing a thorough grounding in search algorithms, heuristic methods, and a broad range of AI topics.

These resources offer valuable insights and deeper understanding for anyone looking to expand their knowledge of heuristic search algorithms and their implementation in solving complex problems like the Railway Station Puzzle.

In the intricate tapestry of computational challenges, the “Land Permutation Problem” stands out. It’s a deceivingly straightforward puzzle derived from the artificial intelligence course in computer science degree, involving a matrix of lands owned by various proprietors. The goal is to minimize the number of different owners’ borders. This territorial jigsaw requires both insightful analysis and strategic algorithmic application. To understand the intricacies and rules of this puzzle, explore the problem statement on GitHub.

The problem as described in the README of the repository is a practical component of my studies.

Problem Definition

The task involves a given matrix of NxM houses representing a map of lands of equal dimension with K owners. The goal is to reduce the number of borders between lands of different owners to a number equal to or less than W. A border exists between two houses with different owners. Owners can be represented by colors, letters, or numbers, with numbers being used in our program.

This table succinctly encapsulates the problem’s components, providing a clear framework for the “Land Permutation Problem.” The constraints ensure that the number of borders in a successor state must not exceed that of the current state, guiding the search towards the objective.

Navigating Solutions: BFS, DFS, and IDDFS Explained

This problem — minimizing the number of borders between lands with different owners — can be modeled as a graph traversal challenge. Each configuration of the land map, representing territories and their borders, can be considered a node (or vertex) in a graph. The initial state is the root node, and each possible action (e.g., swapping two adjacent territories to potentially reduce borders) leads to a successor state, which is a child node in the graph.

So, to solve this enigmatic puzzle we can use three classic algorithms, each with its own strengths and peculiarities. Breadth-First Search (BFS) is like casting a wide net, exploring all neighboring nodes at the current depth before diving deeper. It’s methodical, ensuring no stone is left unturned, but its memory consumption can quickly become a concern as the breadth of exploration expands.

In contrast, Depth-First Search (DFS) opts for a more tunnel-vision approach, diving deep into the problem space one path at a time. Its memory footprint is lighter, making it nimble and efficient in certain mazes of complexity. However, its laser focus can sometimes be a drawback, as it might miss broader solutions found at shallower depths.

Iterative Deepening Depth-First Search (IDDFS), then, marries the thoroughness of BFS with the memory efficiency of DFS. By incrementally deepening the search depth, IDDFS systematically covers the search space, ensuring completeness without the heavy memory burden associated with BFS.

Breadth-First Search (BFS)

Breadth-First Search (BFS) systematically covers a graph by visiting adjacent vertices in successive levels.

How BFS Works:

1. Initialization: BFS begins with the root node (initial state) and explores all its neighbors (successor states).
2. Queue Management: It employs a queue to keep track of the frontier—the set of all nodes visible but not yet explored.
3. Uniform Exploration: At each step, BFS dequeues the next node from the frontier, examines it for a solution, and enqueues all its unvisited neighbors.
4. Memory Consideration: The major downside of BFS in complex puzzles like land permutation is its memory usage, which grows exponentially with the number of levels explored.

Depth-First Search (DFS)

Depth-First Search delves into a graph, extending as far along each path as possible before backtracking to explore other branches.

How DFS Operates:

1. Initialization: Starting at the root, DFS pushes the initial state onto a stack.
2. Stack Utilization: It utilizes a stack to manage the nodes currently being explored.
3. Depth Exploration: DFS continuously explores down one branch until it hits a dead end, then backtracks to explore other branches.
4. Memory Efficiency: Its stack-based approach limits memory usage to the maximum depth of the search space, unlike BFS’s breadth-based memory expansion.
5. Solution Depth: While efficient, DFS is not guaranteed to find the shortest solution in terms of moves or transformations.

Iterative Deepening Depth-First Search (IDDFS)

Iterative Deepening Depth First Search (IDDFS) progressively deepens the reach of Depth-First Search, applying a breadth-like approach to depth-limited searches.

How IDDFS Functions:

1. Depth Iteration: Starting with a shallow depth limit, IDDFS performs a DFS within this limit. If no solution is found, the limit is increased, and the search is repeated.
2. Balance of Efficiency: This approach ensures that the memory advantages of DFS are maintained while achieving the breadth of coverage that BFS offers.
3. Optimality: Like BFS, IDDFS will find the optimal solution in terms of the shortest path to the goal.
4. Revisitation: Each iteration revisits nodes from previous depths, which is the trade-off for its balance between BFS and DFS advantages.

Comparassion

Key:

• b: branching factor (the average number of child nodes per node)
• d: depth of the shallowest solution
• m: maximum depth of the state space (potentially infinite)

This table elucidates the stark contrasts between these search algorithms, particularly in terms of their space and time efficiency. BFS and IDDFS share the same time complexity, reflecting their completeness and ability to find the optimal solution. However, the space complexity of IDDFS is dramatically lower, akin to DFS, making it an appealing choice for problems where space is a limiting factor and completeness is required.

This table elucidates the stark contrasts between these search algorithms, particularly in terms of their space and time efficiency. BFS and IDDFS share the same time complexity, reflecting their completeness and ability to find the optimal solution. However, the space complexity of IDDFS is dramatically lower, akin to DFS, making it an appealing choice for problems where space is a limiting factor and completeness is required.

Implementation

To solve this problem I splitted it into two parts:

• a “framework” designed to solve state space search problems where the objective is to minimize some goal that can be expressed numerically and can be reused for other problems. For that purpose we implemented a variety of search techniques including Breadth-First Search (BFS), Depth-First Search (DFS), and Iterative Deepening Depth-First Search (IDDFS), each encapsulated within its own class and implementing a common abstract search technique interface.
• a class that defines the state space of the Land Permutation problem and can be coupled with the “framework”.

Problem Modeling

The LandMap class represents the state space, defining the layout of lands and borders that need to be manipulated. It includes methods for state evaluation (border count), generating successors, and performing swap operations which are fundamental actions of the search algorithms. These actions ensure that only valid moves towards the objective are considered, preventing an increase in the number of borders.

The countBorders function is designed to iterate through each cell of the land map matrix and check for borders, where a border is defined as a side where adjacent cells have different values (representing different owners). This operation is O(N) where N is the number of cells in the matrix because it performs a constant amount of work for each cell: one comparison with its right neighbor and one with its bottom neighbor (except for cells on the rightmost and bottom edges, which have no neighbors in that direction). Without two loops or nested loops per cell that would increase the time complexity.

To generate the successor states the border count is tested before creating a new object to ensure the move is beneficial towards achieving the goal of minimizing borders, and this pre-validation step is crucial for maintaining the efficiency of the search algorithms. Each potential swap of adjacent territories in the land map is a candidate move that might lead to a new state; however, not all moves lead to a desirable outcome. By assessing the impact of a swap on the number of borders before actually instantiating a new LandMap object, the algorithm effectively filters out successor states that would not contribute to the solution. This pre-emptive check prevents the unnecessary creation and storage of state objects that do not bring the search any closer to the goal, which would otherwise waste computational resources and potentially slow down the search process due to increased memory usage and garbage collection overhead.

Algorithms

In the implementation of the search algorithms we went for a clean and modular approach, going for an algorithm design that are applicable to a broad range of problems beyond just the land permutation challenge. The Breadth-First Search (BFS) class utilizes a Queue data structure to ensure a level-order traversal. The Depth-First Search (DFS) class employs a Stack to dive deep into the search space, backtracking when necessary. This approach is typical for DFS’s deep exploration strategy, which is both memory-efficient and adept at handling problems with vast search spaces.

Iterative Deepening Depth-First Search (IDDFS) marries the strengths of both BFS and DFS by combining the memory efficiency of DFS with the completeness of BFS. IDDFS systematically increases the depth limit, essentially performing a DFS to the current limit, then restarting with a deeper limit, allowing for a progressive exploration that is both thorough and space-conscious. Each of these classes is architected to encapsulate the search logic within its own module, using polymorphism through an interface that defines the structure of a search technique, making the algorithms interchangeable and reusable.

Results

The following table presents the outcomes of applying different search algorithms to the several instances of the land permutation problem with the w1 goals. Each entry outlines the depth reached, the number of states generated, and the execution time recorded for the corresponding search technique and instance. It’s important to note that an execution time marked as “>60s” indicates that the algorithm did not find a solution within the 60-second time limit. For these instances, the depth and generated states reflect the search progress made up to the point of timeout. These metrics are crucial for understanding the performance and efficiency of each algorithm under the constraints of time-limited execution.

Discussion

The execution of search algorithms on the land permutation problem has demonstrated varied outcomes, which reveal the strengths and weaknesses inherent in each method. The Breath First Search (BFS), noted for its exhaustive level-by-level exploration, has proven effective in shallower instances where the breadth of potential states remains manageable. However, its performance notably diminishes as the complexity of the state space increases, often resulting in timeouts. This indicates that while BFS is thorough, its resource consumption makes it less viable for problems with expansive search trees.

Depth-First Search (DFS), on the other hand, showcased its ability to reach deeper into the search space with a significantly lower number of generated states, emphasizing its depth-focused approach. Its memory efficiency shines in instances where the solution lies far from the root, but this comes with the trade-off of potentially overlooking nearer solutions due to its path-focused traversal. The variation in depth and generated states between DFS and BFS highlight the impact of the algorithms’ differing exploration strategies.

Iterative Deepening Depth-First Search (IDDFS) strikes a balance between the two, ensuring completeness while retaining memory efficiency. Although IDDFS also faced timeouts in more complex instances, it consistently covered more ground, as indicated by the greater depth reached before the time cap. The iterative nature of IDDFS allows it to comb through the state space methodically, which is particularly beneficial when the solution’s depth is unknown, making it a robust choice for a wide range of scenarios.

The results table succinctly captures the essence of these algorithms’ performances and provides a clear comparison that aids in selecting the appropriate approach for different instances of the problem. The trade-offs between time complexity, space complexity, and solution optimality become apparent, informing the decision-making process for algorithm selection in problem-solving.

Further information

Geeks for Geeks - BFS

Geeks for Geeks - DFS

Geeks for Geeks - IDDFS

Russell, S., & Norvig, P. (2001). Artificial Intelligence: A Modern Approach (3rd ed). AIMA

Imagine you want to transfer money from one bank account to another. To do this there are several steps involve: withdrawing the amount from the first account, depositing the money in the second account. Since money unfortunately doesn’t grow in transactions, both of these operations must be completed successfully, if the withdrawn isn’t made the deposit can’t be done other, the amount deposited must be the same as the amount withdrawn. If it is not possible to complete the deposit the money must return to the account where it was withdrawn from. So these operations must be treated as a single operation or as we can say a single unit of work.

From banking to baking, now let’s imagine we want to bake chocolate chip cookies, there is a set of actions you have to perform, like, getting all the ingredients, mixing them together, molding the cookies, putting them in the oven and baking them. While you are doing this your mother is also in the kitchen baking a cake. If she were to come and mix some some ingredients of the cake in the mixing bowl while you are preparing your dough, then it would become too runny or too dry. Or imagine the disappointment on your face tasting some bland hard bread because your mother accidentally used the sugar out of your ingredients. So, these operations must be performed with some level of isolation to ensure each task’s integrity and success.

In both scenarios, the concept of “transaction” comes into play. A transaction, in its essence, is a sequence of operations or actions that are treated as a single unit of work. These operations must either all succeed together or fail together, ensuring data integrity and consistency. This concept is pivotal in various domains, including database management and software development.

Understanding transactions

We use databases to store information, but, for this information to keep it’s usefulness it must be accessed and modified from time to time. To maintain the consistency and integrity of the data present this tasks should be performed systematically with a specific set of rules. In Database Management Systems this is called a transaction.

Before diving deeper into transactions, let’s first clarify what we mean by consistency and integrity, as these are foundational to understanding the role and importance of transactions in database management.

Consistency and Integrity

Consistency refers to the requirement that any transaction should bring the database from one valid state to another, ensuring that all data follows all rules and constraints (e.g., data types, triggers, constraints) of the database. For instance, in our cookie-baking analogy, consistency would be akin to following a recipe precisely. Just as using the right proportions of ingredients ensures the cookies turn out as expected (not too runny or too dry), in a database, consistency ensures that all data remains accurate and in the correct format throughout any transaction.

Integrity, on the other hand, involves maintaining the accuracy and reliability of the data over its entire lifecycle. This means that the data in the database is always accurate, and any changes made to it are done correctly. Going back to our baking analogy, integrity is similar to ensuring that the sugar intended for your cookies isn’t mistakenly used in your mother’s cake.

In summary, consistency and integrity in databases are like following a recipe and using the correct ingredients in baking. Just as each step and ingredient must be correctly followed and used to produce the desired cookie outcome, transactions in a database must adhere to rules that preserve data consistency and integrity, ensuring the database remains useful, accurate, and reliable.

Definition

GeeksForGeeks defines a transaction as a set of logically related operations. It is the result of a request made by the user to access the contents of the database and perform operations on it. It consists of various operations and has various states in its completion journey. It also has some specific properties that must be followed to keep the database consistent.

ACID

These proprieties mentioned in the definition followed by transactions are usually referred to has ACID properties, they are Atomicity, Consistency, Isolation and Durability. We already mentioned Consistency.

Atomicity ensures that a transaction is treated as a single unit of work, meaning, that it either completes in its entirety or it doesn’t happen at all. There is no in-between state. If any part of the transaction fails, the entire transaction is rolled back, and the database state is left unchanged as if the transaction never occurred. Baking cookies is not a good example for this, because, if you realize midway that your oven is broken, you can not return to a state where all the ingredients are separated and in their original container, but, I’m sure you get the idea.

Isolation ensures that transactions are securely isolated from each other. This means the operations of one transaction are hidden from other transactions until it’s completed. This property prevents transactions from interfering with each other, ensuring data integrity. Meaning, that the other transaction in your kitchen - your mother - can not interfere with your cookie baking. Like if both of you have separate workstations separate from each other.

Durability guarantees that once a transaction has been committed, it will remain so, even in the event of a power loss, crash, or error. This means the changes made by the transaction are permanently recorded in the database. Once the cookies are baked and taken out of the oven, they don’t magically disappear if the power goes out - only if you eat them, but that is another story.

By adhering to these ACID properties, databases ensure that transactions are processed reliably, maintaining the integrity and consistency of the data.

Operations

We mentioned before that a user can make different types of operations to access the contents of the database.

The information in the database can be read - Read(X) - that is, the value is read from the database and stored in a buffer for displaying or other action. Like checking your account ballance online or your pantry to see if you have all the ingredients needed for the cookies. You are not taking anything out or using it.

During a write operation we write - Write(X) the value from the memory buffer to the database. It must be preceded by a read operation during which the are brought to the buffer and some operations are performed on it - according to what the user requested. Then the modified value is written in the database. Think of checking you balance (Read) before you withdraw (Write) money from you bank account or you check what you have (Read) before putting all of your ingredients together (Write) to bake your cookies.

Commit is an operation that ensures that the integrity of the database is maintained. Operations are only made permanent after all of the work performed by the current transaction is completed, that is, the changes done by the transaction are made permanent in the database. There may be interruptions on transactions (like an error or a power failure), and this way we ensure that the consistency of the data is maintained. Imagine a Commit like you with or apron and the oven gloves with a batch of cookies ready for baking.

Rollback is intimately connected to the transaction, it’s what happens when a transaction is interrupted, all the operations are undone and the database returns to the original state. The cookie example is not a good one, but, it’s like if you decide that the batch isn’t good, and there is a operation that can return all the ingredients to the pantry.

Deadlock

When a transaction needs to read or modify data, to ensure that no other transaction makes conflicting changes, it acquires locks to certain resources. When two or more transactions hold locks on resources the others need to complete their operations, and none can proceed until the other releases its locks. Each transaction is waiting for the other to finish, but neither can without the resources held by the other. This situation is called a deadlock.

Deadlocks can significantly hinder database performance, leading to stalled transactions and system inefficiencies. To manage deadlocks, Database Management Systems (DBMS) employ various strategies, including deadlock detection algorithms that identify and break deadlocks by aborting one or more transactions to free up resources. Another approach is deadlock prevention, which involves designing the database and transactions in a way that avoids the conditions leading to deadlocks.

Imagine you and your mother are both trying to bake in a kitchen with only one oven and one oven mint. If you both are ready to bake and you hold the mint and your mother the oven, neither of you can proceed creating “kitchen deadlock.” In databases, similar scenarios require careful management to ensure smooth operation.

Conclusion

Transactions are fundamental to maintaining the consistency, integrity, and reliability of data in database systems. By understanding and implementing the ACID properties, databases can ensure that transactions are processed in a manner that preserves data integrity, even in the face of errors or system failures. Operations like Read, Write, Commit, and Rollback play crucial roles in transaction management, while mechanisms to handle deadlocks ensure that databases remain efficient and responsive.

Just as following a recipe step by step leads to delicious cookies, adhering to transaction principles ensures that databases function effectively, supporting a wide range of applications from financial services to online shopping. Understanding transactions and their management is essential for anyone working with database systems, providing the foundation for building robust and reliable software solutions.

Further information

Geeks for Geeks

Tutorialspoint

Javapoint