Machine Learning Interview Questions & Answers: Concepts to Practice

The bias-variance tradeoff describes the tension between two sources of prediction error. Bias (underfitting): • Error from oversimplified assumptions in the model • The model cannot capture the underlying patterns in the data • Symptoms: poor performance on both training and validation data • Fixes: use a more complex model, add more features, reduce regularization, increase training time Variance (overfitting): • Error from the model being too sensitive to fluctuations in the training data • The model memorizes noise instead of learning the true signal • Symptoms: excellent training performance but poor validation/test performance • Fixes: get more training data, simplify the model, add regularization (L1/L2, dropout), use ensemble methods, apply early stopping Regularization techniques: • L1 (Lasso) — pushes coefficients to exactly zero, performing feature selection • L2 (Ridge) — shrinks coefficients toward zero but doesn't eliminate them, better when all features are relevant • Elastic Net — combines L1 and L2 for the best of both worlds • Dropout — randomly disables neurons during training (neural networks) • Early stopping — halt training when validation loss stops improving Practical approach: Start with a simple model (high bias, low variance), then gradually increase complexity while monitoring validation performance.

Both are ensemble methods that combine multiple decision trees, but they use fundamentally different strategies. Random Forests (bagging): • Train many trees independently on random subsets of data and features • Final prediction is the average (regression) or majority vote (classification) • Strengths: resistant to overfitting, easy to parallelize, robust to hyperparameter choices • Weaknesses: generally lower accuracy than boosting on structured data, larger model size Gradient Boosted Trees (boosting — XGBoost, LightGBM, CatBoost): • Train trees sequentially — each tree corrects the errors of the previous ones • Final prediction is the sum of all tree predictions • Strengths: typically higher accuracy on structured/tabular data, smaller models • Weaknesses: prone to overfitting without careful tuning, slower to train (sequential), sensitive to hyperparameters Decision guide: • Choose Random Forest when: you want a quick baseline, data is noisy, you have limited time for hyperparameter tuning, or you need fast inference • Choose Gradient Boosting when: you need maximum accuracy on tabular data, you have time for hyperparameter tuning, and you need competitive leaderboard performance • Real-world note: XGBoost/LightGBM dominate Kaggle competitions on tabular data, but Random Forests are often preferred in production for their stability and ease of deployment

Discriminative models: • Learn the decision boundary between classes — model P(y|x) directly • Focus on: what features distinguish class A from class B? • Examples: logistic regression, SVM, neural networks, decision trees • Advantages: often higher accuracy for classification, simpler to train, directly optimize for the task Generative models: • Learn the underlying distribution of each class — model P(x|y) and P(y), then use Bayes' rule for P(y|x) • Focus on: what does data from class A look like? • Examples: Naive Bayes, Gaussian Mixture Models, Hidden Markov Models, GANs, VAEs, diffusion models • Advantages: can generate new data, handle missing features, provide richer information about data structure When to choose each: • Classification tasks with plenty of labeled data → discriminative • Small labeled datasets → generative (they can leverage unlabeled data) • Need to generate synthetic data → generative • Need interpretable class probabilities → depends on specific model choice Modern context: Large language models (GPT, Claude) are generative models that have become remarkably effective at discriminative tasks through in-context learning and fine-tuning.

Backpropagation is the algorithm for computing gradients of the loss function with respect to each weight in the network, enabling gradient descent optimization. How it works: 1. Forward pass — input flows through the network layer by layer, computing activations 2. Compute loss — compare the output with the target using a loss function 3. Backward pass — apply the chain rule to compute gradients layer by layer, from output back to input 4. Update weights — adjust each weight in the direction that reduces the loss (gradient descent) Vanishing gradients: • Gradients shrink exponentially as they propagate through many layers • Common with sigmoid/tanh activations (derivatives < 1 multiply together) • Result: early layers learn extremely slowly or not at all • Solutions: use ReLU activation, residual connections (skip connections), LSTM/GRU for recurrent networks, proper weight initialization (He/Xavier) Exploding gradients: • Gradients grow exponentially, causing unstable training • Common in deep networks and recurrent networks processing long sequences • Result: weights oscillate wildly, loss becomes NaN • Solutions: gradient clipping, batch normalization, careful learning rate selection, LSTM/GRU gating mechanisms Modern approaches: Transformer architectures with layer normalization and residual connections largely mitigate both problems, which is one reason they have become the dominant architecture.

Transfer learning uses a model pretrained on a large dataset as a starting point for a new, related task. Instead of training from scratch, you leverage learned representations. How it works: 1. Start with a model pretrained on a large dataset (e.g., ImageNet for vision, BookCorpus for NLP) 2. Replace or modify the final layer(s) to match your task's output 3. Fine-tune on your smaller, task-specific dataset Fine-tuning strategies: • Feature extraction — freeze all pretrained layers, only train the new head. Best when: your dataset is very small and similar to the pretraining data • Partial fine-tuning — freeze early layers (general features), fine-tune later layers. Best when: moderate dataset size, somewhat different domain • Full fine-tuning — unfreeze all layers, train with a small learning rate. Best when: large dataset available, different domain When transfer learning works best: • Your dataset is too small to train a deep model from scratch • The source and target domains share some common structure • You need to reduce training time and compute costs Domain-specific examples: • Vision: ResNet/EfficientNet pretrained on ImageNet → fine-tune for medical imaging • NLP: BERT/GPT pretrained on web text → fine-tune for sentiment analysis, NER, or question answering • Audio: Wav2Vec pretrained on speech → fine-tune for speech recognition in a specific language

Precision — of all positive predictions, what fraction are actually positive? • Optimize when: false positives are costly (spam filtering — don't send important emails to spam; medical diagnosis — avoid unnecessary treatments) Recall (Sensitivity) — of all actual positives, what fraction did we correctly identify? • Optimize when: false negatives are costly (cancer screening — don't miss cancer; fraud detection — don't miss fraudulent transactions) F1 Score — harmonic mean of precision and recall • Use when: you need a single metric that balances precision and recall • Especially useful for imbalanced datasets where accuracy is misleading • F-beta score lets you weight precision vs recall differently AUC-ROC — area under the Receiver Operating Characteristic curve • Measures the model's ability to discriminate between classes across all thresholds • Use when: you want a threshold-independent measure of model quality • Especially useful for comparing models or when the operating threshold hasn't been decided • Limitation: can be misleading on highly imbalanced datasets — use AUC-PR (Precision-Recall) instead Practical framework: 1. Understand the business cost of false positives vs false negatives 2. Choose the metric that reflects that cost 3. For imbalanced data (>90% one class), avoid accuracy — use F1, AUC-PR, or a custom cost-sensitive metric 4. Always report multiple metrics — never rely on a single number