Data & ML / Research & evaluation · 2025
Evaluating machine learning beyond a single score
Four graduate ML projects combining pipeline development, design under compute constraints, and comparative evaluation across supervised learning, optimization, unsupervised learning, and reinforcement learning.
Course policy prohibits posting these papers publicly. Private sharing with prospective employers is permitted.
Context
Graduate machine learning coursework · Georgia Tech · Fall 2025
My contribution
Implementation and debugging with AI coding agents, adapting experiments to memory and runtime limits, comparative analysis, and four technical reports.
The takeaway
Useful model comparisons make error costs, the scope of optimization, computational tradeoffs, and uncertainty explicit.
Three contributions
- Built and debugged four ML projects. I spent dozens of hours developing pipelines in collaboration with AI coding agents, including Codex and Claude Code. The work included integrating data processing, training, evaluation, and visualization; migrating a neural-network implementation from TensorFlow to PyTorch; and adapting code between classification and regression. I reviewed, tested, and refactored the assisted code.
- Made experiments feasible within available compute. Working with a source dataset containing approximately 7.7 million records, I adapted training subsets, search budgets, and the allocation of time across experiments to memory and runtime limits. I documented where these choices narrowed the evidence; some experiments used subsets of the source data.
- Compared algorithms and authored four technical reports. I used hyperparameter searches, learning curves, repeated trials, statistical tests, and runtime analysis as appropriate to each study, connecting the measurements to algorithm behavior, practical implications, and limitations.
Project context
The course supplied the datasets, algorithm families, and core comparisons. Within that framework, three connected studies examined hotel booking cancellation classification and traffic incident duration regression through supervised learning, optimization, and representation learning. A fourth studied reinforcement learning in simulated environments. My work combined implementation, design choices within those requirements, experimentation, and interpretation.
Four studies
Supervised learning
Methods: decision trees, k-nearest neighbors, linear and RBF support vector machines, and neural networks.
I connected tree depth to complexity and overfitting, compared linear and RBF kernels for predictive quality and computational cost, and investigated network depth versus width at similar parameter counts. I considered high dimensionality and outliers as possible explanations for nearest-neighbor behavior, and examined training and prediction costs separately.
To connect evaluation with practical decisions, I interpreted cancellation errors in terms of unused capacity and overbooking. I also examined extreme duration-prediction residuals for possible data-quality problems or consequential underestimates.
Optimization and generalization
Methods: randomized hill climbing, simulated annealing, genetic algorithms, SGD and Adam-family optimizers, and neural network regularization.
I used optimizer ablations to examine momentum, adaptive learning rates, and bias correction. My discussion related momentum to smoothing oscillations and adaptive learning rates to convergence behavior, while distinguishing optimization speed from held-out predictive quality.
I explicitly qualified comparisons between restricted optimization and whole-network training: the scope of what could change limited what a direct ranking could establish.
Unsupervised learning
Methods: K-means and Gaussian mixture-model clustering; principal component analysis (PCA), independent component analysis (ICA), and random projection; and neural networks using transformed or cluster-derived features.
I examined explained variance for PCA, component loadings and kurtosis for ICA, and reconstruction error and variability across random projections. I evaluated cluster separation, dimensionality, runtime, and downstream predictive error as distinct outcomes.
I also identified a baseline limitation: a network configured for its original inputs may need further tuning to fairly assess transformed inputs. This qualified how much of a performance difference could be attributed to the representation alone.
Reinforcement learning
Methods: value iteration, policy iteration, SARSA, and Q-learning in Blackjack and CartPole.
My report explained that SARSA updates use the action actually taken in the next state, while Q-learning updates use the highest estimated next-action value. I examined how coarse state grids lose distinctions and fine grids leave more states with limited experience.
I distinguished a higher observed mean from a statistically supported improvement, and discussed how stopping conditions and incomplete convergence limited performance claims. A slowing learning curve, stable behavior, and attainment of the task’s success criterion required different evidence.
Evidence and limits
These were coursework experiments. Subsampling, tuning budgets, and reused model configurations limit how far their conclusions transfer. The reports discuss practical implications; they do not establish production performance or measured business impact.
For a public example with a report and interactive results, see my separate neural readmission replication study.