Projects & Highlights

Implicit functions such as Neural Radiance Fields (NeRFs), occupancy networks, and signed distance functions (SDFs) have become pivotal in computer vision for reconstructing detailed object shapes from sparse views. Achieving optimal performance with these models can be challenging due to the extreme sparsity of inputs and distribution shifts induced by data corruptions. To this end, large, noise-free synthetic datasets can serve as shape priors to help models fill in gaps, but the resulting reconstructions must be approached with caution.

Uncertainty estimation is crucial for assessing the quality of these reconstructions, particularly in identifying areas where the model is uncertain about the parts it has inferred from the prior. In this paper, we introduce Dropsembles, a novel method for uncertainty estimation in tuned implicit functions.

We demonstrate the efficacy of our approach through a series of experiments, starting with toy examples and progressing to a real-world scenario. Specifically, we train a Convolutional Occupancy Network on synthetic anatomical data and test it on low-resolution MRI segmentations of the lumbar spine.

Our results show that Dropsembles achieve the accuracy and calibration levels of deep ensembles but with significantly less computational cost.

In modern data science and machine learning, particularly for large-scale problems, optimization plays a crucial role. Mini-batching has emerged as a practical approach, yet its theoretical underpinnings remain incompletely understood. This work aims to bridge this gap by providing rigorous theoretical analysis of mini-batch stochastic gradient descent (SGD) methods.

We prove that mini-batch SGD, applied to consistent least squares problems, converges at the same rate as its deterministic counterpart, given a sufficiently large batch size. Our analysis accommodates versatile sampling procedures, encompassing both standard SGD with uniform sampling and averaging Kaczmarz methods.

We provide both expectation and high-probability bounds, leveraging novel concentration results for products of matrices—a departure from traditional optimization proof techniques. Building on Bollapragada et al.'s work [4], which showed expected convergence for mini-batch SGD with heavy ball momentum, we extend their results to provide high-probability bounds under the same conditions.

Additionally, we present expectation bounds for μ-strongly convex and L-smooth functions that closely approximate quadratics, demonstrating that mini-batch SGD in the interpolation regime converges similarly to gradient descent, given an adequately large batch size.

Our results not only advance the theoretical understanding of mini-batching but also offer practical insights for algorithm selection and tuning in large-scale optimization scenarios. By bridging the gap between stochastic and deterministic methods, this work contributes to the foundation of efficient, scalable optimization techniques for modern machine learning applications.

Assume a data set of patient's covariates (think: blood analysis, symptomatic, etc...) drawn from some population. We give treatment with a certain probability to this population set and we observe the outcome variable on both patient with treatment and without.

Further assume an estimate of the difference of effects between the two groups (treatment vs no treatment) which is assumed to be sparse.

The goal is to show a lower bound on any procedure (algorithm) for recovering the support of the difference in treatment effect.

Table of Contents

1. Linear Algebra Review

• 1.1 Vector Spaces
• 1.2 Some Matrix Properties/Definitions
• 1.3 Eigenvalues and Eigenvectors
• 1.4 Similarity Transformations
• 1.5 Singular Value Decomposition
• 1.6 Inner product, Norms and Metric Spaces
• 1.7 Matrix Norm

2. Direct Methods for Linear Systems of Equations

• 2.1 Gaussian Elimination Methods and LU Factorization
• 2.2 Direct Solvers in Eigen
• 2.3 Exploiting Structure when Solving LSE
• 2.4 Not Discussed

3. Direct Methods for Linear Least Squares

• 3.1 Least Squares Definition and Setup
• 3.2 Geometric Interpretation of Least Squares
• 3.3 Normal Equation
  • 3.3.1 Solving the N.EQ
• 3.4 Orthogonal Transformation for Solving Least Squares Problems
  • 3.4.1 QR decomposition
  • 3.4.2 Computation of QR decomposition
  • 3.4.3 Eigen: QR Decomposition for Solving LS Systems
• 3.5 Moore-Penrose Pseudoinverse
• 3.6 Not Discussed in This Script

4. SVD Based Methods for Least Square Problems

• 4.1 SVD for Solving General Least Squares Problems
• 4.2 SVD in Eigen

Table of Contents

1. Statistical Learning Framework
2. Basics of Regression Analysis and Prediction Tasks
   • Random Design
   • Fixed Design
3. Theory of Linear Regression
   • Multiple Linear Regression in Compact Form
   • Multivariate Linear Regression
4. Theory of Hypothesis Testing
   • Modeling the Problem
5. Statistical Modelling for Linear Regression

Table of Contents

1. Classical Linear Model
   • Modelling Effect of Covariates
   • Parameter Estimation
   • Geometric Properties
   • Analysis of Variance
   • Statistical Properties
   • Asymptotic Properties of LSE
   • Statistical Properties of Residuals
2. Hypotheses Testing
3. Multiple Testing
4. Model Selection
5. Model Diagnostic
6. General Linear Model
7. Robust Regression
8. Generalized Linear Models
9. Penalized Regression
10. Maths Tricks Tutorials