Kevin Jamieson (UC Berkeley) “Bayesian Optimization and Other Potentially Bad Ideas for Hyperparameter Optimization”
Add to Calendar
When:
April 14, 2017 @ 12:00 pm – 1:15 pm
2017-04-14T12:00:00-04:00
2017-04-14T13:15:00-04:00
Where:
Hackerman Hall B17
3400 N Charles St
Baltimore, MD 21218
USA
3400 N Charles St
Baltimore, MD 21218
USA
Cost:
Free
Abstract
Performance of machine learning systems depends critically on tuning parameters that are difficult to set by standard optimization techniques. Such “hyperparamers”—including model architecture, regularization, and learning rates—are often tuned in an outerloop by black-box search methods evaluating performance on a holdout set. We formulate such hyperparameter tuning as a pure-exploration problem of deciding how many resources should be allocated to particular hyperparameter configurations. I will introduce our Hyperband algorithm for this framework and a theoretical analysis that demonstrates its ability to adapt to uncertain convergence rates and the dependency of hyperparameters on the validation loss. I will close with several experimental validations of Hyperband, including experiments on training deep networks where Hyperband outperforms state-of-the-art Bayesian optimization methods by an order of magnitude. I will also describe a highly scalable and asynchronous version of Hyperband I implemented and validated at Google.
Biography
Kevin Jamieson is a postdoctoral researcher working with Professor Benjamin Recht in the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley. He is interested in the theory and practice of machine learning algorithms that sequentially collect data using an adaptive strategy. This includes active learning, multi-armed bandit problems, and stochastic optimization. Kevin received his Ph.D. from the University of Wisconsin, Madison under the advisement of Robert Nowak. Prior to his doctoral work, Kevin received his B.S. from the University of Washington, and an M.S. from Columbia University, both in electrical engineering.