Skip to main

A Layer-Parallel Approach for Training Deep Neural Networks

Jul 31, 2020, 10:00 - 11:00 AM

Virtual Colloquium Link	Join Zoom Meeting Meeting number (access code): 954 8656 9124 Meeting password: 862575
Date	Friday, July 31, 2020
Time	10:00-11:00 am (Eastern Time)
Speaker	Dr. Eric C. Cyr
Affiliation	Sandia National Laboratories
Title	A Layer-Parallel Approach for Training Deep Neural Networks
Abstract	Deep neural networks are a powerful machine learning tool with the capacity to “learn” complex nonlinear relationships described by large data sets. Despite their success training these models remains a challenging and computationally intensive undertaking. In this talk we will present a new layer-parallel training algorithm that exploits a multigrid scheme to accelerate both forward and backward propagation. Introducing a parallel decomposition between layers requires inexact propagation of the neural network. The multigrid method used in this approach stiches these subdomains together with sufficient accuracy to ensure rapid convergence. We demonstrate an order of magnitude wall-clock time speedup over the serial approach, opening a new avenue for parallelism that is complementary to existing approaches. Results for this talk can be found in [1,2]. We will also present related work concerning parallel-in-time optimization algorithms for PDE-constrained optimization. [1] S. Guenther, L. Ruthotto, J. B. Schroder, E. C. Cyr, N. R. Gauger, Layer-Parallel Training of Deep Residual Neural Networks, SIMODs, Vol. 2 (1), 2020. [2] E. C. Cyr, S. Guenther, J. B. Schroder, Multilevel Initialization for Layer-Parallel Deep Neural Network Training, arXiv preprint arXiv:1912.08974, 2019.

* The programs and services offered by George Mason University are open to all who seek them. George Mason does not discriminate on the basis of race, color, religion, ethnic national origin (including shared ancestry and/or ethnic characteristics), sex, disability, military status (including veteran status), sexual orientation, gender identity, gender expression, age, marital status, pregnancy status, genetic information, or any other characteristic protected by law. After an initial review of its policies and practices, the university affirms its commitment to meet all federal mandates as articulated in federal law, as well as recent executive orders and federal agency directives.