SAFARI Live Seminar: Efficient DNN Training at Scale: from Algorithms to Hardware – Gena Pekhimenko

SAFARI Live Seminar: Efficient DNN Training at Scale: from Algorithms to Hardware - Gena Pekhimenko

1343 People Read – 33 People Liked – You Can Also Like

Talk Title: Efficient DNN Training at Scale: from Algorithms to Hardware

Speaker: Gennady Pekhimenko, Assistant Professor at the University of Toronto
http://www.cs.toronto.edu/~pekhimenko/

SAFARI Live Seminar Series Summer 2021 Talk #4

SAFARI Seminar Series

Talk slides (pdf): https://safari.ethz.ch/safari_public_wp/wp-content/uploads/2021/07/SAFARI-Talk-GennadyPekhimenko-August-05-2021.pdf
Talk slides (pptx): https://safari.ethz.ch/safari_public_wp/wp-content/uploads/2021/07/SAFARI-Talk-GennadyPekhimenko-August-05-2021.pptx

More info on this talk: https://safari.ethz.ch/safari-live-seminar-gennady-pekhimenko-5-august-2021/

Abstract:
The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus of systems research is usually quite narrow and limited to (i) inference — i.e. how to efficiently execute already trained models and (ii) image classification networks as the primary benchmark for evaluation. In this talk, we will demonstrate a holistic approach to DNN training acceleration and scalability starting from the algorithm, to software and hardware optimizations, to special development and optimization tools.

In the first part of the talk, I will show our radically new approach on how to efficiently scale backpropagation algorithms used in DNN training (BPPSA, MLSys’20). Then I will demonstrate a new approach on how to train multiple DNN models jointly on the same hardware (HFTA, MLSys’21). I will then demonstrate several approaches to deal with one of the major limiting factors in DNN training: limited GPU/accelerator memory capacity (Echo, ISCA’20 and Gist, ISCA’18). At the end, I will show the performance and visualization tools we built in my group to understand, visualize, and optimize DNN models, and even predict their performance on different hardware.

Speaker Bio:
Gennady Pekhimenko is an Assistant Professor at the University of Toronto, CS department and (by courtesy) ECE department, where he is leading the EcoSystem (Efficient Computing Systems) group. Gennady is also a Faculty Member at Vector Institute and a CIFAR AI chair. Before joining Univ. of Toronto, he spent a year in 2017 at Microsoft Research in Redmond in the Systems Research group. He got his PhD from the Computer Science Department at Carnegie Mellon University in 2016. Gennady is a recipient of Amazon Machine Learning Research Award, Facebook Faculty Research Award, Connaught New Researcher Award, NVIDIA Graduate, Microsoft Research, Qualcomm Innovation, and NSERC CGS-D Fellowships. His research interests are in the areas of systems, computer architecture, compilers, and applied machine learning.

Upcoming SAFARI Live Seminars: https://safari.ethz.ch/safari-seminar-series/
More information on this talk: To Be Linked

Other Related Talks:
================
SAFARI Live Seminar Series

SAFARI Seminar Series

Other suggested videos from this channel:
===================================
Memory-Centric Computing Systems IEDM Tutorial

Intelligent Architectures for Intelligent Machines:

Future Computing Architectures:

Interview with Onur Mutlu on Computing Research and Education:

The Story of RowHammer:

Accelerating Genome Analysis:

Public Lectures Playlist:

Talks from SAFARI Research Group Playlist:

Youtube

Make Beautify

SAFARI Live Seminar: Efficient DNN Training at Scale: from Algorithms to Hardware – Gena Pekhimenko

By Luther