Please use this identifier to cite or link to this item: doi:10.22028/D291-36738
Title: Connections Between Numerical Algorithms for PDEs and Neural Networks
Author(s): Alt, Tobias
Schrader, Karl
Augustin, Matthias
Peter, Pascal
Weickert, Joachim
Language: English
Title: Journal of Mathematical Imaging and Vision
Publisher/Platform: Springer Nature
Year of Publication: 2022
Free key words: Numerical algorithms
Partial differential equations
Neural networks
Nonlinear diffusion
DDC notations: 004 Computer science, internet
510 Mathematics
Publikation type: Journal Article
Abstract: We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights, we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.
DOI of the first publication: 10.1007/s10851-022-01106-x
URL of the first publication:
Link to this record: urn:nbn:de:bsz:291--ds-367386
ISSN: 1573-7683
Date of registration: 11-Jul-2022
Faculty: MI - Fakultät für Mathematik und Informatik
Department: MI - Informatik
MI - Mathematik
Professorship: MI - Prof. Dr. Joachim Weickert
MI - Keiner Professur zugeordnet
Collections:SciDok - Der Wissenschaftsserver der Universität des Saarlandes

Files for this record:
File Description SizeFormat 
Alt2022_Article_ConnectionsBetweenNumericalAlg.pdf1,63 MBAdobe PDFView/Open

This item is licensed under a Creative Commons License Creative Commons