13 - Backpropogation - lec. 18

ucla | CS M146 | 2023-05-31T13:59

---

Table of Contents

• Supplemental
• Lecture
  • Forward and Backward Pass
  • Architecture (DAG)
  • Backprop (DAG) - Scalar
    • Simple DAG
    • Common Gate Reduction
    • Torch Autograd
  • Backprop (Tensors) - Vector
    • Vector (Matrix) Derivatives
    • Jacobian Backprop - implicit
    • Matrix Multiplication - implicit Jacobian
    • Another viewpoint
• Discussion
• Resources

Supplemental

• the purpose of backprop - to calculat gradients for weights and perform gradient descent
• uptream gradient = local gradient * downstream gradient
• gradient of max function = 0 if max=0 or 1 if max=x

Lecture

Forward and Backward Pass

Architecture (DAG)

Backprop (DAG) - Scalar

Simple DAG

Common Gate Reduction

Torch Autograd

Backprop (Tensors) - Vector

Vector (Matrix) Derivatives

Jacobian Backprop - implicit

• must be computed implicitly (diagonally) as constructing jacobians for large data causes memory explosion

Matrix Multiplication - implicit Jacobian

Another viewpoint

• matrix mult is associative so w can do chain rule

Discussion

Resources

---

📌

**SUMMARY
**