Multi-Objective Least-Squares

This module explores multi-objective least-squares optimization, discussing:

• Weighted-sum objectives and their minimization
• Regularized least-squares approaches
• Laplacian regularization techniques
• Nonlinear least-squares (NLLS) methods
• Gauss-Newton method and its applications
• Least-norm solutions of underdetermined equations

Course Lectures

• Overview of Linear Dynamical Systems

  Stephen Boyd

  Play

  This module introduces the concept of linear dynamical systems, discussing their significance and applications in various fields. It provides examples to illustrate how these systems are used in practice, including estimation and filtering techniques.

• Linear Functions (Continued)

  Stephen Boyd

  Play

  This module continues the exploration of linear functions through various practical examples. It covers:

  • Linear static circuit examples
  • Linear elastic structures
  • Cost of production scenarios
  • Network traffic and flow analysis

  Additionally, it introduces linearization and the first-order approximation of functions to simplify complex systems.

• Linearization (Continued)

  Stephen Boyd

  Play

  This module delves into the concept of linearization in various applications, including navigation and range measurement. It discusses:

  • Matrix multiplication as a mixture of columns
  • Block diagram representations
  • A review of linear algebra concepts such as basis and dimension
  • The nullspace of a matrix
• Nullspace of a Matrix (Continued)

  Stephen Boyd

  Play

  This module focuses on the nullspace of a matrix and its significance in linear transformations. It covers:

  • The range of a matrix
  • Inverse and rank of matrices
  • Conservation of dimension principles
  • Applications of fast matrix-vector multiplication
  • Change of coordinates and inner product concepts
• Orthonormal Set of Vectors

  Stephen Boyd

  Play

  This module introduces orthonormal sets of vectors, discussing their geometric interpretations. Key topics include:

  • The Gram-Schmidt procedure and its applications
  • Full QR factorization
  • Orthogonal decomposition induced by matrices
  • Least-squares techniques in the context of orthonormal sets
• Least-Squares

  Stephen Boyd

  Play

  This module covers least-squares methods, providing a geometric interpretation and discussing:

  • Approximate solutions and projections on column spaces
  • Least-squares via QR factorization
  • Estimation techniques and the BLUE property
  • Navigation applications from range measurements
  • Data fitting using least-squares principles
• Least-Squares Polynomial Fitting

  Stephen Boyd

  Play

  This module focuses on least-squares polynomial fitting, discussing:

  • The relationship between the norm of optimal residuals and polynomial degree
  • System identification through least-squares methods
  • Model order selection and cross-validation techniques
  • Recursive least-squares approaches
  • Multi-objective least-squares challenges
• Multi-Objective Least-Squares

  Stephen Boyd

  Playing

  This module explores multi-objective least-squares optimization, discussing:

  • Weighted-sum objectives and their minimization
  • Regularized least-squares approaches
  • Laplacian regularization techniques
  • Nonlinear least-squares (NLLS) methods
  • Gauss-Newton method and its applications
  • Least-norm solutions of underdetermined equations
• Least-Norm Solution

  Stephen Boyd

  Play

  This module discusses least-norm solutions and their derivation through QR factorization. Topics include:

  • Use of Lagrange multipliers in derivation
  • Examples illustrating the concept of transferring mass unit distance
  • Connections to regularized least-squares methods
  • General norm minimization with equality constraints
  • Overview of autonomous linear dynamical systems
• Examples of Autonomous Linear Dynamical Systems

  Play

  This module provides examples of autonomous linear dynamical systems, including:

  • Finite-state discrete-time Markov chains
  • Numerical integration techniques for continuous systems
  • High-order linear dynamical systems
  • Mechanical systems analysis
  • Linearization near equilibrium points and along trajectories
• Solution via Laplace Transform and Matrix Exponential

  Stephen Boyd

  Play

  This module discusses the solution of linear systems via the Laplace transform and matrix exponential, including:

  • Laplace transform solutions of linear equations
  • Examples like the harmonic oscillator and double integrator
  • Characteristic polynomial and eigenvalues
  • Matrix exponential and its time transfer property
• Time Transfer Property

  Stephen Boyd

  Play

  This module covers the time transfer property in linear systems, addressing:

  • Piecewise constant systems and their qualitative behavior
  • Stability analysis and eigenvector properties
  • Scaling interpretations and dynamic behaviors
  • Invariant sets and their significance
  • Markov chain examples illustrating these concepts
• Markov Chain (Example)

  Stephen Boyd

  Play

  This module presents a detailed example of a Markov chain, discussing:

  • Diagonalization and its implications for system stability
  • Distinct eigenvalues and their significance
  • Modal forms and diagonalization examples
  • Jordan canonical form and generalized eigenvectors
• Jordan Canonical Form

  Stephen Boyd

  Play

  This module introduces Jordan canonical form, discussing its significance in linear dynamical systems. Key topics include:

  • Generalized modes and their applications
  • The Cayley-Hamilton theorem and its proof
  • Linear dynamical systems with inputs and outputs
  • Block diagrams and transfer matrices
  • Impulse and step matrices
• DC or Static Gain Matrix

  Stephen Boyd

  Play

  This module focuses on the DC or static gain matrix and its applications, covering:

  • Discretization with piecewise constant inputs
  • Causality in linear systems
  • State concepts and change of coordinates
  • Z-Transform and its significance
  • Symmetric matrices and quadratic forms
• RC Circuit (Example)

  Stephen Boyd

  Play

  This module provides an example using an RC circuit to illustrate concepts of quadratic forms. It discusses:

  • Examples of quadratic forms and their properties
  • Inequalities related to quadratic forms
  • Positive semidefinite and positive definite matrices
  • Matrix inequalities and their implications
  • Ellipsoids and their significance in linear systems
• Gain of a Matrix in a Direction

  Stephen Boyd

  Play

  This module examines the gain of a matrix in a specific direction, discussing:

  • Singular value decomposition (SVD) and its applications
  • Interpretations of SVD in data analysis
  • General pseudo-inverse concepts and regularization techniques
  • Full SVD applications in estimation and inversion
  • Sensitivity of linear equations to data errors
• Sensitivity of Linear Equations to Data Error

  Stephen Boyd

  Play

  This module discusses the sensitivity of linear equations to data errors, including:

  • Low rank approximations and their applications
  • Distance to singularity in data analysis
  • Model simplification techniques
  • Controllability and state transfer in linear systems
  • Reachability concepts for discrete-time linear dynamical systems
• Reachability

  Stephen Boyd

  Play

  This module covers reachability in linear systems, discussing:

  • Controllable systems and least-norm inputs
  • Minimum energy solutions over infinite horizons
  • Continuous-time reachability techniques
  • Impulsive inputs and their effects
  • Least-norm input strategies for achieving reachability
• Continuous-Time Reachability

  Stephen Boyd

  Play

  This final module addresses continuous-time reachability and its implications in linear systems, covering:

  • General state transfer techniques
  • Observability and state estimation setups
  • Observability matrix and its importance
  • Least-squares observers for state estimation
  • Final thoughts on linear algebra and future directions