The so called curse of dimensionality in machine learning is the observation that neural networks with many parameters can be impossibly difficult to train due to the vastness of its parameter space. Another issue that arises in practice is that most of the neural network does not do anything, as a lot of its weights turn out to be redundant. This is because many (if not all) of the problems we’re interested in solving as engineers have some inherent sparsity. Steve Brunton has an excellent video explaining why this is so. ...

Let $\mathcal X$ be a Hilbert space, which means that it is a vector space that has an inner product (denoted by $\langle \cdot, \cdot\rangle _\mathcal X$) and that it is complete, i.e., it doesn’t have er… holes in it. Recall that inner product spaces have a rich geometric structure, and so do Hilbert spaces. The Euclidean space $\mathbb R^n$ is an obvious example, where the inner product is just the dot product. Mathematicians sometimes use ‘Hilbert space’ to refer specifically to infinite-dimensional inner product spaces, but for our purposes, we will conflate the usage of ‘Hilbert space’ to include the finite-dimensional case. ...

Let’s look at the norm balls corresponding to the different $p$-norms in $\mathbb R^n$, where $n$ is the dimension of the space. For a vector $v\in \mathbb R^n$, the $p$-norm is ...

The title is a quote from this math podcast . I mentioned in the last post that Euclidean geometry arises as a special case of the geometry of inner product spaces. And yet, the only spaces that are ’tangible’ to us humans are the $1$, $2$, and $3$ dimensional Euclidean spaces. No other inner product spaces feel nearly as intuitive. In this post I’m showing how the Pythagoras theorem is a special case of a more general feature of inner product spaces. Contents of the last post are pre-requisites for this one. ...

This is an explainer on norms, metrics, and inner products, and their relationships to each other. Norms A norm is any real-valued function $\lVert{}\cdot{}\rVert$ (taking the elements of a corresponding vector space as its arguments), which has the following properties: It is nonnegative, and $0$ only at the ‘zero element’ (for e.g., at the origin of $\mathbb R^n$). $\lVert \alpha x \rVert = |\alpha| \lVert x \rVert$ for any scalar $\alpha$. ...

The title is a reference to The Unreasonable Effectiveness of Mathematics in the Natural Sciences , a very popular paper by Eugene Wigner which explores how mathematics is unreasonably effective at not only explaining, but also predicting scientific phenomena. I had a similar question about the number $2$ which repeatedly shows up in engineering and science, specifically in the form of the $2$-norm of a vector, and seems surprisingly effective at doing what it’s supposed to do. I asked my Estimation Theory instructor at Purdue why this was so, and he told me that I ask too many (but good) questions. I have since then accumulated a variety of answers for why the number $2$ is, in some sense, ✨special✨ During our journey through this post and the next, we will visit the central limit theorem, Gaussian distributions, and Euclidean geometry. ...

One of my motivations for starting a blog was Eugenia Cheng’s book The Joy of Abstraction 1. It’s a surprisingly accessible, gentle introduction to category theory, a topic that is usually only taught to graduate students in math. She compiled part of the book using notes from a class that she teaches at the Art Institute of Chicago, a testament to the aesthetic appreciation that one can expect to gain of category theory irrespective of their academic background! In this post, I will introduce the main ideas in category theory (as I best understand it) and show that it offers an elegant way of thinking about mathematics. ...

In my earlier post I suggested that there is no objective notion of logical truth, that whether a statement is ’true’ can depend on the system of truth that one is operating in. Here we will develop that argument further using the concept of axiomatic systems. This is a long one, but I’m excited to talk about it! Axiomatic Systems An axiomatization is an assignment of rules (axioms) such as “one plus one equals two,” making up an axiomatic system or a formal system. If you happen to be within that formal system, then you must follow all of the assigned rules. The choice of axioms dictates the degree of expressiveness that one has within the formal system. For instance, North Korea has a small number of state-approved haircuts that everyone must choose from, which does not sound like a very expressive system. ...

This is my first post! It discusses the question of whether spoken and written languages like English could be ’logical’ by design. I will break this post up into two parts. The first one does not require a mathematical background whatsoever, whereas the second touches on the concepts of axioms, theorems and proofs. Fallacies The word logic as it is used in everyday parlance refers to informal logic (as opposed to formal logic , which is instead a rigorous mathematical construct). You’ve probably come across logical fallacies such as the false dilemma , which goes something like this: ...