Rank and nullity when X is data

Introduction

Rank and nullity are often introduced as properties of an abstract matrix. In econometrics, however, the matrix \(X\) is usually a dataset. Its \(N\) rows are observations and its \(k\) columns are regressors. Thinking of \(X\) in this way gives rank and nullity direct empirical interpretations. Rank measures how many independent column patterns the regressors contain. Nullity measures how many directions in coefficient space the data cannot distinguish.

The main source of confusion is that \(X\) connects two spaces. Vectors in \(\mathbb{R}^N\) have one coordinate per observation, whereas vectors in \(\mathbb{R}^k\) have one coordinate per regressor. The column space and the residual vector live in the first space. Coefficients and the null space live in the second. The quantities \(N-r\) and \(k-r\) therefore count different kinds of freedom, even though both contain the same rank \(r\).

This post develops that distinction, first through a small geometric example and then through familiar econometric cases. It also explains why \(N-r\) is the residual degrees of freedom, how rank deficiency creates unidentified coefficients, and why near rank deficiency appears as multicollinearity in empirical work.

A data table and a linear map

Consider a dataset with \(N\) observations and \(k\) regressors,

\[ X = \begin{pmatrix} x_{11} & x_{12} & \cdots & x_{1k} \\ x_{21} & x_{22} & \cdots & x_{2k} \\ \vdots & \vdots & \ddots & \vdots \\ x_{N1} & x_{N2} & \cdots & x_{Nk} \end{pmatrix}. \]

Row \(i\), denoted \(x_i^T\), records the \(k\) regressor values for observation \(i\). It is a point in \(\mathbb{R}^k\). Column \(j\), denoted \(x_j\), records regressor \(j\) for all \(N\) observations. It is a vector in \(\mathbb{R}^N\). An intercept, for example, is the column \((1,\ldots,1)^T\) in observation space.

The same matrix also defines a map

\[ X:\mathbb{R}^k \longrightarrow \mathbb{R}^N, \qquad \beta \longmapsto X\beta. \]

The input \(\beta\) contains one coefficient per regressor. The output \(X\beta\) contains one fitted value per observation. Matrix multiplication therefore turns a proposed coefficient vector into an outcome pattern across the sample. This map is the link between the two spaces.

Space	One coordinate represents	Objects in the space
\(\mathbb{R}^N\)	one observation	columns of \(X\), \(y\), \(\hat y\), and residuals
\(\mathbb{R}^k\)	one regressor	rows of \(X\), \(\beta\), and coefficient contrasts

Keeping track of the meaning of one coordinate resolves much of the apparent paradox surrounding rank and nullity.

Dimension counts independent choices

The statement that a subspace has dimension \(r\) does not mean that only \(r\) coordinates can be nonzero. Dimension counts the number of independent numbers needed to select a vector in the subspace.

Equivalently, a subspace of \(\mathbb{R}^m\) defined by \(q\) independent linear restrictions has dimension \(m-q\). Count independent restrictions, not just written equations: an equation implied by the others removes no further freedom. In each example, we therefore need to identify the vector being restricted, where the restrictions come from, and how many are independent.

For example, the plane

\[ \{(a,b,c)\in\mathbb{R}^3:a+b+c=0\} \]

has dimension two. A point on the plane generally has three nonzero coordinates. Two coordinates can be chosen independently, after which the restriction fixes the third. Equivalently, every point on the plane can be constructed from two independent vectors. The plane is two-dimensional because it has two independent directions, not because it uses only two coordinate positions.

This interpretation applies directly to fitted values. If the attainable fitted-value vectors form an \(r\)-dimensional subspace of \(\mathbb{R}^N\), then \(r\) independent numbers suffice to select one of them. Every selected vector still contains a fitted value for each of the \(N\) observations.

Rank measures independent patterns in the data

The rank of \(X\) is

\[ r=\operatorname{rank}(X)\leq \min(N,k). \]

It can be defined as the number of linearly independent columns of \(X\). In data terms, \(r\) is the number of regressors that remain after all exact linear redundancies have been removed. If \(N>k\), full column rank means \(r=k\): no regressor can be written as an exact linear combination of the others in the observed sample.

Rank can equally be defined as the number of linearly independent rows. This equality of row rank and column rank connects two descriptions of the same dataset. The number of independent variables across observations equals the number of independent directions spanned by the observations in regressor space. Thus, a dataset with rank \(r\) contains \(r\) independent directions of linear variation whether it is examined by columns or by rows.

Exact dependence has familiar sources. An intercept together with indicators for every category produces the dummy-variable trap. A column for total expenditure together with exhaustive expenditure categories is also redundant. Duplicate variables, accounting identities, and saturated sets of fixed effects can create the same problem.

The column space contains all possible fitted outcomes

Write the columns of \(X\) as \(x_1,\ldots,x_k\). The column space is

\[ \operatorname{col}(X) = \left\{\beta_1x_1+\cdots+\beta_kx_k:\beta\in\mathbb{R}^k\right\} \subseteq \mathbb{R}^N. \]

It is the set of outcome patterns that can be constructed as linear combinations of the regressors. Since every fitted vector has the form \(\hat y=X\hat\beta\), the column space is precisely the set of fitted-value vectors that the model can produce. Its dimension is \(r\).

Take a small case with \(N=3\) observations and \(k=2\) linearly independent regressors. The columns \(x_1\) and \(x_2\) are two vectors in \(\mathbb{R}^3\), and their linear combinations form a plane through the origin. Figure 1 depicts that plane. The observed outcome \(y\) generally lies outside it. Ordinary least squares selects the point \(\hat y\) in the plane that is closest to \(y\). The residual \(e=y-\hat y\) is perpendicular to the plane.

Figure 1. With three observations and two independent regressors, \(\operatorname{col}(X)\) is a plane in observation space. OLS projects \(y\) onto this plane. The residual \(e=y-\hat y\) is orthogonal to both \(x_1\) and \(x_2\).

Every point on the plane is a triple \((\hat y_1,\hat y_2,\hat y_3)\). Two independent coefficients select a point, but the point has three coordinates. Changing either coefficient generally changes all three fitted values. A low-dimensional column space therefore does not identify a subset of observations that can be fitted while the others are left aside. It restricts the joint patterns across all observations that the model can produce.

For this example, choose a nonzero normal vector \(n\) perpendicular to both columns. Then

\[ \operatorname{col}(X)=\{z\in\mathbb{R}^3:n^Tz=0\}, \qquad \dim\operatorname{col}(X)=3-1=2. \]

This one restriction comes from constructing \(z\) from the two columns: \(n^T(\beta_1x_1+\beta_2x_2)=0\) for every coefficient choice. OLS additionally chooses coefficients whose residual satisfies

\[ \mathcal N(X^T)=\{e\in\mathbb{R}^3:x_1^Te=0,\ x_2^Te=0\}, \qquad \dim\mathcal N(X^T)=3-2=1. \]

These two independent restrictions come from minimizing squared residuals, as derived below. They are independent because the two columns are independent. In coefficient space, \(\operatorname{row}(X)=\mathbb{R}^2\) has zero restrictions. The set \(\mathcal N(X)=\{b\in\mathbb{R}^2:Xb=0\}=\{0\}\) has two independent restrictions: requiring a coefficient change to leave every fitted value unchanged forces both entries to zero.

This also explains why two design matrices with the same column space produce the same fitted values and residuals. Their parameterizations may differ, but they offer OLS the same set of candidate fitted vectors.

Residual degrees of freedom

OLS chooses \(\hat\beta\) to minimize \(L(\beta)=(y-X\beta)^T(y-X\beta)\). Differentiating with respect to the coefficients gives

\[ \nabla_\beta L(\hat\beta)=-2X^T(y-X\hat\beta)=0. \]

Thus the residual \(e=y-X\hat\beta\) satisfies the normal equations

\[ X^Te=0. \]

It is orthogonal to every column of \(X\) and therefore lies in the left null space

\[ \mathcal N(X^T)=\{e\in\mathbb{R}^N:X^Te=0\}. \]

There is one written equation per regressor, but only \(r\) independent restrictions: \(\operatorname{rank}(X^T)=r\). Any linear dependency among columns of \(X\) makes the corresponding residual equation redundant. These restrictions arise from the OLS fitting rule; they do not hold for arbitrary coefficient choices. Subtracting \(r\) independent restrictions from \(N\) residual coordinates gives dimension \(N-r\). This gives the decomposition

\[ \mathbb{R}^N = \operatorname{col}(X)\oplus\mathcal N(X^T), \]

with dimensions \(r+(N-r)=N\). The fitted vector occupies the first subspace and the residual occupies the second.

A regression that only estimates a mean provides the smallest useful example. Let \(N=3\) and let \(X\) contain only an intercept. For

\[ y=(10,14,18)^T, \]

the fitted values and residuals are

\[ \hat y=(14,14,14)^T, \qquad e=(-4,0,4)^T. \]

Using the same intercept for all observations imposes two independent fitted-value restrictions:

\[ \operatorname{col}(X) =\{z\in\mathbb{R}^3:z_1-z_2=0,\ z_2-z_3=0\} =\{(c,c,c)^T:c\in\mathbb{R}\}. \]

Its dimension is \(3-2=1\). The equation \(z_1=z_3\) follows from the other two and adds no restriction. OLS estimates \(c\) by \(\bar y\), so \(\sum_i e_i=\sum_i(y_i-\bar y)=0\). This produces one independent residual restriction:

\[ \mathcal N(X^T)=\{e\in\mathbb{R}^3:e_1+e_2+e_3=0\}, \qquad \dim\mathcal N(X^T)=3-1=2. \]

Any two residual coordinates can be selected and the restriction determines the third. An arbitrary intercept \(c\) instead gives \(\sum_i e_i=3(\bar y-c)\), which need not be zero. In coefficient space, the row space is \(\mathbb{R}\) with zero restrictions, while \(Xb=0\) reduces to the single independent restriction \(b=0\), giving nullity zero. For \(N\) observations, estimating the mean still imposes just one residual restriction, which gives the usual \(N-1\) residual degrees of freedom.

Now consider \(N=100\) and a full-column-rank matrix with \(k=4\). Fitted vectors lie in a four-dimensional subspace of \(\mathbb{R}^{100}\). Using the same four coefficients across observations imposes \(100-4=96\) independent restrictions on fitted values; the later worked example writes them explicitly. OLS minimization separately imposes four independent orthogonality restrictions on residuals, one for each independent column:

\[ x_1^Te=0,\quad x_2^Te=0,\quad x_3^Te=0,\quad x_4^Te=0, \]

so the residual space has dimension \(100-4=96\). The number 96 describes independent directions in which the outcome can depart from the model. It does not describe 96 observations that are freely fitted or ignored.

The phrase “96 free residual coordinates” requires some care. It means that a basis for the residual space contains 96 vectors, or equivalently that 96 coordinates can be used to parameterize a residual vector after a suitable choice of coordinates. It does not mean that any preassigned group of 96 observation-level residuals can always be chosen arbitrarily. That stronger coordinate statement depends on which rows of \(X\) are linearly independent.

For a realized dataset, both \(X\) and \(y\) are fixed, so the OLS residual is fixed as well. Residual degrees of freedom describe the dimension of the space in which the random residual could vary across repeated samples. Under the standard homoskedastic linear model, this is why \(N-r\), rather than a count of zero residuals, determines the denominator of the unbiased error-variance estimator:

\[ \hat\sigma^2=\frac{e^Te}{N-r}. \]

When \(X\) has full column rank, \(r=k\) and this becomes the familiar \(N-k\).

Nullity measures unidentified coefficient directions

The null space of \(X\) is

\[ \mathcal N(X)=\{v\in\mathbb{R}^k:Xv=0\}. \]

It lies in coefficient space. Its restrictions come from requiring a coefficient change to leave every fitted value unchanged: \(Xv=0\). There are \(N\) written equations, one per observation, but only \(r\) are independent because the rows have rank \(r\). Thus the dimension is \(k-r\).

For a concrete example, suppose \(k=4\), the first three columns are independent, and \(x_4=x_1+x_2\). Then \(Xv=(v_1+v_4)x_1+(v_2+v_4)x_2+v_3x_3\), so

\[ \begin{aligned} \mathcal N(X) &=\{v\in\mathbb{R}^4:v_1+v_4=0,\ v_2+v_4=0,\ v_3=0\}\\ &=\{t(1,1,0,-1)^T:t\in\mathbb{R}\}. \end{aligned} \]

The three independent columns force each of their weights to be zero, giving three independent restrictions and dimension \(4-3=1\). In particular,

\[ X \begin{pmatrix} 1\\1\\0\\-1 \end{pmatrix} =x_1+x_2-x_4=0. \]

Every multiple of \((1,1,0,-1)^T\) also lies in the null space. Moving the coefficient vector in this direction has no effect on fitted values:

\[ X(\beta+t v)=X\beta+tXv=X\beta \]

for every scalar \(t\). The data can identify the fitted outcome while failing to identify a unique coefficient vector.

The rank-nullity theorem states

\[ \underbrace{\operatorname{rank}(X)}_{r} + \underbrace{\dim\mathcal N(X)}_{k-r} =k. \]

This is a statement about the domain of the map \(X:\mathbb{R}^k\to\mathbb{R}^N\). Of the \(k\) coefficient directions, \(r\) change the fitted values and \(k-r\) leave them unchanged. The latter directions form the null space. Its dimension, \(k-r\), is the number of independent coefficient directions that the sample cannot distinguish.

The row space supplies the complementary part of coefficient space:

\[ \mathbb{R}^k = \operatorname{row}(X)\oplus\mathcal N(X). \]

The rows of \(X\) are the observations viewed as points in regressor space. Their span has dimension \(r\). A direction in \(\mathcal N(X)\) is orthogonal to every row, so none of the observed data points extends in that direction. Consequently, the sample contains no variation that can separate coefficients along it.

In the four-column example, the identity \(x_4=x_1+x_2\) holds in every observation row and therefore in every linear combination of rows. Hence

\[ \operatorname{row}(X)=\{c\in\mathbb{R}^4:c_4=c_1+c_2\}, \qquad \dim\operatorname{row}(X)=4-1=3. \]

This one restriction is also \(c^T(1,1,0,-1)^T=0\): row-space vectors must be perpendicular to the unidentified direction. Here \(r=3\), so fitted values satisfy \(N-3\) independent restrictions from lying in the span of the first three columns. OLS residuals satisfy three independent restrictions \(x_1^Te=x_2^Te=x_3^Te=0\); the fourth equation follows from \(x_4^Te=x_1^Te+x_2^Te\).

This distinction also determines which coefficient combinations remain estimable under rank deficiency. A contrast \(c^T\beta\) is identified when \(c\) lies in the row space of \(X\), because it is then unchanged across all coefficient vectors that give the same fit. Individual coefficients need not be identified even when a substantively meaningful combination of them is.

The four subspaces of a data matrix

The two decompositions can be placed side by side, with their restriction counts explicit. Let \(n_1,\ldots,n_{N-r}\) be a basis for \(\mathcal N(X^T)\) and \(v_1,\ldots,v_{k-r}\) a basis for \(\mathcal N(X)\). Then

\[ \operatorname{col}(X)=\{z\in\mathbb{R}^N:n_\ell^Tz=0\text{ for }\ell=1,\ldots,N-r\}, \]

\[ \operatorname{row}(X)=\{c\in\mathbb{R}^k:v_j^Tc=0\text{ for }j=1,\ldots,k-r\}. \]

These restrictions arise because column combinations are perpendicular to the left null space, and row combinations are perpendicular to the null space. Using bases makes the restrictions independent. Together with the defining equations for the two null spaces, this gives:

Vector and set	Source of restrictions	Independent restrictions	Dimension
\(z\in\operatorname{col}(X)\)	\(z\) must be a combination of columns	\(N-r\)	\(N-(N-r)=r\)
\(e\in\mathcal N(X^T)\)	\(X^Te=0\), imposed on residuals by OLS	\(r\)	\(N-r\)
\(c\in\operatorname{row}(X)\)	\(c\) must be a combination of rows	\(k-r\)	\(k-(k-r)=r\)
\(v\in\mathcal N(X)\)	\(Xv=0\), requiring unchanged fitted values	\(r\)	\(k-r\)

The two relevant dimension identities are therefore

\[ r+(N-r)=N \]

in observation space and

\[ r+(k-r)=k \]

in coefficient space. The first identity explains residual degrees of freedom. The second is the rank-nullity theorem for \(X\) and explains coefficient identification.

The four spaces in the 100-observation example

The statement that a four-regressor model has four coefficient parameters sometimes leads to a different interpretation: perhaps the model exactly fits four observations and then determines fitted values for the remaining 96. That procedure is possible under an additional condition, but it is interpolation rather than least squares.

Select four observations and stack their rows in a \(4\times4\) matrix \(X_S\). Exact interpolation of their outcomes requires

\[ X_S\beta=y_S. \]

This system has the unique solution \(\beta=X_S^{-1}y_S\) if and only if \(X_S\) is invertible. The condition requires the four selected rows to be linearly independent. Distinct observations alone are insufficient. For example, with an intercept and a treatment indicator, two different control observations have identical regressor rows \((1,0)\) and cannot determine two coefficients.

In that two-regressor example, fitting both controls requires \(\beta_0=y_{s_1}\) and \(\beta_0=y_{s_2}\). These equations are consistent only if \(y_{s_1}=y_{s_2}\), one compatibility restriction on the outcomes arising from identical regressor rows. When it holds, there is just one independent equation on the two coefficients: the intercept is fixed and the treatment coefficient remains free.

Full column rank of the complete matrix \(X\) does not ensure that every selection of four rows is invertible. It does ensure that at least one selection of four rows is invertible. This follows from the equality of row rank and column rank. Coefficients obtained from such a subset can be used to calculate fitted values for all remaining observations, although those fitted values need not be close to their outcomes.

To make the four spaces explicit, keep \(X\in\mathbb{R}^{100\times4}\) fixed with rank four. Let \(S\) index four linearly independent rows and let \(T\) index the remaining 96 rows. Write vectors in observation space as \((z_S,z_T)\), grouping the selected coordinates first. Define

\[ A=X_TX_S^{-1}\in\mathbb{R}^{96\times4}. \]

The column space is

\[ \begin{aligned} \operatorname{col}(X) &=\{X\beta:\beta\in\mathbb{R}^4\}\\ &=\{z\in\mathbb{R}^{100}:z_T=Az_S\}\\ &=\{(u,Au):u\in\mathbb{R}^4\}, \qquad \dim\operatorname{col}(X)=4. \end{aligned} \]

Here the four entries of \(u\) can be chosen independently, and they determine the other 96 entries. The 96 restrictions \(z_T=Az_S\) come directly from using the same four coefficients for every observation: \(z_S=X_S\beta\) implies \(\beta=X_S^{-1}z_S\), and hence \(z_T=X_T\beta=Az_S\). They are independent because each equation fixes a different coordinate of \(z_T\) once \(z_S\) is given, leaving dimension \(100-96=4\). This holds for any coefficient choice, however it was obtained.

The left null space is

\[ \begin{aligned} \mathcal N(X^T) &=\{e\in\mathbb{R}^{100}:X_S^Te_S+X_T^Te_T=0\}\\ &=\{e\in\mathbb{R}^{100}:e_S=-A^Te_T\}\\ &=\{(-A^Tv,v):v\in\mathbb{R}^{96}\}, \qquad \dim\mathcal N(X^T)=96. \end{aligned} \]

This is exactly what 96 OLS residual degrees of freedom means in set notation: each freely chosen \(v\in\mathbb{R}^{96}\) determines one residual vector in this space. Its entries on \(T\) equal \(v\), and the four orthogonality equations determine its entries on \(S\) as \(-A^Tv\). These equations come from the least-squares first-order conditions. They are independent because \(X\) has four independent columns, leaving dimension \(100-4=96\). The formula follows by multiplying \(X_S^Te_S=-X_T^Te_T\) by \((X_S^T)^{-1}\). For fixed \(X\), this set contains all possible OLS residual vectors as \(y\) varies; a particular \(y\) selects just one.

The two spaces in coefficient space are

\[ \operatorname{row}(X) =\left\{\sum_{i=1}^{100}a_i x_i:a\in\mathbb{R}^{100}\right\} =\mathbb{R}^4, \qquad \dim\operatorname{row}(X)=4, \]

where \(x_i\) is the transpose of observation row \(x_i^T\), and

\[ \mathcal N(X) =\{b\in\mathbb{R}^4:x_i^Tb=0\text{ for every }i=1,\ldots,100\} =\{0\}, \qquad \dim\mathcal N(X)=0. \]

The four independent rows indexed by \(S\) already span \(\mathbb{R}^4\), so membership in the row space imposes zero restrictions and its dimension is \(4-0=4\). For the null space, \(Xb=0\) requires a coefficient change to leave all fitted values unchanged. Although this gives 100 written equations, just four are independent: \(X_Sb=0\) forces \(b=0\), which also satisfies the other 96 equations. Its dimension is therefore \(4-4=0\). These four spaces depend only on \(X\); choosing interpolation instead of OLS does not change them.

What changes is the set containing the residuals. Interpolation sets \(\beta=X_S^{-1}y_S\), giving

\[ \hat y=(y_S,Ay_S), \qquad e=y-\hat y=(0,y_T-Ay_S). \]

As \(y\) varies, these residuals range over

\[ \mathcal E_S =\{e\in\mathbb{R}^{100}:e_S=0\} =\{(0,v):v\in\mathbb{R}^{96}\}, \qquad \dim\mathcal E_S=96. \]

Here the four restrictions come from exactly matching the four selected outcomes. Each sets a different residual coordinate to zero, so they are independent and leave dimension \(100-4=96\). The remaining 96 residual coordinates can vary independently as \(y_T\) varies with \(y_S\) held fixed. This space generally differs from \(\mathcal N(X^T)\): interpolation requires \(e_S=0\), whereas OLS requires \(e_S=-A^Te_T\).

Simply assigning four arbitrary coefficient values still places \(X\beta\) in the column space, with the fitted-value restrictions \(z_T=Az_S\). It does not by itself impose either \(e_S=0\) or \(X^Te=0\) on \(e=y-X\beta\). If \(\beta\) is held fixed and \(y\) varies freely, the residual can be any vector in \(\mathbb{R}^{100}\). If both \(\beta\) and \(y\) are fixed, there is just one residual vector. The 96-dimensional residual spaces above arise from the specific fitting rule and its four restrictions.

Singular normal equations

The least-squares first-order conditions are

\[ X^TX\hat\beta=X^Ty. \]

The matrix \(X^TX\) has the same rank and the same null space as \(X\). To see the null-space result, observe that

\[ v^TX^TXv=\lVert Xv\rVert^2. \]

Thus, \(X^TXv=0\) implies \(Xv=0\), and the converse follows immediately. When \(r<k\), the matrix \(X^TX\) is singular and the normal equations cannot select a unique coefficient vector. They can still select a unique fitted vector \(X\hat\beta\), because all least-squares solutions differ only by a vector in \(\mathcal N(X)\).

Viewed as equations in \(\hat\beta\), the \(k\) normal equations contain only \(r\) independent restrictions, leaving \(k-r\) free coefficient directions. If \(\hat\beta_0\) is one solution and \(v_1,\ldots,v_{k-r}\) form a basis for \(\mathcal N(X)\), the complete solution set is

\[ \left\{\hat\beta_0+\sum_{j=1}^{k-r}t_jv_j:t\in\mathbb{R}^{k-r}\right\}. \]

Selecting one member without changing the fit requires \(k-r\) additional linear normalizations that determine these \(t_j\) uniquely. Their independence must hold along the null-space directions; otherwise some ambiguity remains. These are choices made by the analyst or software, not extra information supplied by the observations. In the example \(x_4=x_1+x_2\) with the first three columns independent, the single normalization \(\beta_4=0\) suffices: moving along \((1,1,0,-1)^T\) can always satisfy it without changing fitted values.

Software handles exact dependence in several ways. A regression routine may omit one redundant column, impose a normalization, or return a generalized-inverse solution. These choices can change the reported coefficients while leaving fitted values and residuals unchanged. The omitted category in a fixed-effects model is a standard example: changing the reference category changes the coefficient labels and values, while the model’s fitted outcomes remain the same.

When \(N<k\), rank satisfies \(r\leq N<k\), so \(k-r>0\) necessarily. If exact interpolation is possible, \(X\beta=y\) supplies \(r\) independent restrictions on \(k\) coefficients, leaving a solution set of dimension \(k-r\). Existence itself requires \(y\in\operatorname{col}(X)\): the outcome must satisfy the \(N-r\) compatibility restrictions \(n_\ell^Ty=0\) defined above. With full row rank, \(r=N\), there are zero compatibility restrictions and every \(y\) can be interpolated.

The minimum-norm least-squares solution selects one coefficient vector by imposing \(v_j^T\beta=0\) for \(j=1,\ldots,k-r\). These \(k-r\) independent normalizations remove the null-space component, which adds length without affecting the fit. Ridge and lasso instead add penalties to the objective and can change the fitted values. Their penalties do not supply a fixed count of additional linear equality restrictions of this kind; in particular, their residuals need not satisfy the OLS equations \(X^Te=0\).

Near dependence and multicollinearity

Exact rank deficiency is a sharp condition. Empirical datasets more often contain regressors that are almost linearly dependent. In exact arithmetic such a matrix may have full rank, yet some coefficient combinations remain weakly determined. For a full-column-rank matrix, near dependence leaves the exact counts unchanged: \(N-k\) restrictions on attainable fitted vectors, \(k\) OLS residual restrictions, zero row-space restrictions, and \(k\) independent equations in \(Xv=0\). It introduces no additional exact equality.

The singular value decomposition makes the distinction precise. Write

\[ X=UDV^T, \]

where the positive entries \(d_1,\ldots,d_r\) of \(D\) are the singular values. A zero singular value identifies an exact null-space direction. A small singular value identifies a direction \(v_j\) for which \(Xv_j\) is small: a large movement in coefficients produces only a small movement in fitted values.

Under homoskedastic errors and full column rank,

\[ \operatorname{Var}(\hat\beta\mid X) = \sigma^2(X^TX)^{-1} = \sigma^2V D^{-2}V^T. \]

Sampling variance in direction \(v_j\) is proportional to \(1/d_j^2\). A small singular value therefore produces an imprecisely estimated coefficient combination. Condition numbers and variance-inflation diagnostics measure aspects of this near dependence. Large standard errors in a collinear regression are the statistical counterpart of a nearly flat direction in the data cloud.

Centering changes the geometry in a predictable way. Subtracting each column’s sample mean gives \(\sum_i(x_{ij}-\bar x_j)=0\). Every column of the centered matrix, and every linear combination \(z\) of those columns, therefore satisfies the single independent restriction

\[ \sum_{i=1}^N z_i=0. \]

This restriction comes from preprocessing the regressors, independently of any fitting rule. Its solution space has dimension \(N-1\), so the centered matrix has rank at most \(\min(N-1,k)\). If its actual rank is \(r\), its column space has \(N-r\) independent restrictions in total; centering explicitly accounts for one, and any others depend on the columns’ span. The centering equation restricts fitted vectors and does not by itself require residuals to sum to zero. In regressor space, centering moves the origin to the centroid of the data cloud, which makes its spanned directions describe variation around the sample means.

Implications for empirical specifications

Rank records which comparisons an empirical specification can support. With an intercept and a complete set of group indicators, one column must be omitted or an equivalent restriction must be imposed. In a panel regression with unit and time fixed effects, connectedness of the observation pattern determines the rank of the dummy-variable design. In an event-study specification, the omitted relative-time category supplies the normalization needed to express dynamic coefficients. In each case, the normalization selects a coordinate system for the identified fitted values.

The same logic applies when regressors have limited support. If a treatment indicator never varies within a fixed-effect group, its coefficient is absorbed by those fixed effects. If two policy variables always move together in the observed sample, the data cannot identify their separate effects without further restrictions or additional variation. These are empirical failures of independent variation, represented algebraically by a null-space direction.

Rank should therefore be read as a property of the realized design matrix and the comparisons it contains. A variable may be conceptually distinct from every other variable and still be linearly redundant in a particular sample. Conversely, full rank establishes only the absence of exact redundancy. It does not guarantee substantial independent variation or precise estimates, which is why singular values matter alongside the formal rank.

Conclusion

For an \(N\times k\) data matrix, rank and nullity organize two separate questions. In observation space, rank \(r\) is the dimension of the fitted-value patterns the regressors can generate, while \(N-r\) is the dimension available to residual variation. In coefficient space, the same rank \(r\) counts the directions visible in the data, while nullity \(k-r\) counts the coefficient directions that leave every fitted value unchanged.

The distinction gives a direct interpretation to familiar econometric results. Residual degrees of freedom equal \(N-r\) because OLS residuals lie in \(\mathcal N(X^T)\). Coefficients are unique only when \(k-r=0\) because vectors in \(\mathcal N(X)\) cannot be identified. Near-null directions produce multicollinearity because the sample changes very little when coefficients move along them. The relevant question in each case is which space contains the vector under discussion: one coordinate per observation, or one coordinate per regressor.