Rank and nullity when X is data

Introduction

Rank and nullity are often introduced as properties of an abstract matrix. In econometrics, however, the matrix \(X\) is usually a dataset. Its \(N\) rows are observations and its \(k\) columns are regressors. Thinking of \(X\) in this way gives rank and nullity direct empirical interpretations. Rank measures how many independent column patterns the regressors contain. Nullity measures how many directions in coefficient space the data cannot distinguish.

The main source of confusion is that \(X\) connects two spaces. Vectors in \(\mathbb{R}^N\) have one coordinate per observation, whereas vectors in \(\mathbb{R}^k\) have one coordinate per regressor. The column space and the residual vector live in the first space. Coefficients and the null space live in the second. The quantities \(N-r\) and \(k-r\) therefore count different kinds of freedom, even though both contain the same rank \(r\).

This post develops that distinction, first through a small geometric example and then through familiar econometric cases. It also explains why \(N-r\) is the residual degrees of freedom, how rank deficiency creates unidentified coefficients, and why near rank deficiency appears as multicollinearity in empirical work.

A data table and a linear map

Consider a dataset with \(N\) observations and \(k\) regressors,

\[ X = \begin{pmatrix} x_{11} & x_{12} & \cdots & x_{1k} \\ x_{21} & x_{22} & \cdots & x_{2k} \\ \vdots & \vdots & \ddots & \vdots \\ x_{N1} & x_{N2} & \cdots & x_{Nk} \end{pmatrix}. \]

Row \(i\), denoted \(x_i^T\), records the \(k\) regressor values for observation \(i\). It is a point in \(\mathbb{R}^k\). Column \(j\), denoted \(x_j\), records regressor \(j\) for all \(N\) observations. It is a vector in \(\mathbb{R}^N\). An intercept, for example, is the column \((1,\ldots,1)^T\) in observation space.

The same matrix also defines a map

\[ X:\mathbb{R}^k \longrightarrow \mathbb{R}^N, \qquad \beta \longmapsto X\beta. \]

The input \(\beta\) contains one coefficient per regressor. The output \(X\beta\) contains one fitted value per observation. Matrix multiplication therefore turns a proposed coefficient vector into an outcome pattern across the sample. This map is the link between the two spaces.

Space	One coordinate represents	Objects in the space
\(\mathbb{R}^N\)	one observation	columns of \(X\), \(y\), \(\hat y\), and residuals
\(\mathbb{R}^k\)	one regressor	rows of \(X\), \(\beta\), and coefficient contrasts

Keeping track of the meaning of one coordinate resolves much of the apparent paradox surrounding rank and nullity.

Dimension counts independent choices

The statement that a subspace has dimension \(r\) does not mean that only \(r\) coordinates can be nonzero. Dimension counts the number of independent numbers needed to select a vector in the subspace.

For example, the plane

\[ \{(a,b,c)\in\mathbb{R}^3:a+b+c=0\} \]

has dimension two. A point on the plane generally has three nonzero coordinates. Two coordinates can be chosen independently, after which the restriction fixes the third. Equivalently, every point on the plane can be constructed from two independent vectors. The plane is two-dimensional because it has two independent directions, not because it uses only two coordinate positions.

This interpretation applies directly to fitted values. If the attainable fitted-value vectors form an \(r\)-dimensional subspace of \(\mathbb{R}^N\), then \(r\) independent numbers suffice to select one of them. Every selected vector still contains a fitted value for each of the \(N\) observations.

Rank measures independent patterns in the data

The rank of \(X\) is

\[ r=\operatorname{rank}(X)\leq \min(N,k). \]

It can be defined as the number of linearly independent columns of \(X\). In data terms, \(r\) is the number of regressors that remain after all exact linear redundancies have been removed. If \(N>k\), full column rank means \(r=k\): no regressor can be written as an exact linear combination of the others in the observed sample.

Rank can equally be defined as the number of linearly independent rows. This equality of row rank and column rank connects two descriptions of the same dataset. The number of independent variables across observations equals the number of independent directions spanned by the observations in regressor space. Thus, a dataset with rank \(r\) contains \(r\) independent directions of linear variation whether it is examined by columns or by rows.

Exact dependence has familiar sources. An intercept together with indicators for every category produces the dummy-variable trap. A column for total expenditure together with exhaustive expenditure categories is also redundant. Duplicate variables, accounting identities, and saturated sets of fixed effects can create the same problem.

The column space contains all possible fitted outcomes

Write the columns of \(X\) as \(x_1,\ldots,x_k\). The column space is

\[ \operatorname{col}(X) = \left\{\beta_1x_1+\cdots+\beta_kx_k:\beta\in\mathbb{R}^k\right\} \subseteq \mathbb{R}^N. \]

It is the set of outcome patterns that can be constructed as linear combinations of the regressors. Since every fitted vector has the form \(\hat y=X\hat\beta\), the column space is precisely the set of fitted-value vectors that the model can produce. Its dimension is \(r\).

Take a small case with \(N=3\) observations and \(k=2\) linearly independent regressors. The columns \(x_1\) and \(x_2\) are two vectors in \(\mathbb{R}^3\), and their linear combinations form a plane through the origin. Figure 1 depicts that plane. The observed outcome \(y\) generally lies outside it. Ordinary least squares selects the point \(\hat y\) in the plane that is closest to \(y\). The residual \(e=y-\hat y\) is perpendicular to the plane.

Figure 1. With three observations and two independent regressors, \(\operatorname{col}(X)\) is a plane in observation space. OLS projects \(y\) onto this plane. The residual \(e=y-\hat y\) is orthogonal to both \(x_1\) and \(x_2\).

Every point on the plane is a triple \((\hat y_1,\hat y_2,\hat y_3)\). Two independent coefficients select a point, but the point has three coordinates. Changing either coefficient generally changes all three fitted values. A low-dimensional column space therefore does not identify a subset of observations that can be fitted while the others are left aside. It restricts the joint patterns across all observations that the model can produce.

This also explains why two design matrices with the same column space produce the same fitted values and residuals. Their parameterizations may differ, but they offer OLS the same set of candidate fitted vectors.

Residual degrees of freedom

The residual vector from least squares satisfies the normal equations

\[ X^Te=0. \]

It is orthogonal to every column of \(X\) and therefore lies in the left null space

\[ \mathcal N(X^T)=\{e\in\mathbb{R}^N:X^Te=0\}. \]

The column space has dimension \(r\), so its orthogonal complement has dimension \(N-r\). This gives the decomposition

\[ \mathbb{R}^N = \operatorname{col}(X)\oplus\mathcal N(X^T), \]

with dimensions \(r+(N-r)=N\). The fitted vector occupies the first subspace and the residual occupies the second.

A regression that only estimates a mean provides the smallest useful example. Let \(N=3\) and let \(X\) contain only an intercept. For

\[ y=(10,14,18)^T, \]

the fitted values and residuals are

\[ \hat y=(14,14,14)^T, \qquad e=(-4,0,4)^T. \]

The column space consists of vectors \((c,c,c)^T\). It has dimension one because the scalar \(c\) determines all three coordinates. The residual space consists of vectors satisfying

\[ e_1+e_2+e_3=0. \]

It has dimension two: any two residual coordinates can be selected and the restriction determines the third. This is the geometric source of the \(N-1\) denominator in the usual sample variance.

Now consider \(N=100\) and a full-column-rank matrix with \(k=4\). Fitted vectors lie in a four-dimensional subspace of \(\mathbb{R}^{100}\). Four coefficient values determine all 100 fitted values jointly. Residuals satisfy four independent orthogonality restrictions,

\[ x_1^Te=0,\quad x_2^Te=0,\quad x_3^Te=0,\quad x_4^Te=0, \]

so the residual space has dimension \(100-4=96\). The number 96 describes independent directions in which the outcome can depart from the model. It does not describe 96 observations that are freely fitted or ignored.

The phrase “96 free residual coordinates” requires some care. It means that a basis for the residual space contains 96 vectors, or equivalently that 96 coordinates can be used to parameterize a residual vector after a suitable choice of coordinates. It does not mean that any preassigned group of 96 observation-level residuals can always be chosen arbitrarily. That stronger coordinate statement depends on which rows of \(X\) are linearly independent.

For a realized dataset, both \(X\) and \(y\) are fixed, so the OLS residual is fixed as well. Residual degrees of freedom describe the dimension of the space in which the random residual could vary across repeated samples. Under the standard homoskedastic linear model, this is why \(N-r\), rather than a count of zero residuals, determines the denominator of the unbiased error-variance estimator:

\[ \hat\sigma^2=\frac{e^Te}{N-r}. \]

When \(X\) has full column rank, \(r=k\) and this becomes the familiar \(N-k\).

Nullity measures unidentified coefficient directions

The null space of \(X\) is

\[ \mathcal N(X)=\{v\in\mathbb{R}^k:Xv=0\}. \]

It lies in coefficient space. A vector \(v\) in this space is a combination of regressors that equals zero for every observation. Suppose the fourth column satisfies \(x_4=x_1+x_2\). Then

\[ X \begin{pmatrix} 1\\1\\0\\-1 \end{pmatrix} =x_1+x_2-x_4=0. \]

Every multiple of \((1,1,0,-1)^T\) also lies in the null space. Moving the coefficient vector in this direction has no effect on fitted values:

\[ X(\beta+t v)=X\beta+tXv=X\beta \]

for every scalar \(t\). The data can identify the fitted outcome while failing to identify a unique coefficient vector.

The rank-nullity theorem states

\[ \underbrace{\operatorname{rank}(X)}_{r} + \underbrace{\dim\mathcal N(X)}_{k-r} =k. \]

This is a statement about the domain of the map \(X:\mathbb{R}^k\to\mathbb{R}^N\). Of the \(k\) coefficient directions, \(r\) change the fitted values and \(k-r\) leave them unchanged. The latter directions form the null space. Its dimension, \(k-r\), is the number of independent coefficient directions that the sample cannot distinguish.

The row space supplies the complementary part of coefficient space:

\[ \mathbb{R}^k = \operatorname{row}(X)\oplus\mathcal N(X). \]

The rows of \(X\) are the observations viewed as points in regressor space. Their span has dimension \(r\). A direction in \(\mathcal N(X)\) is orthogonal to every row, so none of the observed data points extends in that direction. Consequently, the sample contains no variation that can separate coefficients along it.

This distinction also determines which coefficient combinations remain estimable under rank deficiency. A contrast \(c^T\beta\) is identified when \(c\) lies in the row space of \(X\), because it is then unchanged across all coefficient vectors that give the same fit. Individual coefficients need not be identified even when a substantively meaningful combination of them is.

The four subspaces of a data matrix

The two decompositions can be placed side by side. They concern different ambient spaces and answer different econometric questions.

Ambient space	Subspace	Dimension	Interpretation
\(\mathbb{R}^N\)	\(\operatorname{col}(X)\)	\(r\)	attainable fitted-value patterns
\(\mathbb{R}^N\)	\(\mathcal N(X^T)\)	\(N-r\)	residual directions orthogonal to all regressors
\(\mathbb{R}^k\)	\(\operatorname{row}(X)\)	\(r\)	coefficient directions explored by the observations
\(\mathbb{R}^k\)	\(\mathcal N(X)\)	\(k-r\)	coefficient directions invisible in the data

The two relevant dimension identities are therefore

\[ r+(N-r)=N \]

in observation space and

\[ r+(k-r)=k \]

in coefficient space. The first identity explains residual degrees of freedom. The second is the rank-nullity theorem for \(X\) and explains coefficient identification.

Interpolation and least squares use observations differently

The statement that a four-regressor model has four coefficient parameters sometimes leads to a different interpretation: perhaps the model exactly fits four observations and then determines fitted values for the remaining 96. That procedure is possible under an additional condition, but it is interpolation rather than least squares.

Select four observations and stack their rows in a \(4\times4\) matrix \(X_S\). Exact interpolation of their outcomes requires

\[ X_S\beta=y_S. \]

This system has the unique solution \(\beta=X_S^{-1}y_S\) if and only if \(X_S\) is invertible. The condition requires the four selected rows to be linearly independent. Distinct observations alone are insufficient. For example, with an intercept and a treatment indicator, two different control observations have identical regressor rows \((1,0)\) and cannot determine two coefficients.

Full column rank of the complete matrix \(X\) does not ensure that every selection of four rows is invertible. It does ensure that at least one selection of four rows is invertible. This follows from the equality of row rank and column rank. Coefficients obtained from such a subset can be used to calculate fitted values for all remaining observations, although those fitted values need not be close to their outcomes.

OLS instead uses all 100 observations to minimize

\[ \sum_{i=1}^{100}(y_i-x_i^T\beta)^2. \]

Typically, all 100 residuals are nonzero. The four coefficients are chosen as a joint compromise over the complete sample, and the resulting four normal equations impose four restrictions on the residual vector. No four observations receive a privileged role.

This comparison also clarifies the coordinate interpretation of 96 residual degrees of freedom. If \(X_S\) is an invertible submatrix formed from four rows, the orthogonality restrictions can determine the four residuals indexed by \(S\) from the other 96. A different invertible selection produces a different parameterization of the same residual space. The underlying dimension remains 96.

Singular normal equations

The least-squares first-order conditions are

\[ X^TX\hat\beta=X^Ty. \]

The matrix \(X^TX\) has the same rank and the same null space as \(X\). To see the null-space result, observe that

\[ v^TX^TXv=\lVert Xv\rVert^2. \]

Thus, \(X^TXv=0\) implies \(Xv=0\), and the converse follows immediately. When \(r<k\), the matrix \(X^TX\) is singular and the normal equations cannot select a unique coefficient vector. They can still select a unique fitted vector \(X\hat\beta\), because all least-squares solutions differ only by a vector in \(\mathcal N(X)\).

Software handles exact dependence in several ways. A regression routine may omit one redundant column, impose a normalization, or return a generalized-inverse solution. These choices can change the reported coefficients while leaving fitted values and residuals unchanged. The omitted category in a fixed-effects model is a standard example: changing the reference category changes the coefficient labels and values, while the model’s fitted outcomes remain the same.

When \(N<k\), rank satisfies \(r\leq N<k\), so \(k-r>0\) necessarily. There are more coefficient directions than the sample can identify. If the system can interpolate \(y\), infinitely many coefficient vectors may fit it perfectly. The minimum-norm least-squares solution sets the component in \(\mathcal N(X)\) to zero. Ridge regression also selects a unique coefficient vector by penalizing its length, while lasso applies a different selection rule through its absolute-value penalty.

Near dependence and multicollinearity

Exact rank deficiency is a sharp condition. Empirical datasets more often contain regressors that are almost linearly dependent. In exact arithmetic such a matrix may have full rank, yet some coefficient combinations remain weakly determined.

The singular value decomposition makes the distinction precise. Write

\[ X=UDV^T, \]

where the positive entries \(d_1,\ldots,d_r\) of \(D\) are the singular values. A zero singular value identifies an exact null-space direction. A small singular value identifies a direction \(v_j\) for which \(Xv_j\) is small: a large movement in coefficients produces only a small movement in fitted values.

Under homoskedastic errors and full column rank,

\[ \operatorname{Var}(\hat\beta\mid X) = \sigma^2(X^TX)^{-1} = \sigma^2V D^{-2}V^T. \]

Sampling variance in direction \(v_j\) is proportional to \(1/d_j^2\). A small singular value therefore produces an imprecisely estimated coefficient combination. Condition numbers and variance-inflation diagnostics measure aspects of this near dependence. Large standard errors in a collinear regression are the statistical counterpart of a nearly flat direction in the data cloud.

Centering changes the geometry in a predictable way. A column-centered data matrix has column sums equal to zero, so its columns lie in the \((N-1)\)-dimensional subspace orthogonal to the all-ones vector. Its rank can therefore be no larger than \(\min(N-1,k)\). In regressor space, centering moves the origin to the centroid of the data cloud, which makes its spanned directions describe variation around the sample means.

Implications for empirical specifications

Rank records which comparisons an empirical specification can support. With an intercept and a complete set of group indicators, one column must be omitted or an equivalent restriction must be imposed. In a panel regression with unit and time fixed effects, connectedness of the observation pattern determines the rank of the dummy-variable design. In an event-study specification, the omitted relative-time category supplies the normalization needed to express dynamic coefficients. In each case, the normalization selects a coordinate system for the identified fitted values.

The same logic applies when regressors have limited support. If a treatment indicator never varies within a fixed-effect group, its coefficient is absorbed by those fixed effects. If two policy variables always move together in the observed sample, the data cannot identify their separate effects without further restrictions or additional variation. These are empirical failures of independent variation, represented algebraically by a null-space direction.

Rank should therefore be read as a property of the realized design matrix and the comparisons it contains. A variable may be conceptually distinct from every other variable and still be linearly redundant in a particular sample. Conversely, full rank establishes only the absence of exact redundancy. It does not guarantee substantial independent variation or precise estimates, which is why singular values matter alongside the formal rank.

Conclusion

For an \(N\times k\) data matrix, rank and nullity organize two separate questions. In observation space, rank \(r\) is the dimension of the fitted-value patterns the regressors can generate, while \(N-r\) is the dimension available to residual variation. In coefficient space, the same rank \(r\) counts the directions visible in the data, while nullity \(k-r\) counts the coefficient directions that leave every fitted value unchanged.

The distinction gives a direct interpretation to familiar econometric results. Residual degrees of freedom equal \(N-r\) because OLS residuals lie in \(\mathcal N(X^T)\). Coefficients are unique only when \(k-r=0\) because vectors in \(\mathcal N(X)\) cannot be identified. Near-null directions produce multicollinearity because the sample changes very little when coefficients move along them. The relevant question in each case is which space contains the vector under discussion: one coordinate per observation, or one coordinate per regressor.