Unipotence of a double commutator with transvections

Abstract

We prove the statement that for every $\sigma\in GL_{n}(\mathbb{R})$ with $n\geq 3$, there exist transvections $\tau_{1},\tau_{2}\in GL_{n}(\mathbb{R})$ such that $\left(\big[[\sigma,\tau_{1}],\tau_{2}\big]-I\right)^{2}=0$. In particular, $\big[[\sigma,\tau_{1}],\tau_{2}\big]$ is unipotent. The proof relies on the existence of a one- or two-dimensional invariant subspace for a real endomorphism.

Keywords: transvections; unipotent matrices; invariant subspaces; general linear groups.

1 Introduction

The following problem from 1986, recorded as Problem 10.46 in the Kourovka Notebook [2, Problem 10.46], is attributed to V. M. Petechuk: for every $\sigma\in GL_{n}(\mathbb{R})$, $n\geq 3$, do there exist transvections $\tau_{1},\tau_{2}$ such that $[[\sigma,\tau_{1}],\tau_{2}]$ is unipotent? Theorem 2.1 answers this question affirmatively, and in fact proves the stronger assertion that $\left(\big[[\sigma,\tau_{1}],\tau_{2}\big]-I\right)^{2}=0$. For background on the terminology and standard concepts used here, including transvections and unipotent elements, see, for example, [3].

Let $V=\mathbb{R}^{n}$ with $n\geq 3$, and let $\sigma\in GL(V)$. Our goal is to construct two transvections $\tau_{1},\tau_{2}\in GL(V)$ such that the iterated commutator $\big[[\sigma,\tau_{1}],\tau_{2}\big]$ differs from the identity by an operator whose square is zero. The main ingredient is the standard fact that every endomorphism of a finite-dimensional real vector space admits a nonzero invariant subspace of dimension one or two [1, Corollary 7.4.3]. Applying this fact to $\sigma$ yields a $\sigma$-invariant subspace $W$ with $\dim W\in\{1,2\}$. We then choose $\tau_{1}$ so that the first commutator $[\sigma,\tau_{1}]$ induces the identity on the quotient $V/W$; this ensures that $[\sigma,\tau_{1}]-\mathrm{id}$ has image contained in $W$, and hence rank at most two. Since $n\geq 3$, the inequality $\operatorname{rank}([\sigma,\tau_{1}]-I)\leq 2<n$ implies that $[\sigma,\tau_{1}]$ fixes a nonzero vector. This fixed vector is then used to construct a second transvection $\tau_{2}$, chosen so that $\left(\big[[\sigma,\tau_{1}],\tau_{2}\big]-I\right)^{2}=0$.

Throughout, $GL_{n}(\mathbb{R})$ denotes the group of invertible $n\times n$ real matrices; we identify $V=\mathbb{R}^{n}$ and regard $GL_{n}(\mathbb{R})$ as the group of invertible linear maps $V\to V$. For $g,h$ in a group $G$, the commutator is $[g,h]:=ghg^{-1}h^{-1}$.

Definition 1.1.

For $u\in V$ and $f\in V^{*}$, let $u\otimes f$ denote the rank-one (or zero) linear operator on $V$ defined by

$$(u\otimes f)(x):=f(x)\,u,\quad x\in V.$$

The notation satisfies some elementary identities, which are used in Section 2. We state them here for later reference.

Lemma 1.2.

For $u,v\in V$, $f,g\in V^{*}$, and $B\in GL(V)$, the following identities hold:

• $\ B(u\otimes f)B^{-1}=(Bu)\otimes(f\circ B^{-1})$.

• $\ (u\otimes f)(v\otimes g)=f(v)(u\otimes g)$.

• $u\otimes f=0$ if and only if $u=0$ or $f=0$.

We now recall the definition of a transvection.

Definition 1.3.

Given nonzero $f\in V^{*}$ and nonzero $u\in V$ with $f(u)=0$, the transvection associated with $f$ and $u$ is the linear map $\tau=I+u\otimes f$, that is, $\tau(x)=x+f(x)u$.

For a transvection $\tau$ as in Definition 1.3, Lemma 1.2(ii) gives

$$(u\otimes f)^{2}=f(u)(u\otimes f)=0.$$

Hence, $\tau$ is invertible with inverse $\tau^{-1}=I-u\otimes f$, that is, $\tau^{-1}(x)=x-f(x)u,$ and hence $\tau\in GL(V)$.

Definition 1.4.

An element $A\in GL_{n}(\mathbb{R})$ is unipotent if $A-I$ is nilpotent, that is, $(A-I)^{k}=0$ for some integer $k\geq 1$.

2 Main result

We now state the theorem announced in the introduction; its proof follows the strategy outlined above, phrased throughout in terms of the rank-one operator notation $u\otimes f$.

Theorem 2.1.

Let $n\geq 3$ and let $\sigma\in GL_{n}(\mathbb{R})$. Then there exist transvections $\tau_{1},\tau_{2}\in GL_{n}(\mathbb{R})$ such that

$$\left(\big[[\sigma,\tau_{1}],\tau_{2}\big]-I\right)^{2}=0.$$

In particular, $\big[[\sigma,\tau_{1}],\tau_{2}\big]$ is unipotent.

Proof.

Every endomorphism of a finite-dimensional real vector space admits a nonzero invariant subspace of dimension one or two [1, Corollary 7.4.3]. Applying this to $\sigma$, there exists a subspace $W\subseteq V$ with $1\leq\dim W\leq 2$ and $\sigma(W)\subseteq W$. Since $\sigma$ is invertible, $\sigma|_{W}$ is injective, and since $W$ is finite-dimensional, it follows that $\sigma|_{W}:W\to W$ is an isomorphism [4, Theorem 3.65]. Hence, $\sigma^{-1}(W)=W$.

Choose $u_{1}\in W\setminus{0}$ and a nonzero functional $f_{1}\in V^{*}$ with $f_{1}(u_{1})=0$, which is possible since the annihilator of $u_{1}$ in $V^{*}$ has dimension $n-1\geq 2$, and set,

$$\tau_{1}:=I+u_{1}\otimes f_{1},\quad\tau_{1}^{-1}=I-u_{1}\otimes f_{1}.$$

Since $u_{1}\neq 0$ and $f_{1}\neq 0$, Lemma 1.2(iii) gives $u_{1}\otimes f_{1}\neq 0$, so $\tau_{1}$ is a transvection; moreover, since $u_{1}\otimes f_{1}$ has image $\operatorname{span}(u_{1})\subseteq W$,

$$(\tau_{1}-I)(V)=(u_{1}\otimes f_{1})(V)\subseteq W.$$ (1)

Define

$$A:=[\sigma,\tau_{1}]=\sigma\tau_{1}\sigma^{-1}\tau_{1}^{-1}.$$

Hence, $\sigma$ and $\sigma^{-1}$ induce mutually inverse automorphisms $\overline{\sigma}$ and $\overline{\sigma}^{-1}$ on $V/W$. By (1), both $\tau_{1}$ and $\tau_{1}^{-1}$ induce the identity on $V/W$. Consequently,

$$\overline{A}=\overline{\sigma}\,I\,\overline{\sigma}^{-1}\,I=I.$$

Therefore,

$$(A-I)(V)\subseteq W,\quad\operatorname{rank}(A-I)\leq\dim W\leq 2.$$ (2)

By rank–nullity, $\dim\ker(A-I)=n-\operatorname{rank}(A-I)\geq n-2\geq 1$. Hence, there exists $u\in V\setminus\{0\}$ such that

$$Au=u,$$ (3)

and since $A$ is invertible, (3) also gives

$$A^{-1}u=u.$$ (4)

Choose a nonzero functional $f\in V^{*}$ with $f(u)=0$, and set

$$\tau_{2}:=I+u\otimes f,\quad\tau_{2}^{-1}=I-u\otimes f.$$

As $u\neq 0$ and $f\neq 0$, Lemma 1.2(iii) gives $u\otimes f\neq 0$, so $\tau_{2}$ is a transvection. We now compute $[A,\tau_{2}]=A\tau_{2}A^{-1}\tau_{2}^{-1}$ using the operator identities for $\otimes$ given in Lemma 1.2. By Lemma 1.2(i) applied to $B=A$, together with (3),

$$A(u\otimes f)A^{-1}=(Au)\otimes(f\circ A^{-1})=u\otimes(f\circ A^{-1}),$$

and hence

$$A\tau_{2}A^{-1}=A(I+u\otimes f)A^{-1}=I+u\otimes(f\circ A^{-1}).$$

Therefore,

$$[A,\tau_{2}]=\bigl(I+u\otimes(f\circ A^{-1})\bigr)\bigl(I-u\otimes f\bigr)=I+u\otimes(f\circ A^{-1})-u\otimes f-(f\circ A^{-1})(u)\,(u\otimes f),$$

where the last term vanishes because, by (4),

$$(f\circ A^{-1})(u)=f(A^{-1}u)=f(u)=0.$$

Hence,

$$[A,\tau_{2}]=I+u\otimes h,\quad h:=f\circ A^{-1}-f.$$ (5)

Moreover, by (4),

$$h(u)=f(A^{-1}u)-f(u)=f(u)-f(u)=0.$$ (6)

From (5),

$$[A,\tau_{2}]-I=u\otimes h,$$

so, by Lemma 1.2(ii) and (6),

$$([A,\tau_{2}]-I)^{2}=(u\otimes h)^{2}=h(u)\,(u\otimes h)=0.$$

Hence $[A,\tau_{2}]$ is unipotent.

Finally, since $A=[\sigma,\tau_{1}]$, we have $\big[[\sigma,\tau_{1}],\tau_{2}\big]=[A,\tau_{2}]$, and therefore

$$\left(\big[[\sigma,\tau_{1}],\tau_{2}\big]-I\right)^{2}=0.$$

∎

Use of LLMs

We used Claude and ChatGPT to verify proofs, identify errors, and assist with formatting. The author takes full responsibility for the manuscript.

Research Funding

The University of Pretoria supported this research through the Research Development Programme (RDP).