#### by Richard Palais

#### What *is* Hilbert’s Fifth Problem?

As
Hilbert
stated it in his lecture delivered before the International
Congress of Mathematicians in Paris in 1900
[e2],
the Fifth Problem is
linked to
Sophus Lie’s
theory of transformation groups
[e1],
i.e., Lie
groups acting as groups of transformations on manifolds. The “groups”
that Lie dealt with were really just neighborhoods of the identity in
what we now call a Lie group, and his group actions were defined only
locally, but we will ignore such local versus global considerations in
what follows. However, it was crucial to the techniques that Lie used
that his manifolds should be analytic and that both the group law and
the functions defining the action of the group on the manifold should
be analytic, that is, given by convergent power series. For Lie, who
applied his theory to such things as studying the symmetries of
differential equations, the analyticity assumptions were natural
enough. But Hilbert wanted to use Lie’s theory as part of his logical
foundations of geometry, and for this purpose Hilbert felt that
analyticity was unnatural and perhaps superfluous. So Hilbert asked if
analyticity could be dropped in favor of mere continuity. More
precisely, if one only assumed a priori that the group __\( G \)__ was a locally
Euclidean topological group,
that the manifold __\( M \)__ was a topological manifold,
and that the action of __\( G \)__ on __\( M \)__ was continuous,
could one nevertheless always choose local coordinates
in __\( G \)__ and __\( M \)__ so that both the group operations
and the action became analytic when expressed in
these coordinates? We shall speak of the problem
in this generality as the unrestricted Hilbert Fifth
Problem. The restricted problem is the important
special case in which __\( G = M \)__ and the action is left
translation. Asking whether we can always find
analytic coordinates in the restricted problem is
clearly the same as asking whether a locally Euclidean
group is necessarily a Lie group.

#### Counterexamples

It turned out that there are many — and in fact many different
kinds of — counterexamples to the unrestricted Hilbert Fifth
Problem. Perhaps the first published counterexample, due to
R. H. Bing
[e15],
is an action of __\( \mathbf{Z}^2 \)__ on __\( \mathbf{S}^3 \)__ whose
fixed-point set is the Alexander Horned Sphere __\( \Sigma \)__. Now __\( \Sigma \)__
is not “tamely embedded” in __\( \mathbf{S}^3 \)__, meaning that there are
points where it is impossible to choose coordinates so that locally
__\( \Sigma \)__ looks like the usual embedding of __\( \mathbf{R}^2 \)__ in
__\( \mathbf{R}^3 \)__. If the action were even differentiable in some
suitable coordinates, then it is easy to see that the fixed-point set
would in fact be tamely embedded. (For an even more bizarre type of
counterexample, recall that in 1960
M. Kervaire
[e24]
constructed a
topological manifold that did not admit any differentiable structure,
providing what can be considered a counterexample even for the case
when __\( G \)__ is the trivial group.)

One could make a case that these examples are “monsters” that could
have been ruled out if Hilbert had phrased his statement of the Fifth
Problem more carefully. But there is a more serious kind of
counterexample that is so elementary that it makes one wonder how much
thought Hilbert had given to the Fifth Problem before proposing it.
Here is a particularly elementary example, due to
Montgomery
and
Zippin
[e22],
with __\( G = \mathbf{R} \)__, the additive group of the real
numbers, and __\( M = \mathbf{C} \)__, the complex plane. Let __\( f \)__ be a
continuous real-valued function defined on the positive real axis, and
define the action __\( \phi :\mathbf{R} \times \mathbf{C} \to \mathbf{C} \)__
by __\( \phi(t , r e^{i \theta} ) := r e^{i (\theta +f (r )t)} \)__ . (In
words, __\( \phi \)__ is a one-parameter group of homeomorphisms of the plane
that rotates each circle centered at the origin into itself, the
circle of radius __\( r \)__ being rotated with angular velocity __\( f (r ) \)__.) If
we choose __\( f (r ) \)__ to equal 1 for __\( r \leq 1 \)__ and 0 for __\( r \geq 2 \)__, the
action is the standard one-parameter group of rotations of
__\( \mathbf{C} \)__ inside the unit disk and is trivial outside the disk of
radius 2, so by the Principle of Analytic Continuation, this action
cannot be made analytic in any coordinate system. What is worse, we
can choose __\( f \)__ to have these properties and also be smooth (meaning
__\( C^{\infty} \)__), so we see that even if we assume a priori that the
action of a Lie Group on a manifold is smooth, it does not follow that
it can be made analytic!

After these counterexamples to the unrestricted Hilbert Fifth Problem became known, a tacit understanding grew up to interpret “the Fifth Problem” as referring to the restricted version: Is every locally Euclidean group a Lie group? and we shall follow this convention below.

#### Early history of the Fifth Problem

It was fairly easy to settle the one-dimensional case. The only
(paracompact) connected manifolds of dimension one are the real line,
__\( \mathbf{R} \)__, and the circle, __\( \mathbf{S}^1 \)__, and both of course are
Lie groups. In 1909
L. E. J. Brouwer
[e3]
showed that a topological group that is
homeomorphic to either of these is in fact isomorphic to it as a
topological group. Using results from Brouwer’s paper,
B. Kerékjártó
[e6]
settled the two-dimensional case in 1931. There seems to have
been little if any published work on the Fifth Problem between the
papers of Brouwer and Kerékjártó, but that is not too surprising
considering that much of the modern mathematical infrastructure
required for a rigorous discussion of topological groups and the Fifth
Problem became available only after a 1926 paper by
O. Schreier
[e4].
The three-dimensional and four-dimensional cases of the Fifth Problem
were settled much later, by Montgomery
[e12]
in 1948 and by Montgomery
and Zippin
[e17]
in 1952.

The first major breakthrough in the general theory came in 1933, when J. von Neumann [e8], using the recently discovered Haar [e7] measure, extended the Peter–Weyl Theorem [e5] to general compact groups and used it to settle the Fifth Problem in the affirmative for compact groups. We will sketch a proof of von Neumann’s theorem below. Several years later, building on von Neumann’s work, Pontryagin [e9] settled the abelian case, and Chevalley [e10] the solvable case.

#### The no small subgroups (NSS) condition

The first time I encountered the phrase “group
without small subgroups” I wondered what kind
of subgroup a “small” one could possibly be. Of
course, what the phrase means is a topological
group without *arbitrarily* small subgroups, i.e.,
one having a neighborhood of the identity that
includes no subgroup except the trivial group. We
shall follow
Kaplansky
[e25]
and call this the NSS
Condition and a group satisfying it an NSS group.
Since NSS may seem a little contrived, here is a
brief discussion of the “why and how” of its use
in solving the Fifth Problem.

It turns out to be difficult to draw useful conclusions about a topological group from the assumption that it is locally Euclidean. So the strategy used for settling the Fifth Problem was to look for a more group-oriented “bridge condition” and use it in a two-pronged attack: on the one hand show that a topological group that satisfies this condition is a Lie group, and on the other show that a locally Euclidean group satisfies the condition. If these two propositions can be proved, then the positive solution of the Fifth Problem follows — and even a little more.

As you may have guessed, NSS turned out to be ideally suited to play
the role of the bridge. In retrospect this is not entirely surprising.
A powerful but relatively elementary property of Lie groups is the
existence of so-called canonical coordinates, or equivalently the fact
that the exponential map is a diffeomorphism of a neighborhood of zero
in the Lie algebra onto a neighborhood __\( U \)__ of the identity in the group
(see below). Since a line through the origin in the Lie algebra maps
to a one-parameter subgroup of the group, it follows that such a __\( U \)__
contains no nontrivial subgroup and hence that Lie groups satisfy NSS.

Starting in the late 1940s Gleason
[1],1
Montgomery
[e16],
and
Iwasawa
[e13]
made several solid advances related to the Fifth Problem. This led
in 1952 to a satisfying denouement to the story of the Fifth Problem,
with Gleason and Montgomery–Zippin carrying out the above two-pronged
attack. First Gleason
[3]
proved that a locally compact group
satisfying NSS is a Lie group, and then immediately afterwards
Montgomery and Zippin
[e17]
used Gleason’s result to prove inductively
that locally Euclidean groups of any dimension satisfy NSS. Their two
papers appeared together in the same issue of the *Annals of
Mathematics*, and at that point one knew that for locally compact
topological groups:
__\[
\text{Locally Euclidean } \Longleftrightarrow \text{NSS } \Longleftrightarrow \text{Lie}.
\]__
(Actually, the above is not quite the full story; Gleason assumed a
weak form of finite dimensionality in his original argument that NSS
implies Lie, but shortly thereafter
Yamabe
[e20]
showed that finite
dimensionality was not needed in the proof.)

#### Cartan’s theorem

Here is a quick sketch of how the proof of the Fifth Problem for a
compact NSS group __\( G \)__ follows. Let __\( \mathcal{H} \)__ denote the Hilbert space __\( L^2 (G) \)__ of
square-integrable functions on __\( G \)__ with respect to Haar measure. Left
translation induces an orthogonal representation of __\( G \)__ on __\( \mathcal{H} \)__, the
so-called regular representation, and, according to the Peter–Weyl
Theorem, __\( \mathcal{H} \)__ is the orthogonal direct sum of finite-dimensional
subrepresentations, __\( \mathcal{H}_i \)__, i.e., __\( \mathcal{H} =\bigoplus^{\infty}_{i=1} \mathcal{H}_i \)__.
Define __\( W_N := \bigoplus^N_{i=1}\mathcal{H}_i \)__.
We will show that for __\( N \)__ sufficiently large, the finite-dimensional
representation of __\( G \)__ on __\( W_N \)__ is faithful or, equivalently, that for
some __\( N \)__ the kernel __\( K_N \)__ of the regular representation restricted to __\( W_N \)__
is the trivial group __\( \{e\} \)__. Since the regular representation itself is
clearly faithful, __\( K_N \)__ is a decreasing sequence of compact subgroups of
__\( G \)__ whose intersection is __\( \{e\} \)__. Thus if __\( U \)__ is an open neighborhood of __\( e \)__
that contains no nontrivial subgroup, __\( K_N \backslash U \)__ is a decreasing sequence
of compact sets with empty intersection and, by the definition of
compactness in terms of closed sets, some __\( K_N \backslash U \)__ must be empty.
Hence __\( K_N \subseteq U \)__, and since __\( K_N \)__ is a subgroup of __\( G \)__, __\( K_N = \{e\} \)__.

#### Following in Gleason’s footsteps

Let’s start with a Lie group __\( G \)__, and let __\( \mathfrak{g} \)__ denote
its Lie algebra. There are (at least!) three equivalent
ways to think of an element of the vector space __\( \mathfrak{g} \)__.
First as a vector __\( v \)__ in __\( T G_e \)__, the tangent space to __\( G \)__
at __\( e \)__; second as the left-invariant vector field __\( X \)__ on
__\( G \)__ obtained by left translating __\( v \)__ over the group;
and third as the one-parameter subgroup __\( \phi \)__ of
__\( G \)__ obtained as the integral curve of __\( X \)__ starting at
the identity. The exponential map __\( \exp : \mathfrak{g} \to G \)__ is
defined by __\( \exp(v ) = \phi(1) \)__. It follows immediately
from this definition that __\( \exp(0) = e \)__ and that the
differential of __\( \exp \)__ at 0 is the identity map of __\( T G_e \)__,
so by the inverse function theorem, __\( \exp \)__ maps a
neighborhood of 0 in __\( \mathfrak{g} \)__ diffeomorphically onto a
neighborhood of __\( e \)__ in __\( G \)__. Such a chart for __\( G \)__ is called
a canonical coordinate system (of the first kind).

Now, suppose we somehow “lost” the differentiable structure of __\( G \)__
but retained our knowledge of __\( G \)__ as a topological group. Is there
some way we could use the latter knowledge to recover the
differentiable structure? That is, can we reconstruct __\( \mathfrak{g} \)__
and the exponential map? If so, then we are clearly close to a
solution of the Fifth Problem. Let’s listen in as Andy ponders this
question.

“Well, if I think of __\( \mathfrak{g} \)__ as being the one-parameter
groups, that’s a group theoretic concept. Let’s see — is there
some way I can invert __\( \exp \)__? That is, given __\( g \)__ in __\( G \)__
close to __\( e \)__, can I find the one-parameter group __\( \phi \)__ such that __\( \phi
(1) = \exp(\phi) = g \)__? Now I know square roots are unique
near __\( e \)__ and in fact __\( \phi(1/2) \)__ is the square root of __\( g \)__. By induction, I
can find __\( \phi(1/2^n) \)__ by starting with __\( g \)__ and taking the square root __\( n \)__
times. And once I have __\( \phi(1/2^n) \)__, by simply taking its __\( m \)__-th power I
can find __\( \phi(m/2^n ) \)__ for all __\( m \)__. So, if I know how
to take square roots near __\( e \)__, then I can compute __\( \phi \)__
at all the dyadic rationals __\( m /2^n \)__, and since they are
dense in __\( \mathbf{R} \)__, I can extend __\( \phi \)__ by continuity to find it
on all of __\( \mathbf{R}! \)__”

This was the stated motivation for Gleason’s paper “Square roots in
locally Euclidean groups”
[1],
and in it he goes on to take the
first step and show that in any NSS locally Euclidean group __\( G \)__, there
are neighborhoods __\( U \)__ and __\( V \)__ of the identity such that every element
in __\( U \)__ has a unique square root in __\( V \)__. Almost immediately after this
article appeared, in a paper called “On a theorem of Gleason”,
Chevalley
[e14]
went on to complete the program Andy outlined. That
is, Chevalley used Gleason’s existence of unique square roots to
construct a neighborhood __\( U \)__ of the identity in __\( G \)__ and a continuous
mapping __\( (g , t ) \mapsto \phi^g (t ) \)__ of __\( U \times \mathbf{R} \)__ into
__\( G \)__ such that each __\( \phi^g \)__ is a one-parameter subgroup of __\( G , \phi^g
(t) \in U \)__ for __\( |t| \leq 1 \)__, and __\( \phi^g (1) = g \)__.

In his key 1952 *Annals* paper “Groups without small
subgroups”
[3],
Gleason decided not to follow up this approach to
the solution of the Fifth Problem and instead used a variant of
von Neumann’s method. His approach was based on the construction of
one-parameter subgroups, but these were used as a tool to find a
certain finite-dimensional invariant linear subspace __\( Z \)__ of the
regular representation of __\( G \)__ on which __\( G \)__ acted faithfully and
appealed to Cartan’s Theorem to complete the proof. The construction
of __\( Z \)__ is a technical tour de force, but it is too complicated to
outline here, and we refer instead to the original paper
[2]
or the
review by Iwasawa.

#### Andy Gleason as mentor

Looking back at how it happened, it seems almost accidental that I became Andy Gleason’s first Ph.D. student — and David Hilbert was partly responsible.

As an undergraduate at Harvard I had developed a very close mentoring relationship with George Mackey, then a resident tutor in my dorm, Kirkland House. We had meals together several times each week, and I took many of his courses. So, when I returned in 1953 as a graduate student, it was natural for me to ask Mackey to be my thesis advisor. When he inquired what I would like to work on for my thesis research, my first suggestion turned out to be something he had thought about himself, and he was able to convince me quickly that it was unsuitably difficult for a thesis topic. A few days later I came back and told him I would like to work on reformulating the classical Lie theory of germs of Lie groups acting locally on manifolds as a rigorous modern theory of full Lie groups acting globally. Fine, he said, but then explained that the local expert on such matters was a brilliant young former Junior Fellow named Andy Gleason who had just joined the Harvard math department. Only a year before he had played a major role in solving Hilbert’s Fifth Problem, which was closely related to what I wanted to work on, so he would be an ideal person to direct my research.

I felt a little unhappy at being cast off like that by Mackey, but of
course I knew perfectly well who Gleason was and I had to admit that
George had a point. Andy was already famous for being able to think
complicated problems through to a solution incredibly fast. “Johnny”
von Neumann had a similar reputation, and since this was the year that
*High Noon* came out, I recall jokes about having a mathematical
shootout — Andy vs. Johnny solving math problems with blazing speed at
the OK Corral. In any case, it was with considerable trepidation that
I went to see Andy for the first time.

Totally unnecessary! In our sessions together I never felt put down. It is true that occasionally when I was telling him about some progress I had made since our previous discussion, partway through my explanation Andy would see the crux of what I had done and say something like, “Oh! I see. Very nice! and then…,” and in a matter of minutes he would reconstruct (often with improvements) what had taken me hours to figure out. But it never felt like he was acting superior. On the contrary, he always made me feel that we were colleagues, collaborating to discover the way forward. It was just that when he saw his way to a solution of one problem, he liked to work quickly through it and then go on to the next problem. Working together with such a mathematical powerhouse put pressure on me to perform at top level — and it was sure a good way to learn humility!

My apprenticeship wasn’t over when my thesis was done. I remember that shortly after I had finished, Andy said to me, “You know, some of the ideas in your thesis are related to some ideas I had a few years back. Let me tell you about them, and perhaps we can write a joint paper.” The ideas in that paper were in large part his, but on the other hand, I did most of the writing, and in the process of correcting my attempts he taught me a lot about how to write a good journal article.

But it was only years later that I fully appreciated just how much I had taken away from those years working under Andy. I was very fortunate to have many excellent students do their graduate research with me over the years, and often as I worked together with them I could see myself behaving in some way that I had learned to admire from my own experience working together with Andy.

Let me finish with one more anecdote. It concerns my favorite of all Andy’s theorems, his elegant classification of the measures on the lattice of subspaces of a Hilbert space. Andy was writing up his results during the 1955–56 academic year, as I was writing up my thesis, and he gave me a draft copy of his paper to read. I found the result fascinating, and even contributed a minor improvement to the proof, as Andy was kind enough to footnote in the published article. When I arrived at the University of Chicago for my first position the next year, Andy’s paper was not yet published, but word of it had gotten around, and there was a lot of interest in hearing the details. So when I let on that I was familiar with the proof, Kaplansky asked me to give a talk on it in his analysis seminar. I’ll never forget walking into the room where I was to lecture and seeing Ed Spanier, Marshall Stone, Saunders Mac Lane, André Weil, Kaplansky, and Chern all looking up at me. It was pretty intimidating, and I was suitably nervous!

But the paper was so elegant and clear that it was an absolute breeze to lecture on it, so all went well, and this “inaugural lecture” helped me get off to a good start in my academic career.