Appendix C

The Vacuum and the Cosmological Constant Problem

Introduction

The cosmological constant problem exists because of a number of key assumptions. These will be identified in this introduction. The second section discusses the issue of vacuum instability and negative energy states. Following this is a section on the relation between gauge invariance, Schwinger terms, and the definition of the vacuum. Given recent observational data from the Supernova Cosmology Project — showing an acceleration of the expansion of the universe at great distances, it is attractive to look at definitions of the vacuum that could lead to reasonable, but non-zero values of the cosmological constant.

While it has never caught a great deal of interest, the redefinition of the vacuum proposed by Solomon1 is one possible approach to achieving this goal. It is included here primarily for heuristic purposes.

The final section looks at the origin of the difficulties in QFT and also serves as a summary.

Perhaps the best introduction to the cosmological constant problem is the review by Weinberg² published over twenty years ago. Although there have been a variety of approaches to the issue since then, the article can still serve to frame the problem, at least in the sense that it is to be addressed here. To avoid confusion, Weinberg’s notation will be used in this introduction.

The Einstein field equations, including the cosmological constant λ, are

The right-hand side of the equation contains the energy-momentum tensor T_µν, and it is here that the first assumption that leads to the cosmological constant problem is made. It is assumed that the vacuum has a non-zero energy density. If such a vacuum energy density exists, Lorentz invariance requires that it have the form3

This allows one to define an effective cosmological constant and a total effective vacuum energy density

Observationally, it is known that the total effective vacuum energy density must be comparable to about .

The second assumption that leads to the cosmological constant problem is that the vacuum energy density is due to the zero-point energy of a quantized field or fields. The concept of the zero-point energy originated with the quantization of the simple harmonic oscillations of a particle with non-vanishing mass. This energy is present for each energy level including the ground state. In QFT, at a given instant of time, the field is defined at each point in space — essentially an independent harmonic oscillation at each point with amplitude and phase depending on the initial conditions. By specifying an equation of motion depending on the amplitude of the field and its partial derivative with respect to time, one couples these otherwise independent oscillations. In the Klein–Gordon equation, for example, this coupling is achieved by the presence of the Laplacian.

The zero-point energy is also present in the relativistic QFT of the neutral Klein–Gordon field and the electromagnetic field. Of course, the zero-point energy of the Klein–Gordon field is infinite, but when used to formulate the cosmological constant problem, it is generally summed up to a cutoff that determines the magnitude of the vacuum energy density. The way this is done is to count the number of normal modes between v and v + dv, multiply by the zero-point energy in each mode, and integrate up to the chosen cutoff. There is some question as to whether it makes sense to count the normal modes for a massive particle when addressing the energy density of the vacuum where no particles are present. The usual justification for this is somewhat confused in the literature, but is generally based on identifying the zero-point energy with vacuum fluctuations.

The number of normal modes between v and v + dv is given by

Transforming to wave number, setting the volume equal to unity, and using units where ℏ = c = 1, and integrating up to a wave number cutoff Λ ≫ 1 results in a vacuum energy density of

As a relativistic wave equation, the Klein–Gordon equation has an energy spectrum that includes both positive and negative energies, corresponding to either sign for the radical in Eq. (C.5). Note that if the integral were performed for both signs and the results added, the vacuum energy density would vanish. In QFT, zero-point energies are inherent in the canonical field quantization method because the ordering of operators in the Hamiltonian is not fixed.

In Eq. (C.5), the vacuum energy density up to a cutoff is calculated for the cosmological constant problem by using only the positive sign for the radical. The reason for this goes back to Dirac’s redefinition of the vacuum where electrons are assumed to fill the negative-energy states, and they and their infinite vacuum charge density are presumed to be unobservable — as are the negative-energy state zero-point energies. In this way, the instability of the vacuum caused by the availability of negative energy states is eliminated. The normal ordering procedure of canonical QFT, in addition to eliminating the zero-point energies, also has the virtue of eliminating the infinite vacuum charge density of the Dirac vacuum. QFT does not, however, eliminate the problem of negative energies as will be seen below.

The value of wave number cutoff to be chosen in Eq. (C.5) depends on one’s view of Einstein’s theory of gravitation. Fields are generally defined in the context of a background metric for spacetime — the Minkowski metric. General relativity has to do with the geometry of spacetime itself, and while there is no a priori reason (or experimental evidence) that the gravitational field need be quantized, it is generally believed that Einstein’s theory of gravitation will no longer hold at very small distances, and in particular, for distances comparable or smaller than the Planck length (Gℏ/c³)^1/2. At this length scale, quantum fluctuations are thought to be the dominant influence on the local spacetime geometry. For this reason, the Planck length is generally chosen to define the cutoff. Such a cutoff is really an expression of our ignorance of the vacuum and the structure (if there is one) of spacetime itself at these distances. Unfortunately, no experimental data exists.

Evaluating the right-hand side of Eq. (C.5) with Weinberg’s choice of Λ = (8πG)^-1/2 (the units are again such that ℏ = c = 1) results in a value for the vacuum energy density of

Comparing this with the observational value of ρ_V ~ 10^-29 g/cm³ tells us that the two terms in the effective vacuum energy density of Eq. (C.3) must cancel to some 118 decimal places. This fine tuning is the cosmological constant problem as it is currently understood.

As discussed above, the Casimir effect, which in QFT is thought to be due to the presence of vacuum fluctuations, is used to argue that the zero-point energies are real. That is, zero-point energies and quantum fluctuations are identified. A key reference for the Casimir effect is the work of Plunien, Müller, and Greiner. 4 Vacuum fluctuations — which represent the terms of the perturbation series for the vacuum expectation value ⟨0|S|0⟩ of the S-matrix — are generally ignored since they can only produce an overall phase factor. However, in the presence of external sources or boundaries, these fluctuations can no longer be so casually dismissed. Although the vacuum state must be invariant under rotations and translations — and therefore must have zero momentum, angular momentum, and energy — the presence of external sources or boundaries (as in the Casimir effect) breaks these fundamental symmetries, allowing vacuum fluctuations to have effects that are observable.

Moreover, if the energy of the vacuum is defined as the difference of zero-point energies in the presence of boundaries ∂Γ of a region Γ and without boundaries, the vacuum energy can be negative; that is,

where E₀[∂Γ] is the zero-point energy in the presence of boundaries and E₀[0] is the zero-point energy in their absence.

Because of the absence of boundaries or external sources, performing the kind of calculation given in Eq. (C.5) has raised questions in the literature with regard to the legitimacy of the approach of simply summing free field modes. See, for example, Lamoreaux⁵ where it is argued that we do not learn much about the properties of the vacuum of free space through the study of Casimir and related zero-point energy effects.

In the absence of interactions as well as boundaries, one could make the argument that there appears to be no reason not to allow the vacuum to have a negative energy spectrum so that both signs of the energy in evaluating Eq. (C.5) should be used, thereby eliminating the cosmological constant problem. In the presence of interactions the issue becomes more complicated, but — as will be seen below — the availability of a negative energy spectrum is one way to restore gauge invariance.

Vacuum Instability and Negative Energy States

At this point, it might be useful to examine the origin of the concern about vacuum instability in the presence of negative energy states. Consider a free electron in a positive energy state E subjected to a periodic perturbation of frequency ω. It is generally agreed that there is a non-vanishing probability for the electron to make a transition to a state of energy (E + ℏω) or (E − ℏω). If ℏω > (E + mc²), it is argued that the transition to (E − ℏω) would be to a state of negative energy. Having set the stage, the argument continues by considering a bound state electron in a hydrogen atom. Since the electron is coupled to the electromagnetic field, such a bound state would rapidly make a radiative transition to a negative energy state, with the result that the hydrogen atom would have no stable existence. Worse yet, since the spectrum has no lower bound, there would be no limit to the radiated energy.

There is an objection to the above argument that shows that such transitions could well be forbidden. Under the usual sign convention, quantum states of positive energy evolve in time as e^-iωt. A state of negative energy, −E, then evolves as e^+iωt, which corresponds to the transformation t → −t. But there are two possibilities for a time reversal operator: it can be unitary or anti-unitary — where t → −t and one also takes the complex conjugate of states and complex numbers. Under a unitary transformation, used for all other discrete and continuous symmetries, the time-reversed state corresponds to a state of negative energy −E. It was Eugene Wigner who introduced the anti-unitary time reversal operator so as to eliminate negative energies.

There are now two arguments that can be given against the transition from a bound state (as in the above argument) to a negative energy state under the periodic perturbation. First, if the Hamiltonian governing the transition is to be CPT invariant — as it must if it is to be an acceptable quantum electrodynamics Hamiltonian — it must satisfy . This will only be the case if the operator CPT is anti-unitary, a consequence of the complex conjugation implicit in the time reversal operator T. But since the transformation from e^-iωt → e^+iωt corresponds to simply t → −t, we have instead CPT , giving the negative energy state. Thus, the Hamiltonian of the transition cannot satisfy CPT invariance.

The second argument has to do with the fact that if an electron in an external field obeys the (quantized) Dirac equation, one cannot rule out the negative energy solutions needed to make up a complete set of wavefunctions. But a wavefunction representing a negative energy state can only be non-zero if it has charge +e.6 This means the argument given above for the transition of an electron to a negative energy state will violate charge conservation.

Thus, a radiative transition of an electron to a negative energy state either violates CPT invariance or the conservation of charge. So, it would appear that such transitions are effectively forbidden. This means there is no reason not to allow negative energies to be summed over in Eq. (C.5), yielding a vanishing vacuum energy density. The attractiveness of Solomon’s approach to redefining the vacuum, to be described below, is that it allows for incomplete cancellation of the zero-point energies leading to a small, but not vanishing vacuum energy density.

QFT and Gauge Invariance

The issue of the gauge invariance of QFT has been dealt with in a variety of ways over the years (an extensive discussion is contained in Solomon’s paper). In essence, the standard vacuum of QFT is only gauge invariant if non-gauge invariant terms are removed. There are two general approaches to the problem: the first is to simply ignore such terms as being physically untenable and remove them so as to maintain gauge invariance; and the second is to use various regularization techniques to cancel the terms. Pauli–Villars regularization⁷ is discussed below.

Returning to Eq. (C.5), another problem with this equation is that the introduction of a momentum-space cutoff destroys translational invariance and may make it difficult to maintain gauge and Lorentz invariance. If, instead of introducing a cutoff, an attempt is made to deal with this infinite integral by Pauli–Villars regularization, the result is the introduction of negative masses. This can be seen as follows. It can be guaranteed that the vacuum expectation value of the energy momentum tensor is Lorentz invariant and hence proportional to η^µν if we use the relativistic formulation

Here p⁰ = E(p) = (p² + m²)^1/2. The domain of integration can be transformed to spherical coordinates in a space of any dimension by the use of the formula

resulting in

The Pauli–Villars regulator masses m_i, where m₁ = m, and associated coefficients c_i, where c₁ = 1, are then introduced as follows:

The number of regulator masses needed depends in general on the integral. Since we are interested in the convergence of the integral at the upper limit where , the radical can be expanded in a series to yield

This integral will converge at the upper limit provided the following relations are satisfied as m_i → ∞:

Moving p² in the numerator of Eq. (C.12) into the series expansion shows that Eqs. (C.13) will be satisfied if n = 3.

Because some of the coefficients must be negative, this procedure introduces negative energies in the form of negative masses that are allowed to become infinite at the end of the calculation. The advantage, at least for QFT, is that gauge and Lorentz invariance are preserved.

If one works in Euclidean space by first performing a Wick rotation, one is left with essentially the same problem: regulator fields for scalar fields obey Fermi statistics, and those for spinor fields obey Bose statistics. This violation of the spin-statistics theorem means that the Hamiltonian cannot be a positive definite operator, again implying the existence of negative energy states.

Solomon has argued that for QFT to be gauge invariant the Schwinger term must vanish, and Schwinger8 long ago showed that for this to be the case the vacuum state cannot be the state with the lowest free field energy. The existence of non-zero Schwinger terms also impacts Lorentz invariance. Lev⁹ has shown that if the Schwinger terms do not vanish the usual current operator , where μ = 0, 1, 2,3 and x is a point in Minkowski space, is not Lorentz invariant.

To begin with, however, it is important to understand the details of Schwinger’s argument. Using Solomon’s notation, the Schwinger term is given by

Taking the divergence of the Schwinger term and using the relation

where is the free-field Hamiltonian when the electromagnetic fourpotential vanishes, results in

Expanding the commutator on the right-hand side of Eq. (C.16) yields the vacuum expectation value

It is here that one makes the assumption that the vacuum is the lowest energy state. This is done by writing |0⟩ = ⟨0| = 0. As a result, Eq. (C.17) may be written as

Multiply both sides of the last equation by f(x)f(y) and integrate over x and y. The right-hand side of Eq. (C.18) becomes

If Schwinger’s “arbitrary linear functional of the charge density” is defined as

the right-hand side of Eq. (C.18) becomes

The left-hand side of Eq. (C.21) — essentially the form used by Schwinger — is here expanded to explicitly show the non-vanishing matrix elements between the vacuum and the other states of necessarily positive energy. This shows that if the vacuum is assumed to be the lowest energy state, the Schwinger term cannot vanish, and the theory is not gauge invariant. Solomon also shows the converse, that if the Schwinger term vanishes, then the vacuum is not the lowest energy state and the theory is gauge invariant.

For the sake of completeness, it is readily shown that the left-hand side of Eq. (C.18) becomes

so that combining Eqs. (C.21) and (C.22) yields a somewhat more explicit form of the result given by Schwinger,

Solomon’s Redefinition of the Vacuum

In QFT, the invariant perturbation theory10 leads to the Dyson chronological operator , where is the interaction Hamiltonian. The adjective “invariant” refers to the fact that time ordering in the Dyson series is Lorentz invariant if the all commute at space-like separations. The chronological product can be expressed in a form where the virtual processes are explicitly represented; that is, as a decomposition into normal products, where a Feynman graph can be used to represent each of the normal products. The vacuum state is empty (although vacuum fluctuations exist) and is the state of lowest energy. The theory is not, however, gauge invariant and a process of regularization and renormalization is used to make it so.

The virtue of the vacuum state to be defined in this section is that it allows QFT to be mathematically consistent in the sense that it becomes gauge invariant without the need for regularization. Because the theory is gauge invariant, the Schwinger term vanishes. The reason for this is that the definition of the vacuum is such that the usual vacuum state, |0⟩, is no longer the state of minimum energy, and there exist states with negative energy. In the context of the cosmological constant problem, this means that even if one believes the summation of zero-point energies given in Eq. (C.5) is legitimate, it must be extended to the unoccupied negative energy states, leading to a significant cancellation in the summation.

Redefining the vacuum state may provide a more elegant means of resolving both the gauge invariance difficulties of QFT and the cosmological constant problem. The specific definition of the vacuum state given below serves, at a minimum, as an heuristic example of this approach.

Because of its intuitive nature, hole theory will be used to introduce Solomon’s redefinition of the vacuum in quantum electrodynamics. He has also implemented this redefinition in the context of QFT, and that will also be described here. It may therefore be useful to recall how the transition to QFT is made.

In terms of positive and negative energy electrons, second quantization gives the Hamiltonian for a free electron,

where is the creation operator for positive energy electrons and creates negative energy electrons. In terms of the number operator N,

where E_p > 0 and the number of negative energy electrons.

Dirac’s hole theory rescales the energy so that

If N^- (for a given p and s) vanishes — i.e., when a negative-energy electron is missing, then N⁺ increases by one and we see a positive energy electron and a hole in the negative energy continuum, which is interpreted as a positron.

For the hole theory approach to dealing with negative energies to work, it is essential that the particles filling the negative energy states obey the Pauli exclusion principle. The technique would not work for bosons associated with the Klein–Gordon equation. While the particles filling the negative energy states must not produce an electric field or contribute to the total charge, energy, or momentum, they nevertheless must respond to an external field.

The transition to QFT is accomplished by setting

so that destroying a negative energy electron is equivalent to creating a positron. The Dirac negative energy sea vanishes since electrons and positrons are treated as separate entities. Provided the b’s satisfy the anti-commutation relations

we have that

As a result,

since |0⟩ contains neither positrons or electrons. The free-electron Hamiltonian in terms of the number operator, now sums over both electrons and positrons and is given by

Thus, one can readily move between hole theory and QFT although, as Solomon has shown, one sometimes obtains differing results because of the way the vacua are defined.

With reference to Fig. 1, Solomon defines a state vector |0, ΔE_W⟩ as the state where a band of negative energy states extending from −m to −(m + ΔE_W) is occupied by a single particle (the exclusion principle holds); all other single particle states are unoccupied.

The vacuum state is defined by Solomon as |0, ΔE_W → ∞⟩, where it is important to understand that the limit ΔE_W → ∞ means that ΔE_W goes to an arbitrarily large but finite number. If ΔE_W were set equal to infinity one would be reproducing the Dirac vacuum. This definition of the vacuum allows transitions from the occupied negative energy states within the band to those beneath the band, thereby making the Schwinger term vanish, and the theory is consequently gauge invariant.

Fig. C.1. The state vector |0, ΔE_W⟩. Only the band of negative energy states extending from −m to −(m + ΔE_W) is occupied.

The vacuum can be similarly redefined in the context of QFT. With reference to Eq. (C.27), the creation and destruction operators in QFT obey

while new states are created by and operating on the vacuum state |0⟩. Solomon11 redefines the vacuum to be |0_R⟩, as R → ∞ with the following restrictions:

New states are created by

Graphically, these conditions may be displayed as in Fig. C.2.

Fig. C.2. Solomon’s redefinition of the vacuum in QFT. Note that d_ps|0_R⟩, |p| > R creates states with energy less than |0_R⟩.

As R → ∞ the usual relations of Eq. (C.32) are recovered. Solomon shows that the Schwinger term vanishes using this redefinition of the vacuum so that the theory is gauge invariant. Again, to achieve this it was necessary to introduce negative energy states.

The key question to be addressed, with this or any redefinition of the vacuum that allows negative-energy states, is stability. First, can positive energy particles be scattered into unoccupied negative energy states? And second, is the vacuum catastrophically unstable, in the sense that the transition rate from occupied negative energy states within the band shown in Fig. 1 to those below the band, so great that the band vanishes essentially instantaneously?

With regard to the first question, the idea here is to make ΔE_W large enough to make the probability of such transitions negligibly small. The answer to the second question is still open, but Solomon has determined that for a field theory of non-interacting zero-mass fermions in the presence of a classical electromagnetic field in a two-dimensional spacetime, the vacuum is stable. It remains to show that this remains the case for four-dimensional spacetime.

Origin of the Inconsistencies in QFT and Summary

The problem of maintaining gauge invariance in QFT when interactions are present, and the difficulties with Schwinger terms discussed above, stem from the now well-known fact that the underlying assumptions of QFT are inconsistent. The essence is contained in Haag’s theorem,12 which is concerned with the interaction picture that forms the basis for perturbation theory. Haag’s theorem is important not least for the fact that it identifies the reason regularization and renormalization are needed in QFT; that is, the underlying assumptions of relativistic QFT are inconsistent in the context of interacting systems.

In QFT, relativistic transformations between states are governed by the continuous unitary representations of the inhomogeneous group SL(2,C) — essentially the complex Poincaré group. One might anticipate that when interactions are present, the unitarity condition might be violated. Indeed, Haag’s theorem states, in essence, that if ϕ₀ and ϕ are field operators defined respectively in Hilbert spaces and , with vacua |0⟩₀ and |0⟩, and if ϕ₀ is a free field of mass m, then a unitary transformation between ϕ₀ and ϕ exists only if ϕ is also a free field of mass m. Another way of putting this is that if the interaction picture is well defined, it necessarily describes a free field.

The assumption that the vacuum state is the minimum energy state, invariant under a unitary transformation, is one of the fundamental assumptions of QFT. But it is now known that the physical vacuum state is not simple and must allow for spontaneous symmetry breaking and a host of other properties, so that the real vacuum bears little relation to the vacuum state of axiomatic QFT. Nevertheless, even if the latter type of vacuum is assumed, the violation of the unitarity condition in the presence of interactions opens up the possibility that the spectral condition, which limits momenta to being within or on the forward light cone, may also be violated thereby allowing negative energy states.

Of course, the way QFT gets around the formal weakness of using the interaction picture is to regularize the singular field functions that appear in the perturbation series followed by renormalization. There is nothing wrong with this approach from a pragmatic point of view, and it works exceptionally well in practice. The reason one may want to look at other approaches, such as redefining the vacuum state and the role of negative energies, is that it may lead to insights into the nature of the vacuum itself, and help resolve the outstanding cosmological constant problem, and could offer a new approach and possibly an alternative to the process of regularization and renormalization.

If it is indeed possible to use the negative energy spectrum to cancel much of the vacuum energy density of Eq. (C.5), there is now observational evidence pointing to what the residual vacuum energy density must be. This comes from the observational data of the Supernova Cosmology Project.13

____________________________________

¹ D. Solomon, “Gauge invariance and the vacuum state,” Can. J. Phys. 76 (1998), 111–127.

² S. Weinberg, “The cosmological constant problem,” Rev. Mod. Phys. 61 (1989), 1–23.

³ The signature of the metric is +2.

⁴ G. Plunien, B. Müller, and W. Greiner, “The Casimir effect,” Phys. Rep. 134 (1986), 87–193.

⁵ S. K. Lamoreaux, “Casimir forces: Still surprising after 60 years,” Physics Today (February 2007) pp. 40–45.

⁶ S. Weinberg, The Quantum Theory of Fields (Cambridge University Press, New York, 1995), Vol. 1, Ch. 14.

⁷ W. Pauli and F. Villars, “On the invariant regularization in relativistic quantum theory,” Rev. Mod. Phys. 21 (1949), 434–444. Because Pauli–Villars regularization does not preserve gauge invariance in non-Abelian gauge theories, dimensional regularization has become the means of choice.

⁸ J. Schwinger, “Field theory commutators,” Phys. Rev. Lett. 3 (1959), 296–297.

⁹ F. M. Lev, “The problem of constructing the current operators in quantum field theory,” arXiv: hep-th/9508158v1 (29 August 1995).

¹⁰ S. S. Schweber, H. A. Bethe, and F. de Hoffman, Mesons and Fields Vol. I Fields (Row, Peterson & Co., White Plains, NY, 1955), p. 192.

¹¹ D. Solomon, “Another look at the problem of gauge invariance in QFT,” arXiv: 0708.2888 (12 September 2007, V2).

¹² R. F. Streater and A.S. Wightman, PCT, Spin & Statistics, and all That (W. A. Benjamin, Inc., New York, 1964), Sect. 4–5. See also, P. Roman, Introduction to Quantum Field Theory (John Wiley & Sons, Inc., New York, 1968), p. 388.

¹³ S. Perlmutter, “Supernovae, dark energy, and the accelerating universe,” Physics Today (April 2003).