Standard formula for compensation of the group velocity of electromagnetic waves corrected

While the vacuum light velocity c represents a natural constant, the propagation velocity of light cair must be considered precisely for any optical distance measurement in air. In case of a monochromatic light wave, the propagation velocity is given by the quotient of c and the air refractive index n. The latter depends on the frequency of the light, but also on temperature, air pressure, relative humidity and carbon dioxide content (e.g. [1,2]). However, many measurement methods use electromagnetic wave packets that propagate at the so-called group velocity:

,

where n_g represents the group refractive index. This particularly applies for methods used in high-precision surveying with total stations. Philip E. Ciddor and Reginald J. Hill published an algorithm in 1999 that provides an analytical solution to the determination equation  [3], with σ representing the wave number (σ = 1 / λ, λ: vacuum wavelength). This algorithm has been officially recommended by the International Association of Geodesy (IAG) for use in geodetic measurements since 1999.

In the course of theoretical work for the currently ongoing European EMPIR research project "Large-scale dimensional measurements for geodesy" (18SIB01, GeoMetre), it has now been uncovered that there is a subtle but relevant sign error in the published algorithm [3] which leads to deviations of the order of a few 10 ^-6 (Figure 1). Unfortunately, this corresponds exactly to the order of magnitude expected for the deviation of the group refractive index from the phase refractive index. Thus, the erroneous result apparently withstands even a somewhat more critical examination, which made the error very difficult to detect. The corrected version of the algorithm and an implementation in Python have been published Open Access by PTB [4,5] and are thus now available for verification of existing implementations of the algorithm and as a basis for accurate length measurements in air.

Figure 1: Difference between the erroneous [3] and the numerically determined correct group refractive index for a given temperature t of 20 °C, an ambient air pressure p of 1013.25 hPa, a relative humidity e of 50 % relative humidity, and a carbon dioxide content xC of 450 ppm. Dashed lines mark vacuum wavelengths of typical light sources used in length metrology. (reproduced from [4] under the OSA Open Access Publishing Agreement).

This EMPIR project is co-funded by the European Union's Horizon 2020 research and innovation programme and the EMPIR Participating States.

References

[1] P. E. Ciddor 1996 “Refractive index of air: new equations for the visible and near infrared” Appl. Opt. 35, 1566–1573 http://dx.doi.org/10.1364/AO.35.001566

[2] G. Bönsch, E. Potulski 1998 "Measurement of the refractive index of air and comparison with modified Edlén's formulae", Metrologia 35, 133 - 139

[3] P. E. Ciddor and R. J. Hill 1999 “Refractive index of air. 2. Group index” Appl. Opt. 38 1663–1667 http://dx.doi.org/10.1364/AO.38.001663

[4] F. Pollinger 2020 "Refractive index of air. 2. Group index: comment" Appl. Opt. 59, 9771-9772 doi.org/10.1364/AO.400796

[5] F. Pollinger 2020 “CiddorPy: python functions for phase and group refractive index in air” figshare doi.org/10.6084/m9.figshare.12515320