Missing Value Imputation by Kalman Smoothing and State Space Models

Source: R/na_kalman.R

Uses Kalman Smoothing on structural time series models (or on the state space representation of an arima model) for imputation.

na_kalman(x, model = "StructTS", smooth = TRUE, nit = -1, maxgap = Inf, ...)

Arguments

x

Numeric Vector (vector) or Time Series (ts) object in which missing values shall be replaced
model

Model to be used. With this parameter the State Space Model (on which KalmanSmooth is performed) can be chosen. Accepts the following input:

  • "StructTS" - For using a structural model fitted by maximum likelihood (using StructTS) (default choice)

  • "auto.arima" - For using the state space representation of arima model (using auto.arima)

For both auto.arima and StructTS additional parameters for model building can be given with the ... parameter

Additionally it is also possible to use a user created state space model (See code Example 5). This state space model could for example be obtained from another R package for structural time series modeling. Furthermore providing the state space representation of a arima model from arima is also possible. But it is important to note, that user created state space models must meet the requirements specified under KalmanLike. This means the user supplied state space model has to be in form of a list with at least components T, Z, h , V, a, P, Pn. (more details under KalmanLike)
smooth

if TRUE - KalmanSmooth is used for estimation, if FALSE - KalmanRun is used. Since KalmanRun is often considered extrapolation KalmanSmooth is usually the better choice for imputation.
nit

Parameter from Kalman Filtering (see KalmanLike). Usually no need to change from default.
maxgap

Maximum number of successive NAs to still perform imputation on. Default setting is to replace all NAs without restrictions. With this option set, consecutive NAs runs, that are longer than 'maxgap' will be left NA. This option mostly makes sense if you want to treat long runs of NA afterwards separately.
...

Additional parameters to be passed through to the functions that build the State Space Models (StructTS or auto.arima).

Value

Vector (vector) or Time Series (ts) object (dependent on given input at parameter x)

Details

The KalmanSmoother used in this function is KalmanSmooth. It operates either on a Basic Structural Model obtained by StructTS or the state space representation of a ARMA model obtained by auto.arima.

For an detailed explanation of Kalman Filtering and Space Space Models the following literature is a good starting point:

  • G. Welch, G. Bishop, An Introduction to the Kalman Filter. SIGGRAPH 2001 Course 8, 1995

  • Harvey, Andrew C. Forecasting, structural time series models and the Kalman filter. Cambridge university press, 1990

  • Grewal, Mohinder S. Kalman filtering. Springer Berlin Heidelberg, 2011

References

Hyndman RJ and Khandakar Y (2008). "Automatic time series forecasting: the forecast package for R". Journal of Statistical Software, 26(3).

See also

na_interpolation, na_locf, na_ma, na_mean, na_random, na_replace, na_seadec, na_seasplit

Author

Steffen Moritz

Examples

# Example 1: Perform imputation with KalmanSmoother and state space representation of arima model
na_kalman(tsAirgap)

#>           Jan      Feb      Mar      Apr      May      Jun      Jul      Aug
#> 1949 112.0000 118.0000 132.0000 129.0000 123.5734 135.0000 148.0000 148.0000
#> 1950 115.0000 126.0000 141.0000 135.0000 125.0000 149.0000 170.0000 170.0000
#> 1951 145.0000 150.0000 178.0000 163.0000 172.0000 178.0000 199.0000 199.0000
#> 1952 171.0000 180.0000 193.0000 181.0000 183.0000 218.0000 230.0000 242.0000
#> 1953 196.0000 196.0000 236.0000 235.0000 229.0000 243.0000 264.0000 272.0000
#> 1954 204.0000 188.0000 235.0000 227.0000 234.0000 260.8078 302.0000 293.0000
#> 1955 242.0000 233.0000 267.0000 269.0000 270.0000 315.0000 364.0000 347.0000
#> 1956 284.0000 277.0000 312.6060 311.6160 311.0830 374.0000 413.0000 405.0000
#> 1957 315.0000 301.0000 356.0000 348.0000 355.0000 414.6160 465.0000 467.0000
#> 1958 340.0000 318.0000 382.6912 348.0000 363.0000 435.0000 491.0000 505.0000
#> 1959 360.0000 342.0000 406.0000 396.0000 420.0000 472.0000 548.0000 559.0000
#> 1960 417.0000 391.0000 419.0000 461.0000 452.5683 535.0000 622.0000 606.0000
#>           Sep      Oct      Nov      Dec
#> 1949 132.7833 119.0000 104.0000 118.0000
#> 1950 157.1427 133.0000 130.8971 140.0000
#> 1951 184.0000 162.0000 146.0000 166.0000
#> 1952 209.0000 191.0000 172.0000 194.0000
#> 1953 237.0000 211.0000 180.0000 201.0000
#> 1954 259.0000 229.0000 203.0000 229.0000
#> 1955 312.0000 274.0000 237.0000 278.0000
#> 1956 355.0000 306.0000 271.0000 306.0000
#> 1957 404.0000 347.0000 298.6147 336.0000
#> 1958 404.0000 359.0000 310.0000 337.0000
#> 1959 463.0000 407.0000 362.0000 390.9089
#> 1960 508.0000 461.0000 390.0000 432.0000

# Example 2: Perform imputation with KalmanRun and state space representation of arima model
na_kalman(tsAirgap, smooth = FALSE)

#>           Jan      Feb      Mar      Apr      May      Jun      Jul      Aug
#> 1949 112.0000 118.0000 132.0000 129.0000 123.9264 135.0000 148.0000 148.0000
#> 1950 115.0000 126.0000 141.0000 135.0000 125.0000 149.0000 170.0000 170.0000
#> 1951 145.0000 150.0000 178.0000 163.0000 172.0000 178.0000 199.0000 199.0000
#> 1952 171.0000 180.0000 193.0000 181.0000 183.0000 218.0000 230.0000 242.0000
#> 1953 196.0000 196.0000 236.0000 235.0000 229.0000 243.0000 264.0000 272.0000
#> 1954 204.0000 188.0000 235.0000 227.0000 234.0000 232.6461 302.0000 293.0000
#> 1955 242.0000 233.0000 267.0000 269.0000 270.0000 315.0000 364.0000 347.0000
#> 1956 284.0000 277.0000 304.4782 302.5073 298.1029 374.0000 413.0000 405.0000
#> 1957 315.0000 301.0000 356.0000 348.0000 355.0000 385.8343 465.0000 467.0000
#> 1958 340.0000 318.0000 392.3223 348.0000 363.0000 435.0000 491.0000 505.0000
#> 1959 360.0000 342.0000 406.0000 396.0000 420.0000 472.0000 548.0000 559.0000
#> 1960 417.0000 391.0000 419.0000 461.0000 442.0008 535.0000 622.0000 606.0000
#>           Sep      Oct      Nov      Dec
#> 1949 141.5691 119.0000 104.0000 118.0000
#> 1950 139.5378 133.0000 141.7595 140.0000
#> 1951 184.0000 162.0000 146.0000 166.0000
#> 1952 209.0000 191.0000 172.0000 194.0000
#> 1953 237.0000 211.0000 180.0000 201.0000
#> 1954 259.0000 229.0000 203.0000 229.0000
#> 1955 312.0000 274.0000 237.0000 278.0000
#> 1956 355.0000 306.0000 271.0000 306.0000
#> 1957 404.0000 347.0000 327.9040 336.0000
#> 1958 404.0000 359.0000 310.0000 337.0000
#> 1959 463.0000 407.0000 362.0000 400.7839
#> 1960 508.0000 461.0000 390.0000 432.0000

# Example 3: Perform imputation with KalmanSmooth and StructTS model
na_kalman(tsAirgap, model = "StructTS", smooth = TRUE)

#>           Jan      Feb      Mar      Apr      May      Jun      Jul      Aug
#> 1949 112.0000 118.0000 132.0000 129.0000 123.5734 135.0000 148.0000 148.0000
#> 1950 115.0000 126.0000 141.0000 135.0000 125.0000 149.0000 170.0000 170.0000
#> 1951 145.0000 150.0000 178.0000 163.0000 172.0000 178.0000 199.0000 199.0000
#> 1952 171.0000 180.0000 193.0000 181.0000 183.0000 218.0000 230.0000 242.0000
#> 1953 196.0000 196.0000 236.0000 235.0000 229.0000 243.0000 264.0000 272.0000
#> 1954 204.0000 188.0000 235.0000 227.0000 234.0000 260.8078 302.0000 293.0000
#> 1955 242.0000 233.0000 267.0000 269.0000 270.0000 315.0000 364.0000 347.0000
#> 1956 284.0000 277.0000 312.6060 311.6160 311.0830 374.0000 413.0000 405.0000
#> 1957 315.0000 301.0000 356.0000 348.0000 355.0000 414.6160 465.0000 467.0000
#> 1958 340.0000 318.0000 382.6912 348.0000 363.0000 435.0000 491.0000 505.0000
#> 1959 360.0000 342.0000 406.0000 396.0000 420.0000 472.0000 548.0000 559.0000
#> 1960 417.0000 391.0000 419.0000 461.0000 452.5683 535.0000 622.0000 606.0000
#>           Sep      Oct      Nov      Dec
#> 1949 132.7833 119.0000 104.0000 118.0000
#> 1950 157.1427 133.0000 130.8971 140.0000
#> 1951 184.0000 162.0000 146.0000 166.0000
#> 1952 209.0000 191.0000 172.0000 194.0000
#> 1953 237.0000 211.0000 180.0000 201.0000
#> 1954 259.0000 229.0000 203.0000 229.0000
#> 1955 312.0000 274.0000 237.0000 278.0000
#> 1956 355.0000 306.0000 271.0000 306.0000
#> 1957 404.0000 347.0000 298.6147 336.0000
#> 1958 404.0000 359.0000 310.0000 337.0000
#> 1959 463.0000 407.0000 362.0000 390.9089
#> 1960 508.0000 461.0000 390.0000 432.0000

# Example 4: Perform imputation with KalmanSmooth and StructTS model with additional parameters
na_kalman(tsAirgap, model = "StructTS", smooth = TRUE, type = "trend")

#>         Jan    Feb    Mar    Apr    May    Jun    Jul    Aug    Sep    Oct
#> 1949 112.00 118.00 132.00 129.00 132.00 135.00 148.00 148.00 133.50 119.00
#> 1950 115.00 126.00 141.00 135.00 125.00 149.00 170.00 170.00 151.50 133.00
#> 1951 145.00 150.00 178.00 163.00 172.00 178.00 199.00 199.00 184.00 162.00
#> 1952 171.00 180.00 193.00 181.00 183.00 218.00 230.00 242.00 209.00 191.00
#> 1953 196.00 196.00 236.00 235.00 229.00 243.00 264.00 272.00 237.00 211.00
#> 1954 204.00 188.00 235.00 227.00 234.00 268.00 302.00 293.00 259.00 229.00
#> 1955 242.00 233.00 267.00 269.00 270.00 315.00 364.00 347.00 312.00 274.00
#> 1956 284.00 277.00 301.25 325.50 349.75 374.00 413.00 405.00 355.00 306.00
#> 1957 315.00 301.00 356.00 348.00 355.00 410.00 465.00 467.00 404.00 347.00
#> 1958 340.00 318.00 333.00 348.00 363.00 435.00 491.00 505.00 404.00 359.00
#> 1959 360.00 342.00 406.00 396.00 420.00 472.00 548.00 559.00 463.00 407.00
#> 1960 417.00 391.00 419.00 461.00 498.00 535.00 622.00 606.00 508.00 461.00
#>         Nov    Dec
#> 1949 104.00 118.00
#> 1950 136.50 140.00
#> 1951 146.00 166.00
#> 1952 172.00 194.00
#> 1953 180.00 201.00
#> 1954 203.00 229.00
#> 1955 237.00 278.00
#> 1956 271.00 306.00
#> 1957 341.50 336.00
#> 1958 310.00 337.00
#> 1959 362.00 389.50
#> 1960 390.00 432.00

# Example 5:  Perform imputation with KalmanSmooth and user created model
usermodel <- arima(tsAirgap, order = c(1, 0, 1))$model
na_kalman(tsAirgap, model = usermodel)

#>           Jan      Feb      Mar      Apr      May      Jun      Jul      Aug
#> 1949 112.0000 118.0000 132.0000 129.0000 128.9291 135.0000 148.0000 148.0000
#> 1950 115.0000 126.0000 141.0000 135.0000 125.0000 149.0000 170.0000 170.0000
#> 1951 145.0000 150.0000 178.0000 163.0000 172.0000 178.0000 199.0000 199.0000
#> 1952 171.0000 180.0000 193.0000 181.0000 183.0000 218.0000 230.0000 242.0000
#> 1953 196.0000 196.0000 236.0000 235.0000 229.0000 243.0000 264.0000 272.0000
#> 1954 204.0000 188.0000 235.0000 227.0000 234.0000 270.8450 302.0000 293.0000
#> 1955 242.0000 233.0000 267.0000 269.0000 270.0000 315.0000 364.0000 347.0000
#> 1956 284.0000 277.0000 292.8668 316.5817 341.4769 374.0000 413.0000 405.0000
#> 1957 315.0000 301.0000 356.0000 348.0000 355.0000 409.7449 465.0000 467.0000
#> 1958 340.0000 318.0000 329.7566 348.0000 363.0000 435.0000 491.0000 505.0000
#> 1959 360.0000 342.0000 406.0000 396.0000 420.0000 472.0000 548.0000 559.0000
#> 1960 417.0000 391.0000 419.0000 461.0000 487.7672 535.0000 622.0000 606.0000
#>           Sep      Oct      Nov      Dec
#> 1949 136.5832 119.0000 104.0000 118.0000
#> 1950 150.9669 133.0000 132.7634 140.0000
#> 1951 184.0000 162.0000 146.0000 166.0000
#> 1952 209.0000 191.0000 172.0000 194.0000
#> 1953 237.0000 211.0000 180.0000 201.0000
#> 1954 259.0000 229.0000 203.0000 229.0000
#> 1955 312.0000 274.0000 237.0000 278.0000
#> 1956 355.0000 306.0000 271.0000 306.0000
#> 1957 404.0000 347.0000 332.3090 336.0000
#> 1958 404.0000 359.0000 310.0000 337.0000
#> 1959 463.0000 407.0000 362.0000 388.2858
#> 1960 508.0000 461.0000 390.0000 432.0000

# Example 6: Same as example 1, just written with pipe operator
tsAirgap %>% na_kalman()

#>           Jan      Feb      Mar      Apr      May      Jun      Jul      Aug
#> 1949 112.0000 118.0000 132.0000 129.0000 123.5734 135.0000 148.0000 148.0000
#> 1950 115.0000 126.0000 141.0000 135.0000 125.0000 149.0000 170.0000 170.0000
#> 1951 145.0000 150.0000 178.0000 163.0000 172.0000 178.0000 199.0000 199.0000
#> 1952 171.0000 180.0000 193.0000 181.0000 183.0000 218.0000 230.0000 242.0000
#> 1953 196.0000 196.0000 236.0000 235.0000 229.0000 243.0000 264.0000 272.0000
#> 1954 204.0000 188.0000 235.0000 227.0000 234.0000 260.8078 302.0000 293.0000
#> 1955 242.0000 233.0000 267.0000 269.0000 270.0000 315.0000 364.0000 347.0000
#> 1956 284.0000 277.0000 312.6060 311.6160 311.0830 374.0000 413.0000 405.0000
#> 1957 315.0000 301.0000 356.0000 348.0000 355.0000 414.6160 465.0000 467.0000
#> 1958 340.0000 318.0000 382.6912 348.0000 363.0000 435.0000 491.0000 505.0000
#> 1959 360.0000 342.0000 406.0000 396.0000 420.0000 472.0000 548.0000 559.0000
#> 1960 417.0000 391.0000 419.0000 461.0000 452.5683 535.0000 622.0000 606.0000
#>           Sep      Oct      Nov      Dec
#> 1949 132.7833 119.0000 104.0000 118.0000
#> 1950 157.1427 133.0000 130.8971 140.0000
#> 1951 184.0000 162.0000 146.0000 166.0000
#> 1952 209.0000 191.0000 172.0000 194.0000
#> 1953 237.0000 211.0000 180.0000 201.0000
#> 1954 259.0000 229.0000 203.0000 229.0000
#> 1955 312.0000 274.0000 237.0000 278.0000
#> 1956 355.0000 306.0000 271.0000 306.0000
#> 1957 404.0000 347.0000 298.6147 336.0000
#> 1958 404.0000 359.0000 310.0000 337.0000
#> 1959 463.0000 407.0000 362.0000 390.9089
#> 1960 508.0000 461.0000 390.0000 432.0000