Time series imputation using transformers and LLMs

TL;DR

Yes. You can frame multivariate time-series imputation as a masked-prediction task and use a Transformer to reconstruct missing values.

If your data is small, structured, and fixed-length, start with interpolation or GRU/autoencoder methods.

Use Transformers when:

• You need to capture long-range or cross-variable dependencies
• Sampling is irregular or heterogeneous
• You want to leverage transfer learning or pretrained representations

Longer answer

In practice, start with strong baselines (interpolation, Kalman filters, KNN/RandomForest, GRU-D/BRITS). Use Transformers when you have long-range dependencies, many variables, or irregular sampling and enough data.

When to use what

• Simple and strong baselines

  • Linear/spline interpolation, moving averages
  • State-space models with Kalman smoothing
  • kNN or tree-based imputers using local context
  • Libraries: scikit-learn SimpleImputer/KNNImputer, statsmodels state space

• Sequence models built for missingness

  • GRU-D: learns decay toward empirical means for missing channels
    Paper: Che et al., 2018
  • BRITS: bidirectional RNN with consistency losses for imputation
    Paper: Cao et al., 2018

• Transformer-style

  • Masked modeling: randomly mask observed values and train the model to reconstruct them
  • Works well for long horizons, many variables, or heterogeneous signals
  • Examples:
    • TST (Time Series Transformer): paper
    • TS2Vec (representation learning usable for imputation): paper
    • iTransformer (variable-wise attention)
  • Diffusion-style imputers for uncertainty:
    • CSDI (Conditional Score-based Diffusion for Imputation): paper