4
$\begingroup$

I have trained an ARIMA model on some 15 minute incremented time series data by using the statsmodels library. I would like to determine how anomalous the next 15 minute increment's data I observe is. I would then like to update the model with that data.

I can predict the range of future outcomes within an accepted error margin like so:

model = arima_model.ARIMA(data.COUNTS,(p,d,q)).fit()
print model.forecast(alpha=.001)[2]

That outputs:

array([[ 16.13395152, 48.47024783]]))

So you could say any number outside that range is anomalous. However, if my next observation were, for example, 50 that would presumably be less anomalous than an observation of 51. Which way do you recommend I use to determine exactly how anomalous an observation is using an ARIMA model?

Also, how should I go about updating my model with the new observation?

Improve this question
$\endgroup$
1
  • $\begingroup$ What package are you using? I cannot find the forecast() function in the ARIMA class. $\endgroup$ Commented Jul 15, 2016 at 9:26

1 Answer 1

Reset to default
1
$\begingroup$

Have you considered using a quantile forecast as your ARIMA?

Quantile forecast is when, instead of predicting the average, you predict quantiles. In your case, you could predict for a given day the probability that an observation is below the 0.5%-percentile and the 99.5%-percentile. That range would define the 1% of "abnormal" values for the next days.

Recapitulating, usually in regression you find the average value by minimizing $\min_\beta \sum_i||X_i\cdot\beta -y_i||^2$.

You could find the median value by minimizing $\min_\beta \sum_i|X_i\cdot\beta -y_i|$.

You can also find any given quantile $\tau$ by minimizing $\min_\beta \sum_{i|y_i\geq X_i\cdot\beta}\tau|X_i\cdot\beta-y_i| + \sum_{i|y_i< X_i\cdot\beta}(1-\tau)|X_i\cdot\beta-y_i|$.

Of course, these minimizations problems become more and more complicated. You can only estimate your $\beta$ for this last one using ADMM or gradient descent.

I don't think you'll find python tools for this. You'll find plenty of packages in R though. For instance, see this page:

One last piece of advise: if you need this forecasting to interoperate with other python code, my recommendation is to do it via CSV files. Just have your python code do the pre-processing and generate a CSV file, then R doing its thing and generating another CSV file, then use this file for the post-processing in python, etc. Some people try to use things like rpy2, but I find it to be overengineering.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.