Marrying disparate datasets in stream

You have to define all the constraints in your data. For example:

• Sample Rate: how often samples are obtained
• Variance: how much uncertainty there is in the sample (in time and magnitude)
• Threshold: consider how much time passes before you declare the data stream dead. Also consider your minimum and maximum values.

Once you've defined that information, understand how sampling works:

• That value remains constant until the next sample is retrieved

Putting it together

Givens:

• GPS sample rate: 1 location every 10 seconds, GPS point is +/- 20m of your actual location (which dissapointingly is a pretty good error bound)
• BPM sample rate: 1 average heart rate every 6 seconds, heart rate is +/- 5 BPM of actual heart rate

You receive data:

0:0:06 118 BPM
0:0:10 34.12345, -77.54321, 100m
0:0:12 119 BPM
0:0:18 117 BPM
0:0:20 34.12354, -77.54312, 99m
...

In your application you can see that by the time we get the first GPS point, the BPM is 118, and when we get the second GPS point the BPM is 117.

You know that because the last sample point was within the normal sample rate so you can simply use that value.

Now, you need to define when the data stream is officially dead. If your last BPM is 40 seconds ago, you probably can't trust that value.

I think what you are looking for here is the basic approach to processing the streams and finding the points that are co-located in time. I'm going to assume you want to do this on a stream of values. If you have this in a relational DB there are ways to do it that are relatively simple to write but probably won't perform well.

Here's the basic approach:

Read the first value from each stream to initialize.

1. Compare the values. If they are close, emit the pair
2. Advance to the next value of the stream whose last read value was earliest
3. Repeat

The main potential complication is that you might find more than value from one of the streams within the range of tolerance of the current value of the other stream. If that's not desirable, you need to either ignore the additional values or find another approach to dealing with that such as averaging.