AWS javascript SDK Kinesis getShardIterator Error: UnexpectedParameter: Unexpected key 'Timestamp' found in params

Question

I am using the aws-sdk(version 2.1.8) to get records out from Kinesis. This is within a node Lambda function. Trying to use ShardIteratorType: AT_TIMESTAMP. I supply a Timestamp in the params object

const AWS = require("aws-sdk");
const kinesis = new AWS.Kinesis({ region: 'us-east-1' });
var params = {
    ShardId: shard.ShardId, /* required */
    ShardIteratorType: 'AT_TIMESTAMP', /* required */
    StreamName: process.env.STREAM_NAME, /* required */
    Timestamp: new Date(2017, 11, 08, 14, 32, 51)
};

kinesis.getShardIterator(params, function (err, data) {
    if (err) {
        return defer.reject(err);
    }
    defer.resolve(data);
});

But I get this error: Unexpected key 'Timestamp' found in params. Checked google, SO, and aws forums, to no avail. I am following the documentation from AWS: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Kinesis.html#getShardIterator-property

Here are the logs from the Lambda function where I use getShardIterator

2017-11-08T15:53:13.551Z    ec3ef291-c49c-11e7-ad35-473206669891    SHARD 
ITERATOR TYPE:  AT_TIMESTAMP
2017-11-08T15:53:13.551Z    ec3ef291-c49c-11e7-ad35-473206669891    
TIMESTAMP:  2017-12-08T14:32:51.000Z

ShardIteratorTypes LATEST and TRIM_HORIZON work fine. It is only when I try to use AT_TIMESTAMP that the issue occurs.

Can you include the code that is calling the SDK, passing the params variable? — jarmod
– jarmod, Commented Nov 8, 2017 at 15:09
added the aws kinesis library init & call to getShardIterator — Blighty
– Blighty, Commented Nov 8, 2017 at 15:26
Using a recent aws-sdk version (2.145.0), I don't have any problems passing the Timestamp parameter. It does fail with InvalidParameterType when passing the timestamp in the format you've specified but supplying the following seems to be acceptable: Timestamp: new Date(2017, 11, 08, 14, 32, 51). — jarmod
– jarmod, Commented Nov 8, 2017 at 15:40
@jarmod I tested using your suggested value for Timestamp - still get the same error. I have updated my post with the logs output. It shows that I have valid values for ShardIteratorType and Timestamp. This is all happening inside a Lambda function, if that means anything. — Blighty
– Blighty, Commented Nov 8, 2017 at 15:58

Blighty · Accepted Answer · 2017-11-14 15:58:37Z

0

I found what was going on. The aws-sdk is taking the Timestamp value and multiplying it by 1000. This is insane - nowhere is this mentioned in the docs, where it simply states

The timestamp of the data record from which to start reading. Used with shard iterator type AT_TIMESTAMP. A timestamp is the Unix epoch date with precision in milliseconds

Therefore, using this

Timestamp: 1510673368611

will fail, even though it is a valid timestamp in milliseconds. You have to divide by 1000

Timestamp: 1510673368.611

This was pretty infuriating to discover. Just validate the value I send, don't alter it and then validate the result. Or, you know, MENTION THAT THIS HAPPENS IN THE DOCS.

answered Nov 14, 2017 at 15:58

Blighty

1995 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

AWS javascript SDK Kinesis getShardIterator Error: UnexpectedParameter: Unexpected key 'Timestamp' found in params

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related