aws cli dynamo db (ValidationException) Error

Question

I'm looking to batch write item to dynamodb using python's boto3 module and i'm getting this. This is the first time i've ever worked with aws cli or boto3. The documentation says validation exception errors occur when there are empty values and possible incorrect data types, but i've played with all those and it doesn't seem to be working.

Does dynamodb only like to have 25 items written to it at a time? how can i control those batches if so?

My request:

client = boto3.client('dynamodb')
response = client.batch_write_item(RequestItems=batch_dict)

Top of batch_dict:

{'scraper_exact_urls': [{'PutRequest': {'Item': {'Sku': {'S': 'T104P3'},
 'pps_id': {'N': '427285976'},
 'scraper_class_name': {'S': 'scraper_class_name'},
 'store_id': {'N': '1197386754'},
 'updated_by': {'S': 'user'},
 'updated_on': {'N': '1480714223'},
 'updated_url': {'S': 'http://www.blah.com'}}}},
 {'PutRequest': {'Item': {'Sku': {'S': 'T104P3'},
 'pps_id': {'N': '427285976'},
 'scraper_class_name': {'S': 'scraper_class_name'},
 'store_id': {'N': '1197386754'},
 'updated_by': {'S': 'user'},
 'updated_on': {'N': '1480714223'},
 'updated_url': {'S': 'http://www.blah.com'}}}},....

Schema:

attributes: "pps_id"=>\Aws\DynamoDb\Enum\Type::NUMBER, "sku"=>\Aws\DynamoDb\Enum\Type::STRING, "scraper_class_name"=>\Aws\DynamoDb\Enum\Type::STRING, "store_id"=>\Aws\DynamoDb\Enum\Type::NUMBER, "updated_url"=>\Aws\DynamoDb\Enum\Type::STRING, "updated_by"=>\Aws\DynamoDb\Enum\Type::STRING, "updated_on"=>\Aws\DynamoDb\Enum\Type::NUMBER, fields: "pps_id", "scraper_class_name",

The Error:

ClientError: An error occurred (ValidationException) when calling the    BatchWriteItem operation: 1 validation error detected: Value .... Map value   must satisfy constraint: [Member must have length less than or equal to 25,   Member must have length greater than or equal to 1]

Possible duplicate of How to write more than 25 items/rows into Table for DynamoDB? — LuFFy
– LuFFy, Commented Mar 9, 2017 at 9:09

Community · Accepted Answer · 2017-05-23 12:32:29Z

13

The BatchWriteItem API works on 25 items at a time. You could use the following code, adapted from the non-copying batching question, to call BatchWriteItem on 25 item chunks

def batch(iterable, n=1):
    l = len(iterable)
    for ndx in range(0, l, n):
        yield iterable[ndx:min(ndx + n, l)]

client = boto3.client('dynamodb')

for x in batch(batch_dict['scraper_exact_urls'], 25):
    subbatch_dict = {'scraper_exact_urls': x}
    response = client.batch_write_item(RequestItems=subbatch_dict)

edited May 23, 2017 at 12:32

CommunityBot

11 silver badge

answered Mar 5, 2017 at 15:56

Alexander Patrikalakis

5,2251 gold badge35 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

omega_prime Over a year ago

Had to do a few adjustments for it to work, as otherwise it was incomplete code for a pre-beginner python user like myself

gildniy · Accepted Answer · 2020-12-17 17:34:24Z

And here is the Javascript version of batch writing/deleting many items, in case someone needs it:

const batchWriteManyItems = async (tableName, itemObjs, chunkSize = 25) => {

        const buildParams = (table) => JSON.parse(`{"RequestItems": {"${table}": []}}`)

        const queryChunk = (arr, size) => {
            const tempArr = []
            for (let i = 0, len = arr.length; i < len; i += size) {
                tempArr.push(arr.slice(i, i + size));
            }

            return tempArr
        }

        await Promise.all(queryChunk(itemObjs, chunkSize).map(async chunk => {
            let itemParams = buildParams(tableName);
            itemParams.RequestItems[tableName] = chunk
            await dynamoDB.batchWriteItem(itemParams).promise()
        }))
    }

NOTE: the itemObjs attribute can take {PutRequest: {Item: ...} for write items and {DeleteRequest: {Item: ...} to delete items

Collectives™ on Stack Overflow

aws cli dynamo db (ValidationException) Error

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related