Replace Container.CreateItemAsync with Container.CreateTransactionalBatch

Question

I want to use Container.CreateTransactionlBatch for bulk insert operation. Currently I am doing this with container.CreateItemAsync method which is taking more time.

Is it possible to replace container.CreateItemAsync with Container.CreateTransactionalBatch?

private async Task AddSubscription(EnableOrDisableSubscriptionCommand command, SubscriptionAction subscriptionAction, IList<int> notificationCategoryTypes)
        {
            List<Task> bulkOperations = new List<Task>();
            foreach (var notificationCategory in notificationCategoryTypes)
            {
                var notificationTypes = Utility.GetNotificationTypes((NotificationCategoryType)notificationCategory);

                foreach (var notificationType in notificationTypes)
                {
                    foreach (var payerAccountSubscriptions in command.Subscriptions)
                    {
                        if (payerAccountSubscriptions.AccountNumbers?.Any() ?? false)
                        {
                            foreach (var accountNumber in payerAccountSubscriptions.AccountNumbers.Where(a => !string.IsNullOrEmpty(a)))
                            {
                                bulkOperations.Add(_repository.Create(subscriptionAction, notificationType,
                                      payerAccountSubscriptions.ColCoId, payerAccountSubscriptions.PayerNumber, accountNumber, command.UserRole,
                                      command.UserId));
                            }
                        }
                        else
                        {
                            bulkOperations.Add(_repository.Create(subscriptionAction, notificationType,
                                payerAccountSubscriptions.ColCoId, payerAccountSubscriptions.PayerNumber, null, command.UserRole,
                                command.UserId));

                        }
                    }
                }
            }
            await Task.WhenAll(bulkOperations);
        }

 public async Task<ItemResponse<Subscription>> Create(SubscriptionAction subscriptionAction, NotificationType notificationType,
            int colCoId, string payerNumber, string accountNumber, UserRole userRole, string userId, string cardId = null)
        {
            var eventType = Utility.GetEventType(notificationType);

            var subscriptionBase = new Subscription
            {
                Id = Guid.NewGuid(),
                IsActive = true,
                Action = subscriptionAction,
                ActionDesc = subscriptionAction.ToString(),
                Version = (int)SubscriptionVersion.V2,
                NotificationType = notificationType,
                NotificationTypeDesc = notificationType.ToString(),
                EventType = eventType,
                EventTypeDesc = eventType.ToString(),
                ColCoId = colCoId,
                PayerNumber = payerNumber,
                AccountNumber = accountNumber,
                CardId = cardId,
                DistributionGroups = new List<string> { userRole.ToString() },
                DistributionUserIds = new List<string> { userId }
            };
            return await CreateItemAsync(subscriptionBase);
        }

public async Task<ItemResponse<T>> CreateItemAsync(T item)
        {
            return await _container.CreateItemAsync<T>(item);
        }

Are you trying to do bulk inserts or are you trying to do a transactional batch? These are not the same thing. — Mark Brown
– Mark Brown, Commented Aug 6, 2020 at 15:21

Matias Quaranta · Accepted Answer · 2020-08-06 15:31:54Z

2

There are two different things.

TransactionalBatch is meant for scenarios where you have a group of operations that you need to commit as a single unit of work, either they all succeed or they all fail. If one of the Batch operations fails, all fail regardless. Reference: https://devblogs.microsoft.com/cosmosdb/introducing-transactionalbatch-in-the-net-sdk/
Bulk operations, which is meant for throughput-optimized scenarios (you want to take advantage of the provisioned throughput and use it completely or as much as possible to push through a medium-to-high volume of data). Reference: https://devblogs.microsoft.com/cosmosdb/introducing-bulk-support-in-the-net-sdk/

By the code you are sharing it looks like you are trying to use Bulk? If so, you need to make sure your CosmosClient instance is created with a CosmosClientOptions that has Bulk on:

CosmosClientOptions options = new CosmosClientOptions() { AllowBulkExecution = true };
CosmosClient cosmosclient = new CosmosClient(connectionString, options);

In your code also, you might want to remove the await from the other methods, and just return the Task, for example:

public Task<ItemResponse<Subscription>> Create(SubscriptionAction subscriptionAction, NotificationType notificationType,
    int colCoId, string payerNumber, string accountNumber, UserRole userRole, string userId, string cardId = null)
{
    // ... OTHER CODE
    return CreateItemAsync(subscriptionBase);
}

public Task<ItemResponse<T>> CreateItemAsync(T item)
{
    return _container.CreateItemAsync<T>(item);
}

answered Aug 6, 2020 at 15:31

Matias Quaranta

15.8k1 gold badge25 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Rakesh Kumar Over a year ago

Thanks! for your reply. Now I am able insert 12,000 items and its take more than 3 mins to insert. How Can I improve the insertion performance. Could you please suggest.

Matias Quaranta Over a year ago

Verify the provisioned throughput (RU/s) is enough to process the volume of data you want to push through, you can check if you are getting throttles (429s) on the metrics. Also verify the CPU on the instance is not spiking up (the bottleneck in that case could be the local CPU). Please mark the answer if it helped solve the initial issue.

Matias Quaranta Over a year ago

Also make sure you are using the latest SDK version, we have performance improvements on each release.

Rakesh Kumar Over a year ago

I have set 3000 Throughtput value for the subscription container, but still i m getting "Response status code does not indicate success: TooManyRequests (429); Substatus: 3200;" in Azure Function logs

Rakesh Kumar Over a year ago

How much Throughput value i need to set to insert/delete 10-12k items in bulk?

|

Collectives™ on Stack Overflow

Replace Container.CreateItemAsync with Container.CreateTransactionalBatch

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related