using the Take inside a while loop

Question

i have a collection with thousands of elements, i do a for loop to process 100 elements at a time and then upload them and pause for 5 seconds

while (EntriesList.Count > 0)
            {
                service.InsertEntries(EntriesList.Take(100).ToList<Entry>());
                System.Threading.Thread.Sleep(5000);
            }

service.InsertEntries takes a List<Entry> as a parameter, also is there a better way to achive such thing?

normally i go with a for loop with a % operator and a temp list, is the above the same?

I usually write myself a AsChunked(int batchSize) method that converts a list into a sequence of batches. — usr
– usr, Commented Nov 6, 2013 at 12:21
ps: "Take" function does not remove elements from the list => Your 'while' will never remove anything from EntriesList. Normal ? — Olivier
– Olivier, Commented Nov 6, 2013 at 12:23
unless there is another thread modifying it... but I can see kind of a conccurency issue :) — Olivier
– Olivier, Commented Nov 6, 2013 at 12:26

Rik · Accepted Answer · 2013-11-06 12:25:24Z

3

This will insert the first 100 entries again and again.

Keep track of where you are and use use Skip to move forward:

int current = 0;
int batchSize = 100;

while (current < EntriesList.Count)
{
    service.InsertEntries(EntriesList.Skip(current).Take(batchSize).ToList());
    current += batchSize;
    System.Threading.Thread.Sleep(5000);
}

answered Nov 6, 2013 at 12:25

Rik

29.3k14 gold badges52 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 12:29:14Z

For this kind of thing you should write a Chunkify extension method. One good example you can get from Marc here. But due to the question it always returns an array with a fixed size and maybe null entries.

So maybe this implementation matches better your requirements:

/// <summary>
/// Divides an enumeration into smaller (same-sized) chunks.
/// </summary>
/// <typeparam name="T">The type of the elements within the sequence.</typeparam>
/// <param name="source">The sequence which should be breaked into chunks.</param>
/// <param name="size">The size of each chunk.</param>
/// <returns>An IEnumerable&lt;T&gt; that contains an IEnumerable&lt;T&gt; for each chunk.</returns>
public static IEnumerable<IEnumerable<T>> Chunkify<T>(this IEnumerable<T> source, int size)
{
    if (source == null)
    {
        throw new ArgumentNullException("source");
    }

    if (size < 1)
    {
        throw new ArgumentOutOfRangeException("size");
    }

    return ChunkifyImpl(source, size);
}

private static IEnumerable<IEnumerable<T>> ChunkifyImpl<T>(IEnumerable<T> source, int size)
{
    using (var iter = source.GetEnumerator())
    {
        while (iter.MoveNext())
        {
            var chunk = new List<T>(size);
            chunk.Add(iter.Current);

            for (int i = 1; i < size && iter.MoveNext(); i++)
            {
                chunk.Add(iter.Current);
            }

            yield return chunk;
        }
    }
}

Now you can use this extension as follows:

foreach(var chunk in EntriesList.Chunkify(100))
{
    service.InsertEntries(chunk);
    Thread.Sleep(5000);
}

Peter · Accepted Answer · 2013-11-06 12:24:11Z

2

EntriesList.Take(100).ToList() will create a new list of 100 entries, you will not remove the 100 entries from your EntriesList.

You can do something like:

List<TYPE> objectsforupload = EntriesList.ToList();
while (objectsforupload.Count > 0)
{
        List<TYPE> uploadlist = objectsforupload.Take(100).ToList();
        objectsforupload = objectsforupload.Where(x => !uploadlist.Contains(x)).ToList();
        service.InsertEntries(objectsforupload);
        System.Threading.Thread.Sleep(5000);
}

answered Nov 6, 2013 at 12:24

Peter

28k8 gold badges69 silver badges85 bronze badges

Comments

Benoit Blanchon · Accepted Answer · 2013-11-06 12:26:10Z

2

Call this after the .Take()

EntriesList.RemoveRange(0, Math.Min(EntriesList.Count, 100));

It will remove the first 100 elements.

EDIT: more complete example:

while (EntriesList.Count > 0)
{
    var poppedEntries = EntriesList.Take(100).ToList<Entry>();
    EntriesList.RemoveRange(0, poppedEntries.Count);
    service.InsertEntries(poppedEntries);
    System.Threading.Thread.Sleep(5000);
}

answered Nov 6, 2013 at 12:26

Benoit Blanchon

14.7k5 gold badges80 silver badges93 bronze badges

Comments

user180326 · Accepted Answer · 2013-11-06 12:28:12Z

1

As it stands, your code does not work, you should also Skip by the same amount and

        while (entries.Count > 0)
        {
            service.InsertEntries(entries.Take(100).ToList<Entry>());
            entries = entries.Skip(100);
            System.Threading.Thread.Sleep(5000);
        }

However, this performs very badly. the IEnumable<> you get back from Skip() will add overhead every time it is called.

So the best advice for this scenario is to use a for loop.

answered Nov 6, 2013 at 12:28

user180326

Collectives™ on Stack Overflow

using the Take inside a while loop

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related