Performance issue loading data from database using EF5 and looping using foreach

Question

I am building a statistics (multi-agents) application, I'm using EF5 to retrive a list of agents from agents table. it's a very large amount of data (arround 50 000 agents per region) needed to be loaded at a time to proceed some immigration logic on them. I found that the (Region).Agents navigation property is very very slow to load, but, executing the same query takes just 1 sec on Sql server. and the foreach too takes a considerable time looping.

foreach (OrigineAgent origineAgent in new List<Agent> Origine.Agents.AsParallel()))
{
    origineAgent.ScoreCalculator = _container.Resolve<IScoreCalculator>();
    origineAgent.AgentExoVariables = AgentVars;
    _migrationManager.Migrate(origineAgent);
}

What do I have to do to increase data loading and looping performance?

Move score calculator resolving out from the loop

Sergey Berezovskiy
– Sergey Berezovskiy

2013-04-13 12:01:30 +00:00
Commented Apr 13, 2013 at 12:01 — Sergey Berezovskiy
– Sergey Berezovskiy, Commented Apr 13, 2013 at 12:01

MarcinJuraszek · Accepted Answer · 2013-04-13 12:04:13Z

2

How about that:

var resolved = _container.Resolve<IScoreCalculator>();
foreach (OrigineAgent origineAgent in Origine.Agents.ToList().AsParallel())
{
    origineAgent.ScoreCalculator = resolved
    origineAgent.AgentExoVariables = AgentVars;
    _migrationManager.Migrate(origineAgent);
}

I've moved _container.Resolve<IScoreCalculator>(); before the loop and fixed your foreach loop source, because the one from your question does not even compile.

answered Apr 13, 2013 at 12:04

MarcinJuraszek

126k15 gold badges200 silver badges267 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

HichemSeeSharp Over a year ago

You're right, but your answer still not resolving performance issue of agents list from database

bytefire Over a year ago

How about starting a separate task for _migrationManager.Migrate(originAgent) and store the task instance in an array. Then at the end of the foreach loop call WaitAll() on that array of tasks?

bytefire · Accepted Answer · 2013-04-15 09:28:20Z

1

Building on MarcinJuraszek's post above, how about following:

List<Task> tasks = new List<Task>();
var resolved = _container.Resolve<IScoreCalculator>();
foreach (OrigineAgent origineAgent in Origine.Agents.ToList().AsParallel())
{
    origineAgent.ScoreCalculator = resolved
    origineAgent.AgentExoVariables = AgentVars;
    Task t = Task.Factory.StartNew((arg) => { _migrationManager.Migrate((OrigineAgent)arg); }, origineAgent);
    tasks.Add(t);
}
Task.WaitAll(tasks.ToArray());

It parallelises calls to _migrationManager.Migrate(origineAgent);

answered Apr 15, 2013 at 9:28

bytefire

4,4702 gold badges32 silver badges37 bronze badges

Collectives™ on Stack Overflow

Performance issue loading data from database using EF5 and looping using foreach

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related