I have been writing applications lately in c# that use a ton of memory or stack overflow due to processing extremely large amounts of data in fun ways. Is there a language better suited for this type of thing? Would I benefit from learning a different language (other than c++) to do this?
Add a comment
|
4 Answers
C# isn't the problem. You may need to reconsider the "fun ways" you're handling memory and data. Provide specific scenarios and questions here to get specific answers and alternatives to potentially-problematic methods and strategies you may be using in your application(s).
Comments
If running on a 32bit system .Net will start giving you out of memory exceptions when you consume ~800mb. This is because it need to allocate continuous blocks of memory. If you have an array or list which needs to be expanded, it will copy the old content to a new one, thus having two instances allocated at the same time.
If you can run 64bit, then you will hit your exceptions on anything from ~2GB and above, all depending on how your application works, and what else is running.
For data larger than your physical memory, I would recommend either memory mapped files, or doing some disk/memory swapping.
Comments
If you are working with large data sets and doing functional manipulation, you might consider looking into a functional language like F# or Haskell.
The will not suffer as readily from recursive issues.
However these languages wont substitute for a good design and attention to how you are doing your operations. Its possible that C# is completely well suited to your problem you might just need to refactor how you are handling the problem space.
Comments
IDL (Interactive Data Language) is specially suited for large, matrix-like sets of data. You must, however, pay attention to using matrix or vector operations and not sequential loops.
If licensing is a problem you can try the free clone GDL, although it may not be as fast as IDL.
How large is your data?
