Python networkx memory error on small dataset

Question

I have a relatively small dataset consisting of ~100k edges and ~7000 nodes. I am trying to use two algorithms to find communities within my dataset. I added the edges and proceeded to run an algorithm that was implemented here: http://perso.crans.org/aynaud/communities/#as-python-module

Data is basically as follows:

Node1,Node2

Node1 is connected to node2 and the graph is undirected.

But the program just runs for a bit then gives me a memory error. I know it works because I tested it on a small dataset and it works properly but it's giving me a memory error. I don't know a lot about these algorithms but I was under the impression that the dataset I have is really small and I am running it on my personal home computer. Is this just a problem with the implementation itself or due to my machine's limitations?

Would any of you have an idea as to how to get this to work (i.e. another piece of software or algorithm) besides running it on a server with a lot more memory?

Thanks!

traceback:

Traceback (most recent call last):
  File "<wingdb_compile>", line 65, in <module>
  File "C:\Python27\lib\site-packages\networkx\drawing\layout.py", line 242, in fruchterman_reingold_layout
    pos=_fruchterman_reingold(A,dim,pos_arr,fixed,iterations)
  File "C:\Python27\lib\site-packages\networkx\drawing\layout.py", line 280, in _fruchterman_reingold
    delta = np.zeros((pos.shape[0],pos.shape[0],pos.shape[1]),dtype=A.dtype)
MemoryError

@Paul: Error seems to happen while trying to draw the network. If you don't need to draw, then remove those part, try again and see if you get an error. — Avaris
– Avaris, Commented Mar 27, 2012 at 20:49
It's actually the layout part of the drawing - which is creating a large numpy matrix in this case - that is filling the memory. You probably won't be able to see much when drawing 7000 nodes and 100K edges anyway...so skip the drawing part and see if it fits in memory then. — Aric
– Aric, Commented Mar 28, 2012 at 17:55

BeeGee · Accepted Answer · 2016-02-10 01:10:12Z

2

I realize this is an old question but NetworkX still has memory issues especially when dealing with algorithms like communities.

Check out graph-tool. It is a python package that boasts much faster performance than NetworkX

I am not affiliated with graph-tool. It has solved memory problems for me in terms of running algorithms on large networks.

Try converting your nodes to integers if you have not done so already. I have found that nodes that are labelled as strings can take up a lot of memory.

Happy networking

answered Feb 10, 2016 at 1:10

BeeGee

8753 gold badges18 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python networkx memory error on small dataset

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related