Using Python to generate a connection/network graph

Question

I have a text file with about 8.5 million data points in the form:

Company 87178481
Company 893489
Company 2345788
[...]

I want to use Python to create a connection graph to see what the network between companies looks like. From the above sample, two companies would share an edge if the value in the second column is the same (clarification from/for Hooked).

I've been using the NetworkX package and have been able to generate a network for a few thousand points, but it's not making it through the full 8.5 million-node text file. I ran it and left for about 15 hours, and when I came back, the cursor in the shell was still blinking, but there was no output graph.

Is it safe to assume that it was still running? Is there a better/faster/easier approach to graph millions of points?

How are the companies connected? I.e. is an edge shared between company A and B if the second column is the same? — Hooked
– Hooked, Commented Oct 25, 2012 at 16:28
Can't say I've had any problems with 8.5million in networkx. How many distinct vertices do you have? Are you using directed/un-directed? Also, when you say "no output graph" - what exactly do you mean? [eg, you haven't tried to print it or something] — Jon Clements
– Jon Clements, Commented Oct 25, 2012 at 16:40
I just mean that I don't see a generated graph...there's no new window that pops up with the drawn graph. — Jon
– Jon, Commented Oct 25, 2012 at 16:58

Hooked · Accepted Answer · 2012-10-25 16:34:34Z

5

If you have 1000K points of data, you'll need some way of looking at the broad picture. Depending on what you are looking for exactly, if you can assign a "distance" between companies (say number of connections apart) you can visualize relationships (or clustering) via a Dendrogram.

Scipy does clustering:

http://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html#module-scipy.cluster.hierarchy

and has a function to turn them into dendrograms for visualization:

http://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.dendrogram.html#scipy.cluster.hierarchy.dendrogram

An example for a shortest path distance function via networkx:

http://networkx.lanl.gov/reference/generated/networkx.algorithms.shortest_paths.generic.shortest_path.html#networkx.algorithms.shortest_paths.generic.shortest_path

Ultimately you'll have to decide how you want to weight the distance between two companies (vertices) in your graph.

answered Oct 25, 2012 at 16:34

Hooked

88.9k46 gold badges197 silver badges271 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Jon Over a year ago

Is there a simpler or preferred way to do this network building in SAS or R?

Hooked Over a year ago

@Jon This answer (even though links are provided) is language agnostic. What are you looking to show with your graph of a million points? General connections, disparate clustering, central hubs? It is unclear what you'd like to get out of your data set as many different questions can be asked of it.

Jon Over a year ago

It is a little vague. I'd like to see clusters and connection points between clusters. The idea is to utilize the data for network outreach to see where singular connections exist between a master cluster and a smaller cluster. These singular business connections can then be utilized for more targeted marketing purposes, etc.

Hooked Over a year ago

@Jon This is a very general question in graph theory (and there is a huge volume of research on the subject!) but I'll point out a few things. networkx as visualization is irrelevant, with this many points you need an algorithmic approach. I would first cluster the vertices (see answer) by some threshold. Then by looking at the dendrogram I would see which clusters are joined when raising the threshold slightly. Those new connections are your "singular business connections".

Jon Over a year ago

I think you've helped me identify the scope of this. Thanks, Hooked.

Roja · Accepted Answer · 2012-10-25 18:10:27Z

You have too many datapoints and if you did visualize the network it won't make any sense. You need to have ways to 1)reduce the number of companies by removing those that are less important/less connected 2)summarize the graph somehow and then visualize.

to reduce the size of data it might be better to create the network independently (using your own code to create an edgelist of companies). This way you can reduce the size of your graph (by removing singletons for example, which may be many).

For summarization I recommend running a clustering or a community detection algorithm. This can be done very fast even for very large networks. Use the "fastgreedy" method in the igraph package: http://igraph.sourceforge.net/doc/R/fastgreedy.community.html (there is a faster algorithm available online as well, this is by Blondel et al: http://perso.uclouvain.be/vincent.blondel/publications/08BG.pdf I know their code is available online somewhere)

Collectives™ on Stack Overflow

Using Python to generate a connection/network graph

2 Answers 2

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related