I wrote something very similar at one time where I needed to write business logic to do a reasonably good job with matching various nick names and diminutive first names to person records with their full legal name. Eg. (James Smith = Jim Smith = Jimmy Smith etc...)

I resolved the problem by essentially compiling and slowly building a CSV file that had a proper name, and each comma delimited value after the first value on a line represented every known diminutive name I could think of. For purposes of memory consumption and processing time I preloaded the data of the CSV file into a map of string sets.

I created a unique key for each map entry for every single first name, proper and diminutive. Why? So that I could easily and quickly fetch *every possible proper and diminutive name by any given proper or dimunitive name. I used the singleton pattern because obviously this object is expensive to build and consumes quite a deal of memory, however once this object is constructed it is insanely quick to fetch not only the proper name from a diminutive name, but also to fetch other possible diminutive names.

As the system encounters new dimunitive names it will proceed to slowly add new matches and increase the success rate.

If the data set in question is relatively manageable, and memory is cheap for your scenario, you might find this to be an excellent design idea.

Unfortunately I don't think you are going to find a catch all regex pattern that is going to arbitrarily find matches like this in a data set. Their are no clear rules to how any two words can compare in your situation. The best way to do this is to pre-associate the data once, or to pre associate manageable or indexed chunks of the data all at once.

EDIT:

If you are dealing with an enormous set of data then obviously the above will not work. In this case I advise looking into a massively parallel solution utilizing a Map Reduce algorithm. Apache Hadoop is a great open source project that manages a lot of the complexity of these kinds of applications for JVM based languages.

I wrote something very similar at one time where I needed to write business logic to do a reasonably good job with matching various nick names and diminutive first names to person records with their full legal name. Eg. (James Smith = Jim Smith = Jimmy Smith etc...)

I resolved the problem by essentially compiling and slowly building a CSV file that had a proper name, and each comma delimited value after the first value on a line represented every known diminutive name I could think of. For purposes of memory consumption and processing time I preloaded the data of the CSV file into a map of string sets.

I created a unique key for each map entry for every single first name, proper and diminutive. Why? So that I could easily and quickly fetch *every possible proper and diminutive name by any given proper or dimunitive name. I used the singleton pattern because obviously this object is expensive to build and consumes quite a deal of memory, however once this object is constructed it is insanely quick to fetch not only the proper name from a diminutive name, but also to fetch other possible diminutive names.

As the system encounters new dimunitive names it will proceed to slowly add new matches and increase the success rate.

If the data set in question is relatively manageable, and memory is cheap for your scenario, you might find this to be an excellent design idea.

Unfortunately I don't think you are going to find a catch all regex pattern that is going to arbitrarily find matches like this in a data set. Their are no clear rules to how any two words can compare in your situation. The best way to do this is to pre-associate the data once, or to pre associate manageable or indexed chunks of the data all at once.

EDIT:

If you are dealing with an enormous set of data then obviously the above will not work. In this case I advise looking into a massively parallel solution utilizing a Map Reduce algorithm. Apache Hadoop is a great open source project that manages a lot of the complexity of these kinds of applications for JVM based languages. I am not sure if such a project exists for Python.