Python MultiProcessing and Directory Creation

It is best for the worker function to only rely on the single argument it gets for determining what to do. Because that is the only information it gets from the parent process every time it is called. This argument can be almost any Python object (including a tuple, dict, list) so you're not really limited in the amount of information you pass to a worker.

So make a list of 2-tuples. Each 2-tuple should consist of (1) the file to get and (2) the directory where to stash it. Feed that list of tuples to map(), and let it rip.

I'm not sure if it is useful to specify the number of processes you want to use. Pool generally uses as many processes as your CPU has cores. That is usually enough to max out all the cores. :-)

BTW, you should only call map() once. And since map() blocks until everything is done, there is no need to call join().

Edit: Added example code below.

import multiprocessing
import requests
import os

def processfile(arg):
    """Worker function to scrape the pages and write them to a file.

    Keyword arguments:
    arg -- 2-tuple containing the URL of the page and the directory
           where to save it.
    """
    # Unpack the arguments
    url, savedir = arg

    # It might be a good idea to put a random delay of a few seconds here, 
    # so we don't hammer the webserver!

    # Scrape the page. Requests rules ;-)
    r = requests.get(url)
    # Write it, keep the original HTML file name.
    fname = url.split('/')[-1]
    with open(savedir + '/' + fname, 'w+') as outfile:
        outfile.write(r.text)

def main():
    """Main program.
    """
    # This list of tuples should hold all the pages... 
    # Up to you how to generate it, this is just an example.
    worklist = [('http://www.foo.org/page1.html', 'dir1'), 
                ('http://www.foo.org/page2.html', 'dir1'), 
                ('http://www.foo.org/page3.html', 'dir2'), 
                ('http://www.foo.org/page4.html', 'dir2')]
    # Create output directories
    dirlist = ['dir1', 'dir2']
    for d in dirlist:
        os.makedirs(d)
    p = Pool()
    # Let'er rip!
    p.map(processfile, worklist)
    p.close()

if __name__ == '__main__':
    main()

Multiprocessing, as the name implies, uses separate processes. The processes you create with your Pool do not have access to the original values of a and b that you are adding 500 to in the main program. See this previous question.

The easiest solution is to just refactor your code so that you pass a and b to fetchAfter (in addition to passing y).

Here's one way to implement it:

#!/usr/bin/env python
import logging
import multiprocessing as mp
import os
import urllib

def download_page(url_path):
    try:
        urllib.urlretrieve(*url_path)
        mp.get_logger().info('done %s' % (url_path,))
    except Exception as e:
        mp.get_logger().error('failed %s: %s' % (url_path, e))

def generate_url_path(rootdir, urls_per_dir=500):
    for i in xrange(100*1000):
        if i % urls_per_dir == 0: # make new dir
           dirpath = os.path.join(rootdir, '%d-%d' % (i, i+urls_per_dir))
           if not os.path.isdir(dirpath):
              os.makedirs(dirpath) # stop if it fails
        url = 'http://example.com/page?' + urllib.urlencode(dict(number=i))
        path = os.path.join(dirpath, '%d.html' % (i,))
        yield url, path

def main():
    mp.log_to_stderr().setLevel(logging.INFO)

    pool = mp.Pool(4) # number of processes is unrelated to number of CPUs
                      # due to the task is IO-bound
    for _ in pool.imap_unordered(download_page, generate_url_path(r'E:\A\B')):
        pass

if __name__ == '__main__':
   main()

See also Python multiprocessing pool.map for multiple arguments and the code
Brute force basic http authorization using httplib and multiprocessing from how to make HTTP in Python faster?