Python pandas: how to vectorize this function

One vectorization approach using only pandas is to do a cartesian join between the two frames and subset. Would start out like:

df['dummy'] = 1
evol['dummy'] = 1
combined = df.merge(evol, on='dummy')
# filter date ranges, multiply etc

This will likely be faster than what you are doing, but is memory inefficient and might blow up on your real data.

If you can take on the numba dependency, something like this should be very fast - essentially a compiled version of what you are doing now. Something similar would be possible in cython as well. Note that this requires that the evol dataframe is sorted and contigous by year, that could be relaxed with modification.

import numba

@numba.njit
def f(data, year_final, year_init, evol_year, evol_factor):
    data = data.copy()
    for i in range(len(data)):
        year_pos = np.searchsorted(evol_year, year_init[i])
        n_years = year_final[i] - year_init[i]
        for offset in range(n_years):
            data[i] += data[i] * evol_factor[year_pos + offset]            
    return data

f(df['data'].values, df['year_final'].values, df['year_init'].values, evol.index.values, evol['evolution'].values)
Out[24]: array([   163673, 594596044,      1277], dtype=int64)

Edit: Some timings with your test data

In [25]: %timeit f(df['data'].values, df['year_final'].values, df['year_init'].values, evol.index.values, evol['evolution'].values)
15.6 µs ± 338 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)


In [26]: %%time
    ...: for index, row in df.iterrows():
    ...:     for year in range(row['year_init'], row['year_final']):
    ...:         factor = evol.at[year, 'evolution']
    ...:         df.at[index, 'data'] += df.at[index, 'data'] * factor
Wall time: 3 ms