Perform multiple calculations among columns using lambda function in pandas dataframe

You can use the Pandas mask function which is more efficient that using `apply' with a function. To make a simple example:

import pandas as pd

df = pd.DataFrame({ 'current_year': [2022, 2023, 2022, 2023, 2022, 2023],
                   'lease_commence_date': [1969, 1974, 2000, 1998, 1983, 1992],
                    'remaining_lease': [50, None, 60 , 55, None, None],
                   })

df['remaining_lease'] = (df['remaining_lease']
                         .mask(df['remaining_lease'].isna(),
                        99 - (df['current_year'] - df['lease_commence_date']))
                        .astype(int))

print(df)

which gives:

   current_year  lease_commence_date  remaining_lease
0          2022                 1969               50
1          2023                 1974               50
2          2022                 2000               60
3          2023                 1998               55
4          2022                 1983               60
5          2023                 1992               68

The casting to int is needed as the None/NaN values will result in a column of float values.

You can use pandas columns operation.

df['remaining_lease'] = df['remaining_lease'].fillna(99 - ( df["current_year"] - df["lease_commense_date"]))

The spike comes from a N**2 memory overhead of columns:

One problem of your code is that: df['remaining_lease'].apply is a Series.apply and goes over each entry in that Series/column. No problem so far but df['current_year'], df['lease_commence_date'] in the lambda function are the whole columns which are constant.
Then, in remaining_lease_year function, i.e lease_year = 99 - (current_year - commense_year) is a Series and not a single value! These than are tried to be stored as whole Series of len(df) into each cell and therefore accumulate until no more memory is free.

One Series is about 927465 * 8 / 1_000_000 = 7.4 MB = 7.08 MiB, take that * 927465 = 6.8 TB

What you can do instead is:

def remaining_lease_year(row):
    x = row["remaining_lease"]
    if math.isnan(x): # if the value is nan
        lease_year = 99 - (row["current_year"] - row["lease_commence_date"])
        return lease_year
    else: #if the value is not nan
        return x

df['remaining_lease'] = df.apply(remaining_lease_year, axis=1)
# Output with the values provided by user19077881:
# df['remaining_lease']:
# 0    50.0
# 1    50.0
# 2    60.0
# 3    55.0
# 4    60.0
# 5    68.0

For a one line solution, you can just use this.

df['remaining_lease'] = df.apply(lambda x: 99 - (x.current_year - x.commense_year) if math.isnan(x.remaining_lease) else x.remaining_lease, axis=1)