Merge object arrays results in duplicates using Newtonsoft

I am trying to merge two JObjects using Newtonsoft.

Here are the models.

    public class Users
    {
        public int userId { get; set; }
        public string name { get; set; }

        public List<Role> roles { get; set; }
    }

    public class Role
    {
        public int roleId { get; set; }
        public string role { get; set; }
    }

I created a user object with some sample data as below:

//create a sample user
Users user = new Users()
{
    userId = 1,
    name = "Harsh",
    roles = new List<Role>()
    {
        new Role()
        {
            roleId = 1,
            role = "Admin"
        },
        new Role()
        {
            roleId = 2,
            role = "Guest"
        },
        new Role()
        {
            roleId = 3,
            role = "OpUser"
        }
    }
};

//try to update role 3 to PartsUser and add a new role 4 - Manager
var userUpdate = new Users()
{
    userId = 1,
    roles = new List<Role>() 
    {
        new Role()
        {
            roleId = 3,
            role = "PartsUser"
        },
        new Role()
        {
            roleId = 4,
            role = "Manager"
        }
    }
};

Now I am trying to merge userUpdate to user object.

JObject dest = JObject.Parse(JsonConvert.SerializeObject(user));

JObject source = JObject.Parse(JsonConvert.SerializeObject(userUpdate));

dest.Merge(source, new JsonMergeSettings() { MergeArrayHandling = MergeArrayHandling.Union });

My expectation was to have a user object with role 3 changed to PartsUser and a new role 4 added for the user.

However, the code results into this:

{
    "userId": 1,
    "name": "Harsh",
    "roles": [
        {
            "roleId": 1,
            "role": "Admin"
        },
        {
            "roleId": 2,
            "role": "Guest"
        },
        {
            "roleId": 3,
            "role": "OpUser"
        },
        {
            "roleId": 3,
            "role": "PartsUser"
        },
        {
            "roleId": 4,
            "role": "Manager"
        }
    ]
}

I want to avoid the duplicate of role 3 but can't seem to find a way to do it. I have tried other MergeArrayHandling options but none seem to work. Anybody faced this issue?