Deleting and re-adding element to array does not update array with Vue in Chrome/Safari

I have a file upload component where I can remove selected files, I tried both this.files.splice(index, 1); and this.$delete(this.files, index);

if I re-add a removed file with this.files = [...this.$refs.fileUpload.files];, it doesn't get added in the files array. If I select a different file it does show up, so I'm guessing this is a reactivity issue.

The simplified component:

<template>
  <div class="file-upload-component">
    <div
      class="upload-area"
    >
      <label for="fileUpload" class="component-info">
        <span>Add files</span>
      </label>
      <input
        type="file"
        style="display:none"
        @change="addFiles"
        ref="fileUpload"
        id="fileUpload"
      />
      <div class="file-list">
        <div class="file-list__file" v-for="(file, index) in files" :key="index">
          <span>{{ file.name }}</span>
            <button
              @click="remove(index)"
              class="cancel"
            >
              remove
            </button>
        </div>
      </div>
    </div>
  </div>
</template>

<script>
export default {
  data() {
    return {
      files: [],
    };
  },
  methods: {
    addFiles() {
      this.files = [...this.$refs.fileUpload.files];
    },
    remove(i) {
      this.$delete(this.files, i);
    },
  },
};
</script>

<style lang="scss" scoped>
.upload-area {
  width: 100%;
  height: 356px;

  .component-info {
    width: 100%;
    height: 100%;
    display: flex;
    justify-content: center;
    align-items: center;
  }

  .file-list {
    margin: 20px;

    .cancel {
      margin-left: 10px;
    }
  }
}
</style>

The strange thing is this way works in Firefox.