NodeJS MongoDB Mongoose export nested subdocuments and arrays to XLSX columns

Question

I have query results from MongoDB as an array of documents with nested subdocuments and arrays of subdocuments.

[
  {
    RecordID: 9000,
    RecordType: 'Item',
    Location: {
      _id: 5d0699326e310a6fde926a08,
      LocationName: 'Example Location A'
    }
    Items: [
      {
        Title: 'Example Title A',
        Format: {
          _id: 5d0699326e310a6fde926a01,
          FormatName: 'Example Format A'
        }
      },
      {
        Title: 'Example Title B',
        Format: {
          _id: 5d0699326e310a6fde926a01,
          FormatName: 'Example Format B'
        }
      }
    ],
  },
  {
    RecordID: 9001,
    RecordType: 'Item',
    Location: {
      _id: 5d0699326e310a6fde926a08,
      LocationName: 'Example Location C'
    },
    Items: [
      {
        Title: 'Example Title C',
        Format: {
          _id: 5d0699326e310a6fde926a01,
          FormatName: 'Example Format C'
        }
      }
    ],
  }
]

Problem

I need to export the results to XLSX in column order. The XLSX library is working to export the top-level properties (such as RecordID and RecordType) only. I also need to export the nested objects and arrays of objects. Given a list of property names e.g. RecordID, RecordType, Location.LocationName, Items.Title, Items.Format.FormatName the properties must be exported to XLSX columns in the specified order.

Desired result

Here is the desired 'flattened' structure (or something similar) that I think should be able to convert to XLSX columns.

[
  {
    'RecordID': 9000,
    'RecordType': 'Item',
    'Location.LocationName': 'Example Location A',
    'Items.Title': 'Example Title A, Example Title B',
    'Items.Format.FormatName': 'Example Format A, Example Format B',
  },
  {
    'RecordID': 9001,
    'RecordType': 'Item',
    'Location.LocationName': 'Example Location C',
    'Items.Title': 'Example Title C',
    'Items.Format.FormatName': 'Example Format C',
  }
]

I am using the XLSX library to convert the query results to XLSX which works for top-level properties only.

  const worksheet: XLSX.WorkSheet = XLSX.utils.json_to_sheet(results.data);
  const workbook: XLSX.WorkBook = { Sheets: { 'data': worksheet }, SheetNames: ['data'] };
  const excelBuffer: any = XLSX.write(workbook, { bookType: 'xlsx', type: 'array' });

  const data: Blob = new Blob([excelBuffer], { type: EXCEL_TYPE });
  FileSaver.saveAs(data, new Date().getTime());

POSSIBLE OPTIONS

I am guessing I need to 'flatten' the structure either using aggregation in the query or by performing post-processing when the query is returned.

Option 1: Build the logic in the MongoDB query to flatten the results.

$replaceRoot might work since it is able to "promote an existing embedded document to the top level". Although I am not sure if this will solve the problem exactly, I do not want to modify the documents in place, I just need to flatten the results for exporting.

Here is the MongoDB query I am using to produce the results:

records.find({ '$and': [ { RecordID: { '$gt': 9000 } } ]},
  { skip: 0, limit: 10, projection: { RecordID: 1, RecordType: 1, 'Items.Title': 1, 'Items.Location': 1 }});

Option 2: Iterate and flatten the results on the Node server

This is likely not the most performant option, but might be the easiest if I can't find a way to do so within the MongoDB query.

UPDATE:

I may be able to use MongoDB aggregate $project to 'flatten' the results. For example, this aggregate query effectively 'flattens' the results by 'renaming' the properties. I just need to figure out how to implement the query conditions within the aggregate operation.

db.records.aggregate({
  $project: {
    RecordID: 1,
    RecordType: 1,
    Title: '$Items.Title',
    Format: '$Items.Format'
  }
})

UPDATE 2:

I have abandoned the $project solution because I would need to change the entire API to support aggregation. Also, I would need to find a solution for populate because aggregate does not support it, rather, it uses $lookup which is possible but time consuming because I would need to write the queries dynamically. I am going back to look into how to flatten the object by creating a function to iterate the array of objects recursively.

can you post a valid sample of the data you want to output as an excel file? the blob you added has a syntaxerror — Peter
– Peter, Commented Aug 28, 2019 at 14:15
@Peter Updated the desired 'flattened' structure with valid syntax. — pengz
– pengz, Commented Aug 28, 2019 at 14:28

Peter · Accepted Answer · 2019-08-28 16:01:25Z

2

+50

Below is a solution for transforming the Mongo data on the server via a function flattenObject which recursively flattens nested objects and returns a 'dot-type' key for nested paths.

Note that the snippet below contains a function that renders and editable table to preview, however, the important part you want (download the file), should be triggered when you run the snippet and click the 'Download' button.

const flattenObject = (obj, prefix = '') =>
  Object.keys(obj).reduce((acc, k) => {
    const pre = prefix.length ? prefix + '.' : '';
    if (typeof obj[k] === 'object') Object.assign(acc, flattenObject(obj[k], pre + k));
    else acc[pre + k] = obj[k];
    return acc;
  }, {});

var data = [{
    RecordID: 9000,
    RecordType: "Item",
    Location: {
      _id: "5d0699326e310a6fde926a08",
      LocationName: "Example Location A"
    },
    Items: [{
        Title: "Example Title A",
        Format: {
          _id: "5d0699326e310a6fde926a01",
          FormatName: "Example Format A"
        }
      },
      {
        Title: "Example Title B",
        Format: {
          _id: "5d0699326e310a6fde926a01",
          FormatName: "Example Format B"
        }
      }
    ]
  },
  {
    RecordID: 9001,
    RecordType: "Item",
    Location: {
      _id: "5d0699326e310a6fde926a08",
      LocationName: "Example Location C"
    },
    Items: [{
      Title: "Example Title C",
      Format: {
        _id: "5d0699326e310a6fde926a01",
        FormatName: "Example Format C"
      }
    }]
  }
];

const EXCEL_MIME_TYPE = `application/vnd.ms-excel`;
const flattened = data.map(e => flattenObject(e));
const ws_default_header = XLSX.utils.json_to_sheet(flattened);
const ws_custom_header = XLSX.utils.json_to_sheet(flattened, {
  header: ['Items.Title', 'RecordID', 'RecordType', 'Location.LocationName', 'Items.Format.FormatName']
});
const def_workbook = XLSX.WorkBook = {
  Sheets: {
    'data': ws_default_header
  },
  SheetNames: ['data']
}

const custom_workbook = XLSX.WorkBook = {
  Sheets: {
    'data': ws_custom_header
  },
  SheetNames: ['data']
}

const def_excelBuffer = XLSX.write(def_workbook, {
  bookType: 'xlsx',
  type: 'array'
});

const custom_excelBuffer = XLSX.write(custom_workbook, {
  bookType: 'xlsx',
  type: 'array'
});

const def_blob = new Blob([def_excelBuffer], {
  type: EXCEL_MIME_TYPE
});

const custom_blob = new Blob([custom_excelBuffer], {
  type: EXCEL_MIME_TYPE
});

const def_button = document.getElementById('dl-def')
/* trigger browser to download file */
def_button.onclick = e => {
  e.preventDefault()
  saveAs(def_blob, `${new Date().getTime()}.xlsx`);
}

const custom_button = document.getElementById('dl-cus')
/* trigger browser to download file */
custom_button.onclick = e => {
  e.preventDefault()
  saveAs(custom_blob, `${new Date().getTime()}.xlsx`);
}

/*
  render editable table to preview (for SO convenience)
*/
const html_string_default = XLSX.utils.sheet_to_html(ws_default_header, {
  id: "data-table",
  editable: true
});

const html_string_custom = XLSX.utils.sheet_to_html(ws_custom_header, {
  id: "data-table",
  editable: true
});
document.getElementById("container").innerHTML = html_string_default;
document.getElementById("container-2").innerHTML = html_string_custom;

<script src="https://cdnjs.cloudflare.com/ajax/libs/xlsx/0.14.3/xlsx.full.min.js"></script>

<head>
  <title>Excel file generation from JSON</title>
  <meta charset="utf-8" />
  <style>
    .xport,
    .btn {
      display: inline;
      text-align: center;
    }
    
    a {
      text-decoration: none
    }
    
    #data-table,
    #data-table th,
    #data-table td {
      border: 1px solid black
    }
  </style>
</head>
<script>
  function render(type, fn, dl) {
    var elt = document.getElementById('data-table');
    var wb = XLSX.utils.table_to_book(elt, {
      sheet: "Sheet JS"
    });
    return dl ?
      XLSX.write(wb, {
        bookType: type,
        bookSST: true,
        type: 'array'
      }) :
      XLSX.writeFile(wb, fn || ('SheetJSTableExport.' + (type || 'xlsx')));
  }
</script>
<div>Default Header</div>
<div id="container"></div>
<br/>
<div>Custom Header</div>
<div id="container-2"></div>
<br/>
<table id="xport"></table>
<button type="button" id="dl-def">Download Default Header Config</button>
<button type="button" id="dl-cus">Download Custom Header Config</button>
<script src="https://cdnjs.cloudflare.com/ajax/libs/FileSaver.js/1.3.8/FileSaver.min.js"></script>

edited Aug 28, 2019 at 16:01

answered Aug 28, 2019 at 15:20

Peter

2,9371 gold badge21 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

pengz Over a year ago

Awesome! I will give this a try asap. Any advice on how to accept a list of column names (as an array of strings) and output the results in the given column order?

Peter Over a year ago

sure, by default, it uses Object.keys for the order. If you give me an example array of columns I can update the example.

pengz Over a year ago

Here is an example, say hypothetically I want to return the results with the Items.Title first. The user will be setting the desired order of the columns in the UI. ['Items.Title', 'RecordID', 'RecordType', 'Location.LocationName', 'Items.Format.FormatName']

Peter Over a year ago

updated my answer, however you will notice a problem due to a limitation of the flattenObject function where indexes are being flattened as well as regular string key, giving us something like Items.0.Format._id. I will try to look into this later but I have to work on other stuff for the remainder of today. good luck

pengz Over a year ago

Thanks so much! So far I'm getting the error message described here (because the actual data has some null values) so I think I just need to check if the value is defined before passing to Object.keys() stackoverflow.com/questions/29721205/…

pengz · Accepted Answer · 2019-08-29 20:20:30Z

I wrote a function to iterate all object in the results array and create new flattened objects recursively. The flattenObject function shown here is similar to the previous answer and I took additional inspiration from this related answer.

The '_id' properties are specifically excluded from being added to the flattened object, since ObjectIds are still being returned as bson types even though I have the lean() option set.

I still need to figure out how to sort the objects such that they are in the order given e.g. RecordID, RecordType, Items.Title. I believe that might be easiest to achieve by creating a separate function to iterate the flattened results, although not necessarily the most performant. Let me know if anyone has any suggestions on how to achieve the object sorting by a given order or has any improvements to the solution.

const apiCtrl = {};

/**
 * Async array iterator
 */
apiCtrl.asyncForEach = async (array, callback) => {
  for (let index = 0; index < array.length; index++) {
    await callback(array[index], index, array)
  }
}

// Check if a value is an object
const isObject = (val) => {
  return typeof val == 'object' && val instanceof Object && !(val instanceof Array);
}

// Check if a value is a date object
const isDateObject = (val) => {
  return Object.prototype.toString.call(val) === '[object Date]';
}

/**
 * Iterate object properties recursively and flatten all values to top level properties
 * @param {object} obj Object to flatten
 * @param {string} prefix A string to hold the property name
 * @param {string} res A temp object to store the current iteration
 * Return a new object with all properties on the top level only
 *
 */
const flattenObject = (obj, prefix = '', res = {}) =>

  Object.entries(obj).reduce((acc, [key, val]) => {
    const k = `${prefix}${key}`

    // Skip _ids since they are returned as bson values
    if (k.indexOf('_id') === -1) {
      // Check if value is an object
      if (isObject(val) && !isDateObject(val)) {
        flattenObject(val, `${k}.`, acc)
      // Check if value is an array
      } else if (Array.isArray(val)) {
        // Iterate each array value and call function recursively
        val.map(element => {
          flattenObject(element, `${k}.`, acc);
        });
      // If value is not an object or an array
      } else if (val !== null & val !== 'undefined') {
        // Check if property has a value already
        if (res[k]) {
          // Check for duplicate values
          if (typeof res[k] === 'string' && res[k].indexOf(val) === -1) {
            // Append value with a separator character at the beginning
            res[k] += '; ' + val;
          }
        } else {
          // Set value
          res[k] = val;
        }
      }
    }

    return acc;

  }, res);

/**
 * Convert DB query results to an array of flattened objects
 * Required to build a format that is exportable to csv, xlsx, etc.
 * @param {array} results Results of DB query
 * Return a new array of objects with all properties on the top level only
 */
apiCtrl.buildExportColumns = async (results) => {

  const data = results.data;
  let exportColumns = [];

  if (data && data.length > 0) {
    try {
      // Iterate all records in results data array
      await apiCtrl.asyncForEach(data, async (record) => {

        // Convert the multi-level object to a flattened object
        const flattenedObject = flattenObject(record);
        // Push flattened object to array
        exportColumns.push(flattenedObject);

      });
    } catch (e) {
      console.error(e);
    }
  }

  return exportColumns;

}

Collectives™ on Stack Overflow

NodeJS MongoDB Mongoose export nested subdocuments and arrays to XLSX columns

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related