Google apps script slow parsing

Question

I was trying to parse vmstat using Google Apps script so that everyone in the company could use this to create a graph of the data. You can find my code here but this code is really slow. Is there something I can do to make this better or isn't Google Apps Script suitable for this? The problem the ammount of rows that needs to be processed. Any suggestions are welcome.

  function doGet(){

  var file = DriveApp.getFileById(id)

  var docContent = file.getAs('application/octet-stream').getDataAsString();
  var data = Charts.newDataTable()
                   .addColumn(Charts.ColumnType.STRING, 'TIME')
                   .addColumn(Charts.ColumnType.NUMBER, 'Memory');

  var lines = docContent.split("\n");
  Logger.log(lines.length);
  var i = 1;

  lines.forEach(function(line) {
    if ((line.indexOf('mem') < 0) && (line.indexOf('free') < 0)) {
      var values = line.match(/\S+/g);
      data.addRow(['5',parseInt(values[3])]); 
      Logger.log(i)
    }
    if (i == 20){
     return;
    }
    i++;
  }); 

  for( var i=0;i< lines.length;i++){
      data.addRow(['5',10]); 
  }
  data.build();

  var chart = Charts.newAreaChart()
      .setDataTable(data)
      .setStacked()
      .setRange(0, 400)
      .setTitle('Memory')
      .build();

 return UiApp.createApplication().add(chart);

}

most of your code is about finding the file in drive where you could just openaById the wanted file. — Zig Mandel
– Zig Mandel, Commented Jul 13, 2015 at 16:45
Tell us more about your vmstat data. How often is it being updated? How are you generating it? — Mogsdad
– Mogsdad, Commented Jul 14, 2015 at 14:10
@ZigMandel Yes thats true but this was just a test in the future we would let the user choose which file he want to use for the graphs. — jackiejarvis
– jackiejarvis, Commented Jul 15, 2015 at 6:37
if you dont include that part it should run fastm if not please remove that extra code so its clear what code should be optimized. — Zig Mandel
– Zig Mandel, Commented Jul 15, 2015 at 6:40
@ZigMandel I updatted my code. the result is still the same. searching for the file was really fast. the problem is by processing the file. More than 5minutes for a big file. — jackiejarvis
– jackiejarvis, Commented Jul 15, 2015 at 6:52

Zoe - Save the data dump · Accepted Answer · 2019-05-29 08:37:57Z

This isn't a problem of code optimization (although the code isn't perfect), as much as division of work.

The accepted approach to web application performance optimization involves separating three concerns; presentation, business logic and data access^ref. With the exception of the generation of the vmstat output, you've got all of that in one place, making the user wait while you locate a file on Google Drive (using two exhaustive searches, btw) and then parse it into a Charts DataTable, and finally generate HTML (via UiApp).

You may find that the accessibility of a Google Apps Script presentation is useful to your organization. (I know in my workplace that our IT folks clamp down on in-house web servers, for example.) If so, consider what you have as prototype, and refactor it to give better perceived performance.

Presentation: Move from UiApp + Charts to HtmlService + Google Visualization. This moves the generation of the chart into the web client, instead of keeping it in the server. This will give a faster page load, to start.
Business Logic: This will be the rules that map your data into the Visualization. Like the Charts Service that is built over it, GViz uses DataTables with column definitions and rows of data.

One option here is to repeat the column definition & data load you already have, except on the client in JavaScript. Doing that will be significantly faster than via Google Apps Script.

A second option, which is even faster, especially with large datasets, is to load the data from an array.
```
google.visualization.arrayToDataTable(...)
```
Either way, you need to get your data to the JavaScript function that will build your chart.
Data Access: (I assume) you're currently running a shell script in Linux that calls vmstat and pipes the output to a file in your local Google Drive folder. (Alternatively, the script may be using the Drive API to push the file to Google Drive.) This file is plain text.

The change I'd make here would be to produce csv output from vmstat, and use Google Apps Script to import the csv into a spreadsheet. Then, you can use Sheet.getSheetValues() to read all the data in one shot, in a server side function to be called from the client JavaScript.

This would not be as fast as a local server solution, but it's probably the best way to do this using the Google Apps Script environment.

Edit: See more about this in my blog post, Converting from UiApp + Chart Service to Html Service + Google Visualization API.

Collectives™ on Stack Overflow

Google apps script slow parsing

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related