0

I've looked at many of the different posts about appending a data frame within a for loop but cannot find a solution to my specific problem. Please read below. Any suggestions would be greatly appreciated!

I'm trying to append a data frame within a for loop without rewriting the entire file. I only want to add the new records each time the loop executes. I'm downloading data from an API. Code to interface with the API and download the data works fine. I use jsonlite to parse the json output and can see the data in the console. Unfortunately, when I try to append the data into a data frame, I get an error message, see below.

Current process:

Libraries used:

library(quantmod)
library(ROAuth)
library(jsonlite)
library(plyr)

Create data frame before for loop:

data_all<-data.frame()

Code to append data at end of loop:

data<-fromJSON(response)

data_all<-rbind(data_all,data)
 }

Error message:

 Error in (function (..., row.names = NULL, check.rows = FALSE, check.names = TRUE,  : 
  row names supplied are of the wrong length

Again, the code to interface with API, download, and parse the data works fine. Here's a sample data set viewed in the console:

$response
$response$`@id`
[1] "8193818b-a60c-462c-9700-f52638a75f49"

$response$elapsedtime
[1] "0"

$response$quotes
$response$quotes$quote
    adp_100  adp_200   adp_50 adv_21 adv_30 adv_90    ask ask_time asksz basis
1   32.5810  30.0390  32.2500   5135   4973   6973   0.00    00:00     0    na
2   78.1520  72.2150  77.3350  90738  75752  62908   0.00    00:00     0    na
3    6.1742   7.0495   5.8037   8651   7541  11224 0.0000    00:00     0    na
4    1.5372   1.8598   1.4709  40501  98865 108243 0.0000    00:00     0    na
5   15.7944  17.2267  14.8303 191035 208140 218453   0.00    00:00     0    na
6     17.67    17.22    16.84 263510 245725 264352 18.970    00:00    21    na
7  485.8671 462.3571 479.1363   3702   3545   4654   0.00    00:00     0    na
8    34.681   35.654   33.258 120094 108802 101061  0.000    00:00     0    na
9    2.2274   1.9028   2.2863 863965 812934 976648  2.510    00:00     2    na
10  16.3700  15.4000  16.4300 385475 316978 268737  0.000    00:00     0    na
      beta    bid bid_time bidsz bidtick chg chg_sign chg_t     cl
1   0.6801   0.00    00:00     0       d   0        e    na  32.65
2  -0.7521   0.00    00:00     0       d   0        e    na  68.28
3  -0.0625 0.0000    00:00     0       d   0        e    na 5.9900
4   3.4096 0.0000    00:00     0       d   0        e    na 1.4787
5   1.4868  12.15    00:00     1       d   0        e    na  16.90
6   0.3458 18.960    00:00    23       d   0        e    na 19.180
7   0.6924   0.00    00:00     0       d   0        e    na 492.80
8   1.8913  0.000    00:00     0       d   0        e    na 34.300
9   0.6425  0.000    00:00     0       d   0        e    na  2.550
10  2.0775  0.000    00:00     0       d   0        e    na 17.105
   contract_size cusip       date                  datetime days_to_expiration
1             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
2             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
3             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
4             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
5             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
6             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
7             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
8             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
9             na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
10            na    na 2017-03-03 2017-03-03T00:00:00-05:00                 na
    div divexdate divfreq divpaydt dollar_value    eps exch exch_desc     hi
1  0.22  20170427       Q 20170515     69329.46   1.68 NASD    NASDAQ  32.80
2  0.34  20161228       Q 20170109   3562746.42   0.89 NASD    NASDAQ  68.10
3    na        na       Q       na    1919.7966  -6.88 NASD    NASDAQ 5.9800
4    na        na       Q       na   83955.2782  -4.13 NASD    NASDAQ 1.6000
5    na        na       Q       na   2559591.04   -2.9 NASD    NASDAQ  17.40
6    na        na       Q       na  5645067.126  -1.15 NASD    NASDAQ 19.205
7  1.05  20170313       Q 20170331    884035.88  15.16 NASD    NASDAQ 490.75
8    na        na       Q       na  2481311.782 1.7492 NASD    NASDAQ 34.385
9    na        na       Q       na  1359155.880  -0.17 NASD    NASDAQ  2.560
10   na        na       Q       na  1951265.457   0.52 NASD    NASDAQ 17.220
    iad idelta igamma imp_volatility incr_vl irho issue_desc itheta ivega
1  0.88     na     na             na    1278   na         na     na    na
2  1.36     na     na             na    6202   na         na     na    na
3    na     na     na             na     100   na         na     na    na
4    na     na     na             na    3583   na         na     na    na
5    na     na     na             na    9589   na         na     na    na
6    na     na     na             na   16732   na         na     na    na
7   4.2     na     na             na     466   na         na     na    na
8    na     na     na             na    7410   na         na     na    na
9    na     na     na             na    8903   na         na     na    na
10   na     na     na             na   14084   na         na     na    na
     last     lo                               name op_delivery op_flag
1   32.80  32.60                AMES NATIONAL CORP.          na       0
2   66.71  66.59              ATN INTERNATIONAL INC          na       1
3  5.9800 5.7243                        ATOMERA INC          na       0
4  1.5101 1.4400                ATOSSA GENETICS INC          na       0
5   17.15  16.85          ATARA BIOTHERAPEUTICS INC          na       1
6  18.960 18.700                       ATRICURE INC          na       1
7  489.75 489.75                       ATRION CORP.          na       0
8  33.560 33.505                     ASTRONICS CORP          na       1
9   2.500  2.460                ANTARES PHARMA INC.          na       1
10 17.070 16.870 AIR TRANSPORT SERVICES GROUP, INC.          na       1
   op_style op_subclass openinterest    opn opt_val pchg pchg_sign   pcls
1        na          na           na  32.60      na 0.00        na  32.80
2        na          na           na  67.91      na 0.00        na  66.71
3        na          na           na 5.7243      na 0.00        na 5.9800
4        na          na           na 1.4500      na 0.00        na 1.5101
5        na          na           na  16.95      na 0.00        na  17.15
6        na          na           na 19.000      na 0.00        na 18.960
7        na          na           na 490.75      na 0.00        na 489.75
8        na          na           na 34.330      na 0.00        na 33.560
9        na          na           na  2.550      na 0.00        na  2.500
10       na          na           na 17.100      na 0.00        na 17.070
        pe    phi    plo   popn pr_adp_100 pr_adp_200 pr_adp_50    pr_date
1  19.5238  32.95  32.65  32.90    32.5817    30.0207   32.2551 2017-03-02
2  74.9551  70.54  68.03  70.23   781.5260    72.1521   77.4760 2017-03-02
3        0 6.2400 5.7700 5.7700     6.1742     7.0571    5.8094 2017-03-02
4        0 1.4803 1.4200 1.4515     1.5372     1.8732    1.4715 2017-03-02
5        0  17.35  16.25  16.40    15.7944    17.2638   14.8294 2017-03-02
6        0 19.590 18.520 18.880     176.73      17.21     16.82 2017-03-02
7  32.3054 493.30 488.35 488.35   485.8671   462.2427  478.6029 2017-03-02
8  19.1859 35.425 33.980 34.730    346.814     35.661    33.292 2017-03-02
9        0  2.565  2.480  2.510     2.2274     1.8969    2.2805 2017-03-02
10 32.8269 17.340 17.040 17.340   163.7500    15.4012   16.4200 2017-03-02
   pr_openinterest prbook  prchg prem_mult put_call   pvol qcond rootsymbol
1               na   1.79  -0.25        na       na   5559    38         na
2               na   1.60  -1.95        na       na  49971    38         na
3               na 1.0678 0.2200        na       na   4653    38         na
4               na 0.5741 0.0272        na       na  22806    38         na
5               na   1.83   0.50        na       na 294468    38         na
6               na  3.710  0.300        na       na 692148    38         na
7               na   5.58   4.45        na       na   3315    38         na
8               na  2.206 -0.430        na       na 105349    38         na
9               na  7.812  0.040        na       na 569164    38         na
10              na  3.178 -0.235        na       na 119506    38         na
   secclass sesn       sho strikeprice symbol tcond  timestamp tr_num tradetick
1         0   na   9311000          na   ATLO    29 1488790771     28         e
2         0   na  16144000          na   ATNI    29 1488790771   1031         e
3         0   na   5357000          na   ATOM   191 1488790771      8         e
4         0   na   3788000          na   ATOS    73 1488790771    105         e
5         0   na  28861000          na   ATRA    29 1488790771   1476         e
6         0   na  33240000          na   ATRC    29 1488790771   2541         e
7         0   na   1824000          na   ATRI    29 1488790771     72         e
8         0   na  21692000          na   ATRO    29 1488790771    848         e
9         0   na 155087000          na   ATRS    29 1488790771   2772         e
10        0   na  59568000          na   ATSG    29 1488790771   1108         e
   trend under_cusip undersymbol     vl volatility12   vwap  wk52hi wk52hidate
1     na          na          na   2116       0.2336  32.76    37.0   20170106
2     na          na          na  53289       0.2874  66.85   87.80   20170103
3     na          na          na    329       0.5125 5.8352 10.2000   20160830
4     na          na          na  54562       0.9774 1.5387    6.40   20160309
5     na          na          na 150230       0.7351  17.03   25.73   20160805
6     na          na          na 297923       0.4456 18.948   20.40   20161115
7     na          na          na   1806       0.2029 489.49     522   20161215
8     na          na          na  73551       0.4079 33.735   41.00   20161209
9     na          na          na 544378       0.6475  2.496    2.67   20170215
10    na          na          na 114571       0.3617 17.031   17.60   20170224
   wk52lo wk52lodate xdate xday xmonth xyear  yield
1    23.0   20160311    na   na     na    na 2.6829
2   62.70   20160817    na   na     na    na 2.0387
3  5.5000   20170228    na   na     na    na     na
4    1.30   20161230    na   na     na    na     na
5   12.45   20170131    na   na     na    na     na
6   13.44   20160512    na   na     na    na     na
7     373   20160323    na   na     na    na 0.8576
8   25.90   20160504    na   na     na    na     na
9    0.67   20160318    na   na     na    na     na
10  11.70   20160307    na   na     na    na     na


$response$error
[1] "Success"

Do I need to process the data by converting the NA values prior to appending? Should I use a different library or function(s) to append the data without rewriting the entire file? Thanks very much for any suggestions.

Improve this question
7
  • Dou you want to add row by row using for loop? Commented Mar 6, 2017 at 8:38
  • Yes. I realize there are more efficient ways to append data in R. However, my objective with this project is to download data, perform calculations on the combined file and then repeat the process many times. Commented Mar 6, 2017 at 8:45
  • Provide the sample dataset. Commented Mar 6, 2017 at 8:49
  • Thanks for your questions. I posted a sample data set above. It was from the console. Commented Mar 6, 2017 at 9:06
  • 1
    Please provide a small, reproducible example. Commented Mar 6, 2017 at 10:40

1 Answer 1

Reset to default
3

This code is reproducible and appends a JSON to a data.frame.

library(jsonlite)
library(dplyr)
data_all <- data.frame()
response <- toJSON(mtcars, pretty=TRUE)
for(i in 1:3){
  data <- fromJSON(response)
  data_all <- bind_rows(data_all, data)
}

While data is only 32 rows, data_all has 96 rows.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks very much for the suggestion. I was able to use your code and can now append the data frame. Thanks again!
Thanks again everyone for your help. Quick follow-up question. How do I work with the data in the combined file (data_all). I try to convert one of the fields to numeric and then take the mean but I get an error.
Best to ask new questions in new posts :-) but maybe you can look into dplyr (in case you didn't use it before). You can mutate a column. mutate(df, col = as.numeric(col)). mean would then be simple as mean(df$col).
takje, thanks for your help again! I'll do as you suggest. If I can't get it to work, then I'll create a new post.
@user3806824 if there is not one which already describes this exact problem of course :-) always search for duplicates!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.