rbindlist and nested data.table, different behavior with/without using get

Question

I am loading some JSON data using jsonlite which is resulting in some nested data similar (in structure) to the toy data.table dt constructed below. I want to be able to use rbindlist to bind the nested data.tables together.

Setup:

> dt <- data.table(a=c("abc", "def", "ghi"), b=runif(3))
> dt[, c:=list(list(data.table(d=runif(4), e=runif(4))))]
> dt
     a         b            c
1: abc 0.2623218 <data.table>
2: def 0.7092507 <data.table>
3: ghi 0.2795103 <data.table>

Using the NSE built into data.table, I can do:

> rbindlist(dt[, c])
            d          e
 1: 0.8420476 0.26878325
 2: 0.1704087 0.59654706
 3: 0.6023655 0.42590380
 4: 0.9528841 0.06121386
 5: 0.8420476 0.26878325
 6: 0.1704087 0.59654706
 7: 0.6023655 0.42590380
 8: 0.9528841 0.06121386
 9: 0.8420476 0.26878325
10: 0.1704087 0.59654706
11: 0.6023655 0.42590380
12: 0.9528841 0.06121386

which is exactly what I expect/want. Furthermore, the original dt remains unmodified:

> dt
     a         b            c
1: abc 0.2623218 <data.table>
2: def 0.7092507 <data.table>
3: ghi 0.2795103 <data.table>

However, when manipulating the data.table within a function I generally want to use get with string column names:

> rbindlist(dt[, get("c")])
           V1         V2
 1: 0.8420476 0.26878325
 2: 0.1704087 0.59654706
 3: 0.6023655 0.42590380
 4: 0.9528841 0.06121386
 5: 0.8420476 0.26878325
 6: 0.1704087 0.59654706
 7: 0.6023655 0.42590380
 8: 0.9528841 0.06121386
 9: 0.8420476 0.26878325
10: 0.1704087 0.59654706
11: 0.6023655 0.42590380
12: 0.9528841 0.06121386

Now the column names have been lost and replaced by the default "V1" and "V2" values. Is there a way to retain the names?

In the development version (v1.9.5) the problem is worse than simply lost names though. After executing the statement: rbindlist(dt[, get("c")]) the entire data.table becomes corrupt:

> dt
Error in FUN(X[[3L]], ...) : 
  Invalid column: it has dimensions. Can't format it. If it's the result of data.table(table()), use as.data.table(table()) instead.

To be clear, the lost names issue happens in both v1.9.4 (installed from CRAN) and v1.9.5 (installed from github), but the corrupt data.table issue seems to affect v1.9.5 only (as of today - July 8, 2015).

If I were able to stick with the NSE version of things everything runs smoothly. My issue is that sticking with the NSE version would involve writing multiple NSE functions calling each other which seems to get messy pretty fast.

Are there any (non-NSE-based) known work-arounds? Also, is this a known issue?

Or more generally var <- "c" ; rbindlist(dt[, eval(as.name(var))]) — David Arenburg
– David Arenburg, Commented Jul 8, 2015 at 14:32
Submitted an issue here will see what @Arun says. Either Way, if you won't run get, rbindlist(dt[[var]]) should also work on the devel version, while it will work fine regardless of get on v < 1.9.5. (as mentioned by @Frank) — David Arenburg
– David Arenburg, Commented Jul 8, 2015 at 15:06

jangorecki · Accepted Answer · 2020-11-21 18:15:11Z

This must have been fixed in last 5 years since this Q was asked. Now I am getting expected results.

> library(data.table)
data.table 1.13.3 IN DEVELOPMENT built 2020-11-17 18:11:47 UTC; jan using 4 threads (see ?getDTthreads).  Latest news: r-datatable.com
> dt <- data.table(a=c("abc", "def", "ghi"), b=runif(3))
> dt[, c:=list(list(data.table(d=runif(4), e=runif(4))))]
> dt
     a         b                 c
1: abc 0.2416624 <data.table[4x2]>
2: def 0.0222938 <data.table[4x2]>
3: ghi 0.3510681 <data.table[4x2]>
> rbindlist(dt[, c])
            d          e
 1: 0.5485731 0.32366420
 2: 0.5457945 0.45173251
 3: 0.6796699 0.03783026
 4: 0.4442776 0.03121024
 5: 0.5485731 0.32366420
 6: 0.5457945 0.45173251
 7: 0.6796699 0.03783026
 8: 0.4442776 0.03121024
 9: 0.5485731 0.32366420
10: 0.5457945 0.45173251
11: 0.6796699 0.03783026
12: 0.4442776 0.03121024
> rbindlist(dt[, get("c")])
            d          e
 1: 0.5485731 0.32366420
 2: 0.5457945 0.45173251
 3: 0.6796699 0.03783026
 4: 0.4442776 0.03121024
 5: 0.5485731 0.32366420
 6: 0.5457945 0.45173251
 7: 0.6796699 0.03783026
 8: 0.4442776 0.03121024
 9: 0.5485731 0.32366420
10: 0.5457945 0.45173251
11: 0.6796699 0.03783026
12: 0.4442776 0.03121024
> dt
     a         b                 c
1: abc 0.2416624 <data.table[4x2]>
2: def 0.0222938 <data.table[4x2]>
3: ghi 0.3510681 <data.table[4x2]>

Collectives™ on Stack Overflow

rbindlist and nested data.table, different behavior with/without using get

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related