I have read in some lengthy data with read.csv(), and to my surprise the data is coming out as factors rather than numbers, so I'm guessing there must be at least one non-numeric item in the data. How can I find where these items are?
For example, if I have the following data frame:
df <- data.frame(c(1,2,3,4,"five",6,7,8,"nine",10))
I would like to know that rows 5 and 9 have non-numeric data. How would I do that?
df <- data.frame(x = c(1,2,3,4,"five",6,7,8,"nine",10))
The trick is knowing that converting to numeric via as.numeric(as.character(.)) will convert non-numbers to NA.
which(is.na(as.numeric(as.character(df[[1]]))))
## 5 9
(just using as.numeric(df[[1]]) doesn't work - it just drops the levels leaving the numeric codes).
You might choose to suppress the warnings:
which.nonnum <- function(x) {
which(is.na(suppressWarnings(as.numeric(as.character(x)))))
}
which.nonnum(df[[1]])
To be more careful, you should also check that the values weren't NA before conversion:
which.nonnum <- function(x) {
badNum <- is.na(suppressWarnings(as.numeric(as.character(x))))
which(badNum & !is.na(x))
}
lapply(df, which.nonnum) will report 'bad' values for all columns of the data frame.
is.numeric()?
is.numeric() applied to a factor simply drops the levels.
df <- data.frame(value = c(1,2,3,4,"-",6,7,8,"x",10)); df %>% filter(value %>% as.numeric() %>% is.na()) %>% count(value) can find what are the non numeric data in the column.
lapply(df, which.nonnum)). If you like you could ask that as a new question, linking back to this one.An alternative could be to check which entries in the vector contain any characters other than a number:
df <- data.frame(c(1,2,3,4,"five",-6,7.1,-8.059,"nine",10))
which(!grepl('^-?(0|[1-9][0-9]*)(\\.[0-9]+)?$',df[[1]]))
## 5 9
[^0-9.] to allow for decimals.[^0-9.-] for minus signs? (I'm not quite sure how regex distinguishes - as character from - as range separator)