Tensorflow CSV decode error

Question

I am using TensorFlow 0.10.0rc0. I have CUDA Driver Version = 7.5 and CUDNN 4 on Ubuntu 14.04.

I have a simple CSV file which has a single line like this:

"field with
newline",0

where the newline has been added by pressing the enter key in VIM on Ubuntu. I am able to read this file in pandas using the read_csv function, where the text field is shown as containing a single \n character.

But when I try to read it in TensorFlow, I get the following error:

tensorflow.python.framework.errors.InvalidArgumentError: Quoted field has to end with quote followed by delim or end

My tensor flow code to read CSV uses this function to read a single row:

def read_single_example(filename_queue, skip_header_lines, record_defaults, feature_index, label_index):
    reader = tf.TextLineReader(skip_header_lines=skip_header_lines)
    key, value = reader.read(filename_queue)
    record = tf.decode_csv(
        value,
        record_defaults=record_defaults)
    features, label = record[feature_index], record[label_index]
    return features, label

If I read using pandas and replace all newlines with spaces, the TensorFlow code is able to parse the CSV successfully.

But it will be really helpful if newlines can be handled within the TensorFlow CSV pipeline itself.

the rfc4180 specs says its allowed, and python's default dialect of csv is 'excel' which should be capable though. — YOU
– YOU, Commented Aug 17, 2016 at 12:55

rlys · Accepted Answer · 2018-08-16 18:15:45Z

1

The issue here is that TextLineReader splits the file on new lines, before it is parsed by the csv decoder. With tf.data, you can use tf.contrib.data.CsvDataset, which parses this file correctly according to RFC4180.

answered Aug 16, 2018 at 18:15

rlys

4802 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Eric Platon · Accepted Answer · 2016-08-17 13:47:00Z

0

TensorFlow's CSV reader is pretty strict, in my experience with it, with regards to RFC4180.

Making sure your files use CRLF at the end of each line, as well as in quoted fields, should allow processing.

Note: I have been using this up to 0.9 so far. I did not try on RCs from 0.10.

answered Aug 17, 2016 at 13:47

Eric Platon

10.2k6 gold badges43 silver badges50 bronze badges

3 Comments

Ramnath Over a year ago

The same error exists in Tensorflow 0.9 also. I have given my entire csv file above in the question. As far as I know, this csv file is obeys RFC4180. Please see @you 's comment above.

Eric Platon Over a year ago

Sorry for the long blank. Have you resolved your issue? An update would be great, like an answer to your own question. I did not manage to reproduce now.

Ramnath Over a year ago

Hi @EricPlaton I raised the issue in Tensorflow github and I assume they are working on it. github.com/tensorflow/tensorflow/issues/3851

Collectives™ on Stack Overflow

Tensorflow CSV decode error

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related