tensorflow, decode_csv with variable data length

Question

I want to write sequence to sequence, using tensorflow. my input data shape is

[input_length, target_length, input , target]

and they have all different lengths. how can I use tf.decode_csv? I tried to make record_defaults with maximum input length. but All shapes must be fully defined in record_defaults............ I can't figure out about this.

    csv_file = tf.train.string_input_producer([file_name], name='file_name')
    reader = tf.TextLineReader()
    _, line = reader.read(csv_file)
    record_defaults = [[0] for row in range(20)]
    data = tf.decode_csv(line,record_defaults=record_defaults,field_delim=',')
    len_error = tf.slice(data,[0],[1])
    len_target = tf.slice(data, [1], [1])
    error = tf.slice(data,[2],len_error)
    target = tf.slice(data, 2+len_error , len_target)

I have tried this as well and couldn't make it work using this approach. Seems indeed that your rows have to all be equal length to the max in order to make it work. If you define the defaults for Max, but have a row with fewer observations, it will probably throw an error saying 'expected X observation but got Y'. Maybe something like having extra ',' in the CSV file to simulate the blanks is worth a try? Haven't tried myself. I often find it easier to read text files outside of TF, like with numpy/pandas and then feed values to TF, but this way you lose nice TF shuffling and multi file ques — VS_FF
– VS_FF, Commented May 8, 2017 at 8:44
wow.... you are so kind. I have to try another file type. thank you very very much — Amily
– Amily, Commented May 11, 2017 at 17:16

Peter Hawkins · Accepted Answer · 2017-05-12 14:37:23Z

1

Yes, tf.decode_csv does require all rows to be the same size. If this does not work for you, consider filing a feature request on Github.

You could also preprocess your CSV file to pad all of the entries out to the same number of columns; you can use the record_defaults argument to tf.decode_csv to leave the fields empty but supply default values.

answered May 12, 2017 at 14:37

Peter Hawkins

3,21121 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

tensorflow, decode_csv with variable data length

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related