How to do string-style split on tf tensor with string values using TF-1.4

Question

I have a tensor(e.g. labels) which read data from external data source. the value of the tensor is a string which has format "label1,label2" (e.g. "0,1"). Now I want to split the string values into a list using delimiter ',' so the result would be like ['0', '1'].
I've tried flowing:

# option-1 with error: tensor type has no attribute split.
label_list = input_label_tensor.split(',') 

# option-2 with error: module has no method split.
label_list = tf.strings.split(input_label_tensor, ',')

# option-3 with error: 
label_list = tf.string_split(input_label_tensor, ',')

the error from option-3 is:


  File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/string_ops.py", line 113, in string_split
    source = ops.convert_to_tensor(source, dtype=dtypes.string)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 836, in convert_to_tensor
    as_ref=False)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 926, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 774, in _TensorTensorConversionFunction
    (dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float32: 'Tensor("StagingArea_get:467", shape=(?,), dtype=float32, device=/job:worker/task:0/device:CPU:0)'

what's the correct way to do the split operation? I'm using TF-1.4

AloneTogether · Accepted Answer · 2021-12-03 07:39:26Z

2

My best bet is that input_label_tensor is not actually a string tensor. For example this works:

%tensorflow_version 1.x
import tensorflow as tf

input_label_tensor = tf.constant(['0', '1'], dtype=tf.string)
label_list = tf.string_split(input_label_tensor, ',')

But if I use a float tensor, I am able to reproduce your error:

input_label_tensor = tf.constant([0, 1], dtype=tf.float32)
label_list = tf.string_split(input_label_tensor, ',')

To access the values of a SparseTensor try something like this:

%tensorflow_version 1.x
import tensorflow as tf

input_label_tensor = tf.constant(['0', '1'], dtype=tf.string)
label_list = tf.string_split(input_label_tensor, ',').values
x, y = label_list[0], label_list[1]

Update 1:

%tensorflow_version 1.x
import tensorflow as tf

input_label_tensor = tf.constant(['0,1', '1,0', '1,1', '0,0'], dtype=tf.string)
label_list = tf.string_split(input_label_tensor, ',')

c = tf.sparse.to_dense(label_list)
c = tf.string_to_number(c, tf.float32)

with tf.Session() as sess:
    result = c.eval()
   
print(result)

[[0. 1.]
 [1. 0.]
 [1. 1.]
 [0. 0.]]

edited Dec 3, 2021 at 7:39

answered Dec 2, 2021 at 6:03

AloneTogether

26.8k5 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

dingx Over a year ago

Thanks @AloneTogether for your reply. in my case, input_label_tensor is read from external data source. and the column in the external data source is of type string. and the value is like '0,1'. after read in batch, it will be like ['0,1', '0,0','1,0'] how does the float32 come?

AloneTogether Over a year ago

That is a good question, but I think it is pretty obvious that you are not working with string tensors..Have you tried casting the tensor to tf.string? If you read your data with pandas, for example, it might be converting the strings to floats automatically.

dingx Over a year ago

Thanks @AloneTogether. I figured out how the float32 comes. input_label_tensor is read with API tf.data.TableRecordDataset(). and the API has a param record_default which specify the default value for each column AND the column type. I've set the default to 0. which is a float. I thought that the target column type will be inferred from source column. but it turns out that during TF graph building the data is not read yet and thus data type can't be inferred.

dingx Over a year ago

But there is another question: after split the labels into two columns, i want to access each column but failed with error: "TypeError: 'SparseTensor' object is not iterable ". the code is like: label_list = tf.string_split(input_label_tensor, ','); labe1, label2 = label_list

AloneTogether Over a year ago

@dingx So my answer answered your first question? Because your second question does not have anything to do directly with your first one..

|

Collectives™ on Stack Overflow

How to do string-style split on tf tensor with string values using TF-1.4

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related