Concurrent Tensorflow String Split

Question

I need to do some preprocessing before feeding the input to my network. I am working on my data iterator and making some functions to make the data ready. My train set is a large file and each line has following scheme. Each line is semicolon seperated which could also be empty. Each split then contains different events which are como seperated and finaly each observation has 5 singnals which are colon seperated.

1:1560629595635:183.94:z1:Happy,2:1560629505635:100:z1:Sad;5:1561929595635:1:z1:Happy,13:1561629595635:12:j1:Sad;50:15616295956351:10:f1:Sad

My objective is to split each line in following order:

1. ';' Split
2. ',' Split
3. ':' Split
*Note: length after the last split = 5

In following function that I wrote, if I specify indx for the function it works fine for only one fragment output of ';' split.

def __semicolon_coma_colon_split(line, table, indx):
    sources = tf.strings.split(line, ';')
    pairs = tf.strings.split(sources[indx], ',')
    items = tf.strings.split(pairs, ':').to_tensor()
    return (tf.strings.to_number(items[:, 0], out_type=tf.int32),
            tf.strings.to_number(items[:, 1], out_type=tf.int64),
            tf.strings.to_number(items[:, 2], out_type=tf.float32),
            tf.cast(table.lookup(items[:, 3]), tf.int32),
            tf.cast(table.lookup(items[:, 4]), tf.int32))

And the current output shape is

(<tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>)

However, I need to run this cuncurrently on all the splits after ';' and the expected output tensor would look like a tuple of tuples that each contains 5 tensors;

((<tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>), 
(<tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>, <tf.Tensor: shape=(2,)>),
(<tf.Tensor: shape=(1,)>, <tf.Tensor: shape=(1,)>, <tf.Tensor: shape=(1,)>, <tf.Tensor: shape=(1,)>, <tf.Tensor: shape=(1,)>))

desertnaut · Accepted Answer · 2020-08-10 18:57:14Z

1

Can't you do this with split built-in method for strings?

edited Aug 10, 2020 at 18:57

desertnaut

60.8k32 gold badges155 silver badges183 bronze badges

answered Aug 10, 2020 at 14:49

Mohammad Sadra Sharifzadeh

3271 gold badge4 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Concurrent Tensorflow String Split

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related