How to use tf.py_function for working with tf.string?

Question

I have tensor with strings (tf.string) and I want to split this strings by regexp and make some preprocessing.

For example I have function

py_split(x):
    x = x.lower()
    x = re.split(r"(http:\/\/)|(https:\/\/)|(\W)", x)
    return x

and I need to use it in tensorflow-transform graph for using with tf serving in future.

But TF does not allow me to work with tf.string's same as normal strings in python.

How can I solve my problem without making new TF op in C++?

P.S. I use TensorFlow 1.13

Stewart_R · Accepted Answer · 2019-06-16 10:37:37Z

1

Slightly tricky because TensorFlow (at least to my knowledge) doesn't have a regex split function.

If there is a character that you can be sure your input strings won't contain you could do a slightly messy workaround using tf.strings.regex_replace() and tf.strings.split(). We first use regex_replace in order to replace the match with our special character then use split to split on the special character

For example, if we could be sure our input strings would never contain the char | then we could proceed as follows:

def split(x):
  x = tf.strings.regex_replace(x, "(http:\/\/)|(https:\/\/)|(\W)", "|")
  return tf.strings.split(tf.expand_dims(x, 0), '|').values

so that, split("http://www.bbc.co.uk") say, gives us:

[b'', b'www', b'bbc', b'co', b'uk']

answered Jun 16, 2019 at 10:37

Stewart_R

14.5k11 gold badges67 silver badges118 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mihail Salnikov Over a year ago

Thank you very much, but result of this operation isn't same because re.split not remove delimiters. For example for string: "Hello, world" re.split return ["Hello", ",", " ", "world"]

Collectives™ on Stack Overflow

How to use tf.py_function for working with tf.string?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related