Ways of using value provider parameter in Python Apache Beam

Question

Right now I'm just able to grab the RunTime value inside a class using a ParDo, is there another way to get to use the runtime parameter like in my functions?

This is the code I got right now:

class UserOptions(PipelineOptions):
    @classmethod
    def _add_argparse_args(cls, parser):
        parser.add_value_provider_argument('--firestore_document',default='')

def run(argv=None):

    parser = argparse.ArgumentParser()

    pipeline_options = PipelineOptions()

    user_options = pipeline_options.view_as(UserOptions)

    pipeline_options.view_as(SetupOptions).save_main_session = True

    with beam.Pipeline(options=pipeline_options) as p:

        rows = (p 
        | 'Create inputs' >> beam.Create(['']) 
        | 'Call Firestore' >> beam.ParDo(
                CallFirestore(user_options.firestore_document)) 
        | 'Read DB2' >> beam.Map(ReadDB2))

I want the user_options.firestore_document to be usable in other functions without having to do a ParDo

Hola! What do you mean by RunTime value? Do you mean this user_options.firestore_document variable? — Pablo
– Pablo, Commented Oct 4, 2019 at 17:01
What do you mean by user_options.firestore_document to be usable in other functions without having to do a ParDo ? — guillaume blaquiere
– guillaume blaquiere, Commented Oct 5, 2019 at 12:03
@Pablo Hola! Yes, that is my RunTime value, that value will only be filled when the template is executed and not when the template is build, right now if I try to read that value inside any function that isn't a ParDo function I will get errors about it being empty — Luis Antonio Gutierrez Flores
– Luis Antonio Gutierrez Flores, Commented Oct 7, 2019 at 13:06
@guillaumeblaquiere What I mean is that if I try to use that variable in any function, which is only passed when executing the template and not when creating it, the template won't generate and I will start getting errors about my variable being empty — Luis Antonio Gutierrez Flores
– Luis Antonio Gutierrez Flores, Commented Oct 7, 2019 at 13:09
@Pablo Oh so those can only be used inside a ParDo :( I was able to achieve what I was trying to do with the ParDo, but thought there was another way to do it, Thanks! — Luis Antonio Gutierrez Flores
– Luis Antonio Gutierrez Flores, Commented Oct 10, 2019 at 14:07

Pablo · Accepted Answer · 2019-10-10 16:58:06Z

6

The only way in which you can use value providers are in ParDos, and Combines. It is not possible to pass a value provider in a create, but you can define a DoFn that returns the value provider you pass to it in the constructor:

class OutputValueProviderFn(beam.DoFn):
  def __init__(self, vp):
    self.vp = vp

  def process(self, unused_elm):
    yield self.vp.get()

And in your pipeline, you would do the following:

user_options = pipeline_options.view_as(UserOptions)

with beam.Pipeline(options=pipeline_options) as p:
  my_value_provided_pcoll = (
      p
      | beam.Create([None])
      | beam.ParDo(OutputValueProviderFn(user_options.firestore_document))

That way you wouldn't use it in a Create, as it's not possible, but you could still get it in a PCollection.

answered Oct 10, 2019 at 16:58

Pablo

11.1k1 gold badge49 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Manuel Huppertz Over a year ago

Hey @pablo, that was really helpful to me today, thanks!. But where was I supposed to find that information? Is it in the docs? I tried to find it but couldn't. Any recommendations for sources?

Collectives™ on Stack Overflow

Ways of using value provider parameter in Python Apache Beam

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related