How to loop over two python Generators

Question

I have two python generators. Say

1) txn_gen, yield the dictionary values like

{'id': 1,'ref_no': 4323453536, 'amt': 678.00, 'txn_date': '12-11-2019'}
.
.
.
{'id':10000000 , 'ref_no':8523118426, 'amt':98788.00, 'txn_date': '12-11-2019'}

2) acc_gen, yield the dictionary values like

{'ref_no': 4323453536, 'acc_no': 123456789, 'amt': 98789.00}
.
.
.
{'ref_no': 8523118426, 'acc_no': 123456789, 'amt': 45654567.00}

I want to loop txn_gen over acc_gen for ref_no matching. I am looping like this.

for gen1 in txn_gen:
     for gen2 in acc_gen:
          if gen1[1] == gen2[0]:
               print(gen2)

But I am getting only one match value ie., the first match value. I am expecting millions of match values.

I want to improve the performance as I have millions of records.

Tim Pierce · Accepted Answer · 2019-12-09 01:24:18Z

1

A generator can only be evaluated once. After you've consumed all the values in acc_gen, and go on to the next value in txn_gen, you cannot loop through acc_gen again.

For this kind of analysis, you can iterate through txn_gen and save each ref_no in a hash table, and then iterate through acc_gen to look up their ref_no fields.

answered Dec 9, 2019 at 1:24

Tim Pierce

5,7041 gold badge18 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Selcuk · Accepted Answer · 2019-12-09 01:24:47Z

1

Once you have consumed a generator you can't iterate it again. One way is to convert them (or at least the inner one) to a list if the memory cost is acceptable:

acc_gen = list(acc_gen)
for gen1 in txn_gen:
   ...

If you cannot justify the space complexity, you must reset or re-initialise acc_gen before the second for statement.

answered Dec 9, 2019 at 1:24

Selcuk

60.1k12 gold badges114 silver badges119 bronze badges

Collectives™ on Stack Overflow

How to loop over two python Generators

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related