How to make recursive HTTP calls using RxJs operators?

Question

I use the following method in odder to retrieve data by passing pageIndex (1) and pageSize (500) for each HTTP call.

this.demoService.geList(1, 500).subscribe(data => {
    this.data = data.items;
});

The response has a property called isMore and I want to modify my method in odder to continue HTTP calls if isMore is true. I also need to merge the returned values and finally return the accumulated values.

For example, assuming that there are 5000 records and until 10th HTTP call, the service returns true for isMore value. After 10th HTTP call, it returns false and then this method sets this.data value with the merged 5000 records. For this problem, should I use mergeMap or expand or another RxJs operator? What is the proper way to solve this problem?

Update: I use the following approach, but it does not merge the returned values and not increase the pageIndex. For this reason it does not work (I tried to make some changes, but could not make it work).

let pageIndex = 0;
this.demoService.geList(pageIndex+1, 500).pipe(
    expand((data) => {
        if(data.isComplete) {
            return of(EMPTY);
        } else {
            return this.demoService.geList(pageIndex+1, 500);
        }
    })
).subscribe((data) => {
    //your logic here
});

Update II:

of({
    isMore : true,
    pageIndex: 0,
    items: []
  }).pipe(
    expand(data => demoService.geList(data.pageIndex+1, 100)
    .pipe(
      map(newData => ({...newData, pageIndex: data.pageIndex+1}))
    )),
    // takeWhile(data => data.isMore), //when using this, it does not work if the total record is less than 100
    takeWhile(data => (data.isMore || data.pageIndex === 1)), // when using this, it causing +1 extra HTTP call unnecessarily
    map(data => data.items),
    reduce((acc, items) => ([...acc, ...items]))
  )
  .subscribe(data => {
    this.data = data;
  });

Update III:

Finally I made it work by modifying Elisseo's approach as shown below. Howeveri **I need to make it void and set this.data parameter in this getData() method. How can I do this?

getData(pageIndex, pageSize) {
  return this.demoService.geList(pageIndex, pageSize).pipe(
    switchMap((data: any) => {
      if (data.isMore) {
        return this.getData(pageIndex+1, pageSize).pipe(
          map((res: any) => ({ items: [...data.items, ...res.items] }))
        );
      }
      return of(data);
    })
  );
}

I want to merge the following subscribe part to this approach but I cannot due to some errors e.g. "Property 'pipe' does not exist on type 'void'."

.subscribe((res: any) => {
    this.data = res;
});

I just need to accumulate returned values after recursive HTTP call. I think it is a general situation for paging or recursive needs. Any help please? — Jack
– Jack, Commented Jan 31, 2021 at 17:17
I hope RxJs and JavaScript Gurus see this problem and give a proper solution approach :) — Jack
– Jack, Commented Jan 31, 2021 at 17:17
if you use pagination, shouldn't you get the data after an event occurs for example pageIndex changed? why are you trying to get every data if your approach is pagination? if you want to get every data why you're set upper bound to get the data? Couldn't understand the logic you're trying to achieve . — Talha Akca
– Talha Akca, Commented Jan 31, 2021 at 18:19
Thanks amigo. Actually this method is used for pagination and called click next / prev button, you are right. But besides this, I also need to get records by checking the isMore value of the response, because giving a fixed number would not solve the problem as the records increase in the future. — Jack
– Jack, Commented Jan 31, 2021 at 18:27
In this example do not think pagination, just think that we need a recursive call by using the page number starting from 1 and then check the isMore value of the response. If it is true, continue to recursive call and merge it the previous result. If false, just return the records in the first response. — Jack
– Jack, Commented Jan 31, 2021 at 18:27

Eliseo · Accepted Answer · 2021-02-01 19:13:21Z

6

getData(pageIndex, pageSize) {
    return this.demoService.getList(pageIndex, pageSize).pipe(
      switchMap((data: any) => {
        if (!data.isCompleted) {
          return this.getData(pageIndex+1, pageSize).pipe(
            map((res: any) => ({ data: [...data.data, ...res.data] }))
          );
        }
        return of(data);
      })
    );
  }

stackblitz NOTE: I updated pasing as argument pageIndex+1 as @mbojko suggest -before I wrote pageIndex++

UPDATE 2

Using expand operator we need take account that we need feed the "recursive function" with an object with pageIndex -it's necesarry in our call- for this, when we make this.demoService.getList(data.pageIndex+1,10) we need "transform the result" adding a new property "pageIndex". for this we use "map"

  getData() {
    //see that initial we create "on fly" an object with properties: pageIndex,data and isCompleted
    return of({
      pageIndex:1,
      data:[],
      isCompleted:false
    }).pipe(
      expand((data: any) => {
        return this.demoService.getList(data.pageIndex,10).pipe(
            //here we use map to create "on fly" and object
            map((x:any)=>({
              pageIndex:data.pageIndex+1, //<--pageIndex the pageIndex +1
              data:[...data.data,...x.data], //<--we concatenate the data using spread operator
              isCompleted:x.isCompleted}))  //<--isCompleted the value
        )
      }),
      takeWhile((data: any) => !data.isCompleted,true), //<--a take while
            //IMPORTANT, use "true" to take account the last call also
      map(res=>res.data)  //finally is we only want the "data" 
                          //we use map to return only this property
    )
  }

Well we can do a function like this:

  getData() {
    of({pageIndex:1,data:[],isCompleted:false}).pipe(
      expand((data: any) => {
        return this.demoService.getList(data.pageIndex,10).pipe(
            tap(x=>{console.log(x)}),
            map((x:any)=>({
              pageIndex:data.pageIndex+1,
              data:[...data.data,...x.data],
              isComplete:x.isComplete}))
        )
      }),
      takeWhile((data: any) => !data.isComplete,true), //<--don't forget the ",true"
    ).subscribe(res=>{
       this.data=res.data
    })
  }

See that in this case we don't return else simple subscribe to the function and equal a variable this.data to res.data -it's the reason we don't need the last map

Update 3 by Mrk Sef

Finally, if you don't want your stream to emit intermittent values and you just want the final concatenated data, you can remove the data concatenation from expand, and use reduce afterward instead.

  getData() {
    of({
      pageIndex: 1,
      data: [],
      isCompleted: false
    })
      .pipe(
        expand((prevResponse: any) => this.demoService.getList(prevResponse.pageIndex, 10).pipe(
            map((nextResponse: any) => ({
              ...nextResponse,
              pageIndex: prevResponse.pageIndex + 1
            }))
          )
        ),
        takeWhile((response: any) => !response.isCompleted, true),
        // Keep concatenting each new array (data.items) until the stream
        // completes, then emit them all at once
        reduce((acc: any, data: any) => {
          return [...acc, ...data.data];
        }, [])
      )
      .subscribe(items => {
        this.data=items;
      });
  }

edited Feb 1, 2021 at 19:13

answered Jan 31, 2021 at 19:22

Eliseo

58.9k4 gold badges39 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

13 Comments

mbojko Over a year ago

Instead of pageIndex++; return this.getData(pageIndex, pageSize), just return this.getData(pageIndex + 1, pageSize).

Jack Over a year ago

@Eliseo Yes, exactly what I want and it does not send extra HTTP requests. There are 810 records and if I set page size 100, it sends 9 HTTP calls, if the page size is 500, it sends 2 HTTP calls :)

Jack Over a year ago

I needed to make some modifications but finally made it work. On the other hand, how can I make it void and set this.data in the getData method instead of onInit?

Jack Over a year ago

By the way I cannot vote up, as I have not enough reputation. Could you pls vote up?

Eliseo Over a year ago

@Max, I just corrected a few the code, changing the "pageIndex", Really I don't like so much the before getList(data.pageIndex+1,10) but you need start with 1

|

Talha Akca · Accepted Answer · 2021-01-31 23:25:38Z

2

It doesn't matter if you're total record change as long as api response give you the isMore flag.

I'm skipping the part how to implement reducer action event i'm assuming you've already done that part. So i will just try to explain with pseudo codes.

You have a table or something like that with pagination data. on intial state you can just create an loadModule effect or using this fn:

getPaginationDataWithPageIndex(pageIndex = 1){ this.store.dispatch(new GetPaginationData({ pageIndex: pageIndex, dataSize: 500})); }

in your GetPaginationData effect

... map(action => {
return apicall.pipe(map((response)=> {
   if(response.isMore){
    return new updateState({data:response.data, isMore: responseisMore})
} else {
   return new updateState({isMore: response.isMore}),
}
}})
`

all you have to left is subscribing store in your .ts if isMore is false you will not display the next page button. and on your nextButton or prevButton's click method you should have to just dispatch the action with pageIndex

edited Jan 31, 2021 at 23:25

answered Jan 31, 2021 at 19:01

Talha Akca

4023 silver badges10 bronze badges

6 Comments

Jack Over a year ago

Forgot everything and just have a look at my Update II please. I added 2 comments regarding the problem wit that approach. Any idea?

Jack Over a year ago

Thanks Talha, but I am confused and would be happy if you edit your answer according to the method that use. Update II is better to edit if I fix the extra 1 HTTP call problem.

Jack Over a year ago

Thanks a lot for your help. I want to vote up your solution, but unfortunately could not as I have no enough reputation. Any vote up for the question please? Regards.

Talha Akca Over a year ago

@Max glad that i can help

Jack Over a year ago

Do you have any idea about my Update III?

|

katzenhut · Accepted Answer · 2021-02-01 09:34:57Z

0

I do not think recursion is the correct approach here:

interval(0).pipe(
    map(count => this.demoService.getList(count + 1, 500)),
    takeWhile(reponse => response.isMore, true),
    reduce((acc, curr) => //reduce any way you like),
).subscribe();

This should make calls to your endpoint until the endpoint returns isMore === false. The beautiful thing about interval is that we get the count variable for free.

But if you are set on using recrsion, here is the rxjs-way to do that using the expand-operator (see the docs). I find it slightly less readable, as it requires an if-else-construct which increases code complexity. Also the outer 'counter' variable just isn't optimal.

let index = 1;
this.demoService.geList(index, 500).pipe(
    expand(response => response.isMore ? this.demoService.geList(++index, 500) : empty()),
    reduce((acc, curr) => //reduce here)
).subscribe();

edited Feb 1, 2021 at 9:34

answered Jan 31, 2021 at 18:44

katzenhut

1,74218 silver badges26 bronze badges

11 Comments

Jack Over a year ago

Thanks for help. I also tried takeWhile before asking the question, but in this case I send an extra 1 HTTP call as I check the condition after call. But maybe in yoru approach the problem not occurs. However, I could not make it work due to this.processResponse. In this line I need to merge each result and I do not know should I use HTTP call method instead of interval(1000). Any idea?

Jack Over a year ago

You could also edit teh approach in my Update.

katzenhut Over a year ago

@Max - I m having trouble picturing this. could you explain what you expect 'data' to be in your subscribe()?

katzenhut Over a year ago

@Max - have you tried using reduce? rxjs-dev.firebaseapp.com/api/operators/reduce

Jack Over a year ago

I expect object array ad data. Reduce may be good option, but why could not you post an answer according to my code in Update section of the question? There is no interval etc in my question.

|

Collectives™ on Stack Overflow

How to make recursive HTTP calls using RxJs operators?

3 Answers 3

13 Comments

6 Comments

11 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

13 Comments

6 Comments

11 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related