Skip to main content
Software Engineering

Return to Question

edited title
Link
Benhassine Mohamed
Benhassine Mohamed
  • 13
  • 4

Ingest data from our partener : API POST vs Event DriveDriven Architecture

added 1057 characters in body
Source Link
Benhassine Mohamed
Benhassine Mohamed
  • 13
  • 4

THE HISTORY OF THE REQUIREMENT :THE HISTORY OF THE REQUIREMENT :

MY PARTNER SEND cvs files via MFT to MY CLIENT , many times a day MY .

MY CLIENT has a listener ( cron job in linux and batch ) , when a file is received in a folder , hethe client consumes it and store it in database ( SQL SERVER)

Regarding data , the estimated trafic : 700 files a day Millions of lines in cvs file to be consumed

The new solution there is to transferno order in data via API and not MFT :, each time a file is received it is processed by batch .

The new solution is to transfer data via API REST CALL and not OLD MFT :

The need is to retrieve huge data from a partner by calling it'sAPI REST APIs   (JSON) to finally store the result in our SQL database (not too much processing to do in between)

SOLUTION 1 : Event Driven Architecture :SOLUTION 1 : Event Driven Architecture and PARTNER EXPOSE API REST TO RETRIEVE DATA :

MY CLIENT : Based on this message , wemy client CALL the REST API ( GET exposed by our PARTNER ) to GET THE DATA FROM JSON RESPONSE of the call HTTP ( this will be implemented for example by spring batch )

SOLUTION 2 : THE PARTNER DO NOT EXPOSE API , MY CLIENT WILL DO THIS !SOLUTION 2 : THE PARTNER DO NOT EXPOSE API , MY CLIENT WILL DO THIS !

OUR PARTNER ==> THIS time , our Partner will call THIS POST API REST AND send all data in REQUEST BODY , as a result , the partner update our SQL DATABASE when he calls our API REST ( CLIENT CALL THE POST API WHEN HE WANTS TO UPDATE DATA)

Notes :MY Notes :

My requirements : evaluates the pros and cons of these solutionsMy requirements : evaluates the pros and cons of these solutions

For my part, I opt for the first solutionSOLUTION 1 because :

pos1 : the processing will be asynchronous pos2 : we will be the master of our data to be added to our database (EDD is it good as an argument ? ) since pos3 : processing in almost real time cons1 : architecture is complex when in we move to cloud cons2 : Partner is not familiar perhaps with this solution

  • the processing will be asynchronous
  • and we will be the master of our data to be added to our database
  • processing in almost real time
  • architecture is complew
  • Partner is not familiar perhaps with this solution

The second solution ( we expose API , the partner simply call it ) pos1: is rather light (less expensive) it is an advantage, cons1 : but has the disadvantage that the processing will be done synchronously cons2 : that we will not be master of the data to add to our REF (unless checks to be made in the POST action to be exposed) cons3 : call of API will be an additional charge by our partner

  • is rather light (less expensive) it is an advantage,
  • but has the disadvantage that the processing will be done synchronously - and that we will not be master of the data to add to our REF (unless checks to be made in the POST action to be exposed)
  • call of API will be an additional charge by our partner

Thank you for your enrichmentsfeedbacks : (advantages / disadvantages) cordiallyRegards

THE HISTORY OF THE REQUIREMENT :

MY PARTNER SEND cvs files via MFT to MY CLIENT , many times a day MY CLIENT has a listener , when a file is received , he consumes it and store it in database

Regarding data , the estimated trafic : 700 files a day Millions of lines in cvs file to be consumed

The new solution is to transfer data via API and not MFT :

The need is to retrieve huge data from a partner by calling it's REST APIs (JSON) to finally store the result in our SQL database (not too much processing to do in between)

SOLUTION 1 : Event Driven Architecture :

MY CLIENT : Based on this message , we CALL the REST API ( exposed by our PARTNER ) to GET THE DATA FROM JSON RESPONSE of the call HTTP

SOLUTION 2 : THE PARTNER DO NOT EXPOSE API , MY CLIENT WILL DO THIS !

OUR PARTNER ==> THIS time , our Partner will call THIS POST API REST AND send all data in REQUEST BODY , as a result , the partner update our SQL DATABASE when he calls our API REST

Notes :

My requirements : evaluates the pros and cons of these solutions

For my part, I opt for the first solution (EDD) since

  • the processing will be asynchronous
  • and we will be the master of our data to be added to our database
  • processing in almost real time
  • architecture is complew
  • Partner is not familiar perhaps with this solution

The second solution

  • is rather light (less expensive) it is an advantage,
  • but has the disadvantage that the processing will be done synchronously - and that we will not be master of the data to add to our REF (unless checks to be made in the POST action to be exposed)
  • call of API will be an additional charge by our partner

Thank you for your enrichments (advantages / disadvantages) cordially

THE HISTORY OF THE REQUIREMENT :

MY PARTNER SEND cvs files via MFT to MY CLIENT , many times a day .

MY CLIENT has a listener ( cron job in linux and batch ) , when a file is received in a folder , the client consumes it and store it in database ( SQL SERVER)

Regarding data , the estimated trafic : 700 files a day Millions of lines in cvs file to be consumed there is no order in data , each time a file is received it is processed by batch .

The new solution is to transfer data via API REST CALL and not OLD MFT :

The need is to retrieve huge data from a partner by calling API REST   (JSON) to finally store the result in our SQL database (not too much processing to do in between)

SOLUTION 1 : Event Driven Architecture and PARTNER EXPOSE API REST TO RETRIEVE DATA :

MY CLIENT : Based on this message , my client CALL the REST API ( GET exposed by our PARTNER ) to GET THE DATA FROM JSON RESPONSE of the call HTTP ( this will be implemented for example by spring batch )

SOLUTION 2 : THE PARTNER DO NOT EXPOSE API , MY CLIENT WILL DO THIS !

OUR PARTNER ==> THIS time , our Partner will call THIS POST API REST AND send all data in REQUEST BODY , as a result , the partner update our SQL DATABASE when he calls our API REST ( CLIENT CALL THE POST API WHEN HE WANTS TO UPDATE DATA)

MY Notes :

My requirements : evaluates the pros and cons of these solutions

For my part, I opt for the SOLUTION 1 because :

pos1 : the processing will be asynchronous pos2 : we will be the master of our data to be added to our database ( is it good as an argument ? ) pos3 : processing in almost real time cons1 : architecture is complex when in we move to cloud cons2 : Partner is not familiar perhaps with this solution

The second solution ( we expose API , the partner simply call it ) pos1: is rather light (less expensive) it is an advantage, cons1 : but has the disadvantage that the processing will be done synchronously cons2 : that we will not be master of the data to add to our REF (unless checks to be made in the POST action to be exposed) cons3 : call of API will be an additional charge by our partner

Thank you for your feedbacks : (advantages / disadvantages) Regards

added 1057 characters in body
Source Link
Benhassine Mohamed
Benhassine Mohamed
  • 13
  • 4

THE HISTORY OF THE REQUIREMENT :

MY PARTNER SEND cvs files via MFT to MY CLIENT , many times a day MY CLIENT has a listener , when a file is received , he consumes it and store it in database

Regarding data , the estimated trafic : 700 files a day Millions of lines in cvs file to be consumed

The new solution is to transfer data via API and not MFT :

The need is to retrieve huge data from a partner by calling it's REST APIs (JSON) to finally store the result in our SQL database (not too much processing to do in between)

EDDSOLUTION 1 : If we start with Event Driven Architecture  : partner X sends

THERE ARE STEPS OF THIS SOLUTION :

OUR PARTNER : create an event in our BROKER QUEUE (message) to a queue to notify Message indicates that there is new data to recover, my client consumesget : name API RESSOURCE TO CALL / parameters / Bussines Domain )

MY CLIENT : GET the data by callingmessage and read it from the Rest APIs of partener followingQUEUE

MY CLIENT : Based on this eventmessage , we CALL the REST API (batch which does this example Spring Batch exposed by our PARTNER ) to GET THE DATA FROM JSON RESPONSE of the call HTTP

we expose a RESTMY CLIENT : STORE The result of this huge response on SQL DATABASE without many modification

SOLUTION 2 : THE PARTNER DO NOT EXPOSE API and HTTP POST, MY CLIENT WILL DO THIS !

THERE ARE STEPS OF THISE SOLUTION : We expose ourselves a

MY CLIENT ==> EXPOSE A REST API (VERB POST) to help our partner to update dataPARTNER (huge volume) in our repositories: THIS ENDPOINT IS a POST HTTP OPERATION

OUR PARTNER ==> THIS time (this action, our Partner will save the JSONcall THIS POST API REST AND send all data in the database) soREQUEST BODY , as a result , the partner will callupdate our newSQL DATABASE when he calls our API REST to update our data

For my part, I opt for the first solutionNotes :

THE PARTNER IS OPEN to ALL OUR SUGGESTIONS (EDD SOLUTION 1 or 2 ) since the processing will be asynchronous and we estimates that it there will be the master of ourhuge data to be added to our databasetransfers between Partner / processing in almost real timeClient

The second solution is rather lightMy requirements (less expensive) it is an advantage, but has the disadvantage that: evaluates the processing will be done synchronouslypros and that we will not be mastercons of these solutions

For my part, I opt for the data to add to our REFfirst solution (unless checks to be made in the POST action to be exposedEDD) / call charge for our partnersince

  • the processing will be asynchronous
  • and we will be the master of our data to be added to our database
  • processing in almost real time
  • architecture is complew
  • Partner is not familiar perhaps with this solution

is there any other criteria to base myself on in order to pick the right decision ?The second solution

  • is rather light (less expensive) it is an advantage,
  • but has the disadvantage that the processing will be done synchronously - and that we will not be master of the data to add to our REF (unless checks to be made in the POST action to be exposed)
  • call of API will be an additional charge by our partner

The need is to retrieve huge data from a partner by calling it's REST APIs (JSON) to finally store the result in our SQL database (not too much processing to do in between)

EDD: If we start with Event Driven Architecture: partner X sends an event (message) to a queue to notify that there is new data to recover, my client consumes the data by calling the Rest APIs of partener following this event (batch which does this example Spring Batch)

we expose a REST API and HTTP POST: We expose ourselves a REST API (VERB POST) to help our partner to update data (huge volume) in our repositories (this action will save the JSON in the database) so the partner will call our new API REST to update our data

For my part, I opt for the first solution (EDD) since the processing will be asynchronous and we will be the master of our data to be added to our database / processing in almost real time

The second solution is rather light (less expensive) it is an advantage, but has the disadvantage that the processing will be done synchronously and that we will not be master of the data to add to our REF (unless checks to be made in the POST action to be exposed) / call charge for our partner

is there any other criteria to base myself on in order to pick the right decision ?

THE HISTORY OF THE REQUIREMENT :

MY PARTNER SEND cvs files via MFT to MY CLIENT , many times a day MY CLIENT has a listener , when a file is received , he consumes it and store it in database

Regarding data , the estimated trafic : 700 files a day Millions of lines in cvs file to be consumed

The new solution is to transfer data via API and not MFT :

The need is to retrieve huge data from a partner by calling it's REST APIs (JSON) to finally store the result in our SQL database (not too much processing to do in between)

SOLUTION 1 : Event Driven Architecture  :

THERE ARE STEPS OF THIS SOLUTION :

OUR PARTNER : create an event in our BROKER QUEUE ( Message indicates that there is new data to get : name API RESSOURCE TO CALL / parameters / Bussines Domain )

MY CLIENT : GET the message and read it from the QUEUE

MY CLIENT : Based on this message , we CALL the REST API ( exposed by our PARTNER ) to GET THE DATA FROM JSON RESPONSE of the call HTTP

MY CLIENT : STORE The result of this huge response on SQL DATABASE without many modification

SOLUTION 2 : THE PARTNER DO NOT EXPOSE API , MY CLIENT WILL DO THIS !

THERE ARE STEPS OF THISE SOLUTION :

MY CLIENT ==> EXPOSE A REST API to our PARTNER : THIS ENDPOINT IS a POST HTTP OPERATION

OUR PARTNER ==> THIS time , our Partner will call THIS POST API REST AND send all data in REQUEST BODY , as a result , the partner update our SQL DATABASE when he calls our API REST

Notes :

THE PARTNER IS OPEN to ALL OUR SUGGESTIONS ( SOLUTION 1 or 2 ) and we estimates that it there will be huge data transfers between Partner / Client

My requirements : evaluates the pros and cons of these solutions

For my part, I opt for the first solution (EDD) since

  • the processing will be asynchronous
  • and we will be the master of our data to be added to our database
  • processing in almost real time
  • architecture is complew
  • Partner is not familiar perhaps with this solution

The second solution

  • is rather light (less expensive) it is an advantage,
  • but has the disadvantage that the processing will be done synchronously - and that we will not be master of the data to add to our REF (unless checks to be made in the POST action to be exposed)
  • call of API will be an additional charge by our partner
Source Link
Benhassine Mohamed
Benhassine Mohamed
  • 13
  • 4