Save json file data in to Rails database

Question

I have a Rails API using rails 5.2.1 and I'm developing a Web Scraping API that extracts web data using the Mechanize Gem.

What I'm going to do is, I'm going to pass all of this web scraping data data to React.js but first I need to get this data on my database which I'm having hard time how to figure this out.

So when I extract my data and I receive it all in a JSON file (called data.json), which I'm stunned I got until here haha.

{
  "Brazil":[
  {"Jungle Plants":[bla bla bla ]},
  {"Desert Plants":[ bla bla bla ]}],

  "Egypt":[
  {"Jungle Plants":[bla bla bla ]},
  {"Desert Plants":[ bla bla bla ]}]
  
  and so on...
}

So the next thing I need to do is separate my JSON data into my database which I already have a migration setup like this:

class CreatePlants < ActiveRecord::Migration[5.2]
  def change
    create_table :plants do |t|
      t.string :country_name
      t.string :plant_categories

      t.timestamps
    end
  end
end

So up in the table what I want to is, I want that the country name like 'Brazil' or 'Egypt' or whatever country name is goes to the country_name column and so for the plant_categories.

My question is, how can I organize/separate my data.json file in to my database? I will appreciate so much your help! <3

You could read the JSON file and iterate over the objects and store(iterate using k,v and store that in the respective columns). Is there some problem with that ? — Alok Swain
– Alok Swain, Commented Aug 28, 2020 at 21:45
Not that there's a problem with that, I have like dozens of options and on those dozens of options I get totally blocked on what is the best approach for this. I like what you said, so this means I can read the file and iterate over these. But where can I read the json file? On a controller? or a model? I appreciate your response. — theKid
– theKid, Commented Aug 28, 2020 at 21:48
It's better to incapsulate import logic in a separate class (service/importer). Then you can call that importer after you parse data. Model or Controller shouldn't contain such kinds of logic. — Maksim P
– Maksim P, Commented Aug 28, 2020 at 23:11

Patricio S · Accepted Answer · 2020-08-29 22:25:04Z

2

This would be my approach, I would use a JSON or JSONB column to save each array as they come from the body, so I would change the migration to this:

class CreatePlants < ActiveRecord::Migration[5.2]
  def change
    create_table :plants do |t|
      t.string :country_name
      t.jsonb :data

      t.timestamps
    end
  end
end

So let's say you have your parsed response (after you applied JSON.parse on the body), resulting in the following hash:

response =
  {
    "Brazil": [
      { "Jungle Plants": [ bla bla bla ] },
      { "Desert Plants": [ bla bla bla ] }
    ],
    "Egypt": [
      { "Jungle Plants": [ bla bla bla ] },
      { "Desert Plants": [ bla bla bla ] }
    ]

    and so on...
  }

I'd do:

response.each { |key, value| Plant.create!(country_name: key, data: value) }

That way, if you execute the following, after the previous statement:

Plant.find_by(country_name: 'Brazil').data

You would get your array:

[{ "Jungle Plants": [ bla bla bla ] }, { "Desert Plants": [ bla bla bla ] }]

But it is your decision, you can perfectly tackle this "problem" in different ways, for example, you could save the arrays as an string and then retrieve them from the DB and apply a JSON.parse or apply the serialize method in the model definition to convert them to an array again. Another option is to filter the response even further and save them in many more columns.

I would say that the best way depends on what you want to do with the data afterwards, but for flexibility reasons my choice would be the one I stated before, to save the array as they come from the API to a JSONB column and then do whatever I want with them (another advantage is that saving the data as JSONB allows you to perfom queries on them in ways you couldn't do with just a string)

edited Aug 29, 2020 at 22:25

answered Aug 29, 2020 at 5:09

Patricio S

2,1303 gold badges24 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

theKid Over a year ago

By the way, where can I apply JSON my parse? In a controller? In my model? Or do you recommend creating a service folder a create my Parse there? I would appreciate your help!

Patricio S Over a year ago

I generally create service objects to handle the core logic of my apps (in app/services directory), that way my controllers and models are as thin as possible, but again, you can do it wherever you want.

Collectives™ on Stack Overflow

Save json file data in to Rails database

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related