I want to insert data into MySQL database with Django

Question

I am doing web scraping and I want to store the information in the database. I have the connection with the database in the 'settings' file like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'dbname',
        'USER': 'root',
        'PASSWORD': 'password',
        'HOST': 'localhost',
        'PORT': 'port'
    }
}

The web scraping I have it in another file like this:

from bs4 import BeautifulSoup
import requests

url = "https://www.somepage.com"
result = requests.get(url)
soup = BeautifulSoup(result.text, "html.parser")

find_by_class = soup.find('div', attrs={"class":"body"}).find_all('p')

I want to store what is in find_by_class in the database.

Also I have created the models.

from django.db import models

class SomeModel(models.Model):
    description = models.TextField(max_length=1000)

Show us what you have tried to connect your code snippet with your model! So far I see no relation between the two. — Klaus D.
– Klaus D., Commented Nov 24, 2021 at 13:25
@KlausD. I tried to make an from models import SomeModel in the scraping file and then tried this description = find_by_class . But of course it didn't work. — Kyogi
– Kyogi, Commented Nov 24, 2021 at 13:35

Adrian Mole · Accepted Answer · 2021-11-24 14:22:52Z

2

I assume you have done migrations in database (if not then inside your project's root folder, run python manage.py makemigrations and then python manage.py migrate).

Now you need to use the following snippet:

from bs4 import BeautifulSoup
import requests
from .models import SomeModel

url = "https://www.somepage.com"
result = requests.get(url)
soup = BeautifulSoup(result.text, "html.parser")
find_by_class = soup.find('div', attrs={"class":"body"}).find_all('p') 
for each in find_by_class:
    SomeModel.objects.create(description=each.text)

edited Nov 24, 2021 at 14:22

Adrian Mole

52.2k193 gold badges61 silver badges101 bronze badges

answered Nov 24, 2021 at 13:35

Dipan Bhusal

212 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Kyogi Over a year ago

Yes I did migrations and create 0001_initial file.

Kyogi Over a year ago

This appear in the output: "ModuleNotFoundError: No module named 'models' "

Dipan Bhusal Over a year ago

The file containing scrapping code and models should be in same django app folder. else import model from someapp like from appname.models import SomeModel

Kyogi Over a year ago

Now I get this: 'ImportError: attempted relative import with no known parent package'. Try to create a new project to verify and the same happens.

Kyogi Over a year ago

Remove the '.' from models and get this:

ImproperlyConfigured( django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the  environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

Collectives™ on Stack Overflow

I want to insert data into MySQL database with Django

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related