How to run scripts from a HTML file using Python?

Question

I am building a scraper in Python and want to call scripts, which are in a HTML code, like web browsers do while entering a website. Is it possible to run scripts from Python level on the HTML code?

The purpose is to get the same HTML code as in 'DOM Inspector' in web browser, not the code you can simply download with

requests.get("https://example.com")

or when you use 'View Source' mode...

Let's say I get the code:

<!DOCTYPE html>
<html>
  <body>
    <h1>The script element</h1>
    <p id="demo"></p>

    <script>
      document.getElementById("demo").innerHTML = "Hello JavaScript!";
    </script>
  </body>
</html>

If I run the script it will change the content of <p> element and results will be

<!DOCTYPE html>
<html>
  <body>
    <h1>The script element</h1>
    <p id="demo">Hello JavaScript!</p>
  </body>
</html>

So how can I run it and get the HTML code evaluated with all scripts from a page?

EDIT 1: I know the library "Selenium", but I am trying to solve my problem without using browser simulators, just Python and JavaScript...

Thanks in advance!

Vishal Pandey · Accepted Answer · 2020-11-29 15:41:03Z

0

You should use Brython then only you can combine HTML and Python language else you have to Learn framework like Django please let me know if it was helpful for you.

answered Nov 29, 2020 at 15:41

Vishal Pandey

3592 gold badges4 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Timeler · Accepted Answer · 2020-11-29 15:42:02Z

0

You can use the library "selenium" to load the page, there are also elements included to click on website/ you can interact with it. to then scrape it, you might want to use the library "beautifulsoup4". you need to install both in cmd using

pip install selenium

and

pip install beautifulsoup4

answered Nov 29, 2020 at 15:42

Timeler

3871 silver badge11 bronze badges

1 Comment

AKedzierski Over a year ago

Selenium is too slow, as it launches a browser... I am looking for a solution which uses only Python and JavaScript

batuhand · Accepted Answer · 2020-11-29 15:42:53Z

0

Maybe you can get the response and write it to a html file

answered Nov 29, 2020 at 15:42

batuhand

4853 silver badges15 bronze badges

1 Comment

AKedzierski Over a year ago

Yeah, but this is exactly my problem... I get a HTML file (or just string) and now I want to run its scripts as browsers do... But I don't know how from Python level only

avarice · Accepted Answer · 2020-11-29 15:48:30Z

0

You are looking for a cross-platform testing framework, named Selenium

You can launch pages using a webdriver in browsers like Firefox or Chrome, and interact with page like you are scripting.

Docs

answered Nov 29, 2020 at 15:48

avarice

15.6k3 gold badges19 silver badges38 bronze badges

2 Comments

AKedzierski Over a year ago

Thanks, but... Selenium is pretty slow for me. As you said "it lauches pages in browsers" but I am trying to run scripts only using Python and JavaScript and no other apps. The performance matters for me and browsers slow down everything significantly...

AKedzierski Over a year ago

I have already get a HTML file (as a string) and now I want to execute all its scripts as browsers do, but I don't know how...

frankr6591 · Accepted Answer · 2020-11-29 16:59:27Z

This is partially answered by remove tags/stype-tags from html with lxml

It is unclear what exactly you need. From the example, you are just looking to remove <script ... /script> elements...

From The purpose is to get the same HTML code as in 'DOM Inspector' in web browser it seems you want to parse the HTML using python, ie. parse, display and show details of all the elements.

I personally prefer low level and lxml for html seems to do the trick...

Sample code:

from lxml import etree
from lxml import html
from lxml.html.clean import Cleaner

h = '''<!DOCTYPE html>
<html>
  <body>
    <h1>The script element</h1>
    <p id="demo"></p>

    <script>
      document.getElementById("demo").innerHTML = "Hello JavaScript!";
    </script>
  </body>
</html>'''

# Using etree 
root = etree.fromstring(h)
print(etree.tostring(root).decode())

# Code came from previous reference link above: https://stackoverflow.com/questions/8554035/remove-all-javascript-tags-and-style-tags-from-html-with-python-and-the-lxml-mod
cleaner = Cleaner()
cleaner.javascript = True # This is True because we want to activate the javascript filter
cleaner.style = True      # This is True because we want to activate the styles & stylesheet filter

# using html parser
print("\nWITH JAVASCRIPT & STYLES")
print(html.tostring(html.fromstring(h)).decode())
print("\nWITHOUT JAVASCRIPT & STYLES")
print(html.tostring(cleaner.clean_html(html.fromstring(h))).decode())

output:

<html>
  <body>
    <h1>The script element</h1>
    <p id="demo"/>

    <script>
      document.getElementById("demo").innerHTML = "Hello JavaScript!";
    </script>
  </body>
</html>

WITH JAVASCRIPT & STYLES
<html>
  <body>
    <h1>The script element</h1>
    <p id="demo"></p>

    <script>
      document.getElementById("demo").innerHTML = "Hello JavaScript!";
    </script>
  </body>
</html>

WITHOUT JAVASCRIPT & STYLES
<div>
  <body>
    <h1>The script element</h1>
    <p id="demo"></p>


  </body>
</div>

Collectives™ on Stack Overflow

How to run scripts from a HTML file using Python?

5 Answers 5

Comments

1 Comment

1 Comment

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

1 Comment

1 Comment

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related