Selecting menu item using PhantomJS

Question

I have simple PhantomJS script to parse Javascript content of website to html. (Some data is then extracted from the html code using other tool.)

var page = require('webpage').create();
var fs = require('fs');// File System Module
var output = '/tmp/sourcefile'; // path for saving the local file
page.open('targeturl', function() { // open the file
  fs.write(output,page.content,'w'); // Write the page to the local file using page.content
  phantom.exit(); // exit PhantomJs
});

(I got these lines of code from http://kochi-coders.com/2014/05/06/scraping-a-javascript-enabled-web-page-using-beautiful-soup-and-phantomjs/)

This used to work when all targets had direct links. Now they are behind the same url and there is drop down menu:

<select id="observation-station-menu" name="station" onchange="updateObservationProductsBasedOnForm(this);">
  <option value="101533">Alajärvi Möksy</option>
  ...    
  <option value="101541">Äänekoski Kalaniemi</option>
  </select>

This is the menu item I would actually like to load:

<option value="101632">Joensuu Linnunlahti</option>

Because of this menu my script only downloads data related to the default location. How I load contents of other item from the menu and download html content of that item instead?

My target site is this: http://ilmatieteenlaitos.fi/suomen-havainnot

(If there is better way than PhantomJS for doing this I could use it just as well. My interest is in dealing with the data once get it scraped and I chose PhantomJS just because it was the first thing that worked. Some options might be limited because my server is a Raspberry Pi and might not work on it: Python Selenium: Firefox profile error)

Gustavo F · Accepted Answer · 2015-12-16 18:27:34Z

3

Since the page have jQuery, you can do:

page.open('targeturl', function() { // open the file
  page.evaluate(function() {
    jQuery('#observation-station-menu').val('101632').change();
  });  //change the checkbox, then fires the event
  fs.write(output,page.content,'w'); // Write the page to the local file using page.content
  phantom.exit(); // exit PhantomJs
});

answered Dec 16, 2015 at 18:27

Gustavo F

2,20614 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user5542121 Over a year ago

Your variant is probably better, since it will be easier to update I guess.

Madoc Comadrin Over a year ago

I ran this but the resulting file still contained information of the default choice. From the file: <option value="100971" selected="selected">Helsinki Kaisaniemi</option>

Artjom B. Over a year ago

@MadocComadrin Well, duh, there is a delay between changing the value and seeing a new image. You need to add a delay to your script.

Community · Accepted Answer · 2017-05-23 12:15:33Z

1

You could directly call the function, which is defined in the underlying js on that page:

var page = require('webpage').create();
var fs = require('fs');// File System Module
var output = '/tmp/sourcefile'; // path for saving the local file
page.open('targeturl', function() { // open the file
  page.evaluate(function() {
     updateObservationProducts(101632, 'weather');
  });
  window.setTimeout(function () {
    fs.write(output,page.content,'w'); // Write the page to the local file using page.content
    phantom.exit(); // exit PhantomJs
  }, 1000); // Change timeout as required to allow sufficient time 

});

For the waiting to render, see this phantomjs not waiting for "full" page load , I copy pasted a part from rhunwicks solution.

edited May 23, 2017 at 12:15

CommunityBot

11 silver badge

answered Dec 16, 2015 at 18:28

user5542121

1,06312 silver badges28 bronze badges

1 Comment

Madoc Comadrin Over a year ago

This acted similarly to the other answer. The passed without errors but contained data of the default selection.

Collectives™ on Stack Overflow

Selecting menu item using PhantomJS

2 Answers 2

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related