Submit form using requests package

Question

I know that there are many similar questions in SO, but I've tried every combination of get, post, data, params, etc in the requests package and I can not get the form to submit. I either get back an empty string, or the same page (ie: not the page I'm expecting to get after hitting Submit)

import requests
webserver = 'http://stev.oapd.inaf.it/cgi-bin/cmd'

# Returns the landing site
r = requests.get(webserver, data={'submit_form': 'Submit'}).text
# Returns an empty string
r = requests.get(webserver, params={'submit_form': 'Submit'}).text
# Returns an empty string
r = requests.post(webserver, data={'submit_form': 'Submit'}).text
# Returns the landing site
r = requests.post(webserver, params={'submit_form': 'Submit'}).text

The site changed recently, and I remember that one of these commands definitely used to work. Why is neither working now?

Did you add the other required form values on the page into the params sent with the request? — blhsing
– blhsing, Commented Apr 11, 2019 at 21:38
The form is supposed to work as is, meaning it already has default parameters loaded. I did try adding more parameters, but it made no change. — Gabriel
– Gabriel, Commented Apr 11, 2019 at 21:39
The default values are specified only in the browser. The actual request still needs to have all the required values. Make sure all the form values are sent as params. — blhsing
– blhsing, Commented Apr 11, 2019 at 21:40

Nazim Kerimbekov · Accepted Answer · 2019-04-12 21:45:44Z

As you mentioned the website has changed recently. It appears that it is now using form-data in order to receive the POST requests and also requires the default values to be sent.

All in all, your code should look like this:

import requests

webserver = 'http://stev.oapd.inaf.it/cgi-bin/cmd'

headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5"}

files = {
    'submit_form': (None,"Submit"),
    'cmd_version': (None,"3.2"),
    'track_parse': (None,"parsec_CAF09_v1.2S"),
    'track_colibri': (None,"parsec_CAF09_v1.2S_S35"),
    'track_postagb': (None,"no"),
    "n_inTPC": (None,"10"),
    "eta_reimers": (None,"0.2"),
    "kind_interp": (None,"1"),
    "kind_postagb": (None,"-1"),
    "photsys_file": (None,"tab_mag_odfnew/tab_mag_ubvrijhk.dat"),
    "photsys_version": (None,"YBC"),
    "dust_sourceM": (None,"dpmod60alox40"),
    "dust_sourceC": (None,"AMCSIC15"),
    "kind_mag": (None,"2"),
    "kind_dust": (None,"0"),
    "extinction_av": (None,"0.0"),
    "extinction_coeff": (None,"constant"),
    "extinction_curve": (None,"cardelli"),
    "imf_file": (None,"tab_imf/imf_kroupa_orig.dat"),
    "isoc_isagelog": (None,"0"),
    "isoc_agelow": (None,"1.0e9"),
    "isoc_ageupp": (None,"1.0e10"),
    "isoc_dage": (None,"0.0"),
    "isoc_lagelow": (None,"6.6"),
    "isoc_lageupp": (None,"10.13"),
    "isoc_dlage": (None,"0.0"),
    "isoc_ismetlog": (None,"0"),
    "isoc_zlow": (None,"0.0152"),
    "isoc_zupp": (None,"0.03"),
    "isoc_dz": (None,"0.0"),
    "isoc_metlow": (None,"-2"),
    "isoc_metupp": (None,"0.3"),
    "isoc_dmet": (None,"0.0"),
    "output_kind": (None,"0"),
    "output_kind": (None,"0"),
    "output_evstage": (None,"1"),
    "lf_maginf": (None,"-15"),
    "lf_magsup": (None,"20"),
    "lf_deltamag": (None,"0.5"),
    "sim_mtot": (None,"1.0e4"),
    ".cgifields": (None,"track_parsec"),
    ".cgifields": (None,"extinction_coeff"),
    ".cgifields": (None,"dust_sourceC"),
    ".cgifields": (None,"output_kind"),
    ".cgifields": (None,"extinction_curve"),
    ".cgifields": (None,"dust_sourceM"),
    ".cgifields": (None,"isoc_ismetlog"),
    ".cgifields": (None,"output_gzip"),
    ".cgifields": (None,"photsys_version"),
    ".cgifields": (None,"track_colibri"),   
    }

r = requests.post(webserver, headers=headers, files = files)

Hope this helps!

Thank you Fozoro, this worked (although headers are not needed and there are some minor errors in the dictionary like the photsys_version element duplicated)
I've put the header just in case. Oops, thanks for pointing it out just edited my answer.

Collectives™ on Stack Overflow

Submit form using requests package

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related