Posting Form Data With idHTTP in Delphi

Question

I'm using Delphi with IdHTTP lib to get a form from this URL (http://apply.dataprocessors.com.au), handle the puzzle and post with the correct answer. But, apparently I am not handling the method POST properly, once the response retrieved is: "Problem with submission. Please try again."

Below is the Delphi source code:

procedure TForm1.Button1Click(Sender: TObject);
var
  sGET, sPOST, sGET_count: String;
  countCircles: integer;
  parameters: TStringList;
begin
  parameters := TStringList.Create;
  sGET := http.Get('http://apply.dataprocessors.com.au');
  sGET_count := StringReplace(sGET, '$', ' ', [rfReplaceAll, rfIgnoreCase]);
  sGET_count := StringReplace(sGET_count, 'unfilled', '$', [rfReplaceAll, rfIgnoreCase]);
  countCircles := 120 - sGET_count.CountChar('$');

  parameters.Add('title=submit');
  parameters.Add('value=' + countCircles.ToString());
  parameters.Add('submit=Submit');
  http.Request.ContentType := 'application/x-www-form-urlencoded';
  sPOST := http.Post('http://apply.dataprocessors.com.au/?', parameters);
  txtPost.Lines.Add(sPOST);
end;

Form retrieved

HTML code:

<html>
  <head>
  </head>
<body>
<form action="" method="POST">
      <input type="hidden" name="title" value="submit">
      <p>How many filled circles do you see:? <br><img src="unfilled_O.gif" height="12" width="12"><br></p>
      <p>Answer: <input type="text" name="value" value="" size="10"></p>
      <p><input type="Submit" value="Submit">
</form></body>
</html>

Response retrieved from POST method:

<html>
  <head>
  </head>
  <body>
  <p>Problem with submission. Please try again.</p></body>
</html>

Use a HTTP proxy to capture your browser communication and to see how a "working" request looks like — mjn
– mjn, Commented Oct 27, 2015 at 10:24
no need for that, use F12 function in any modern browser (or use firebug with firefox) and check out the network request. — whosrdaddy
– whosrdaddy, Commented Oct 27, 2015 at 10:51
It seems that you did not set the TIdHttp.Request.ContentType (should be 'application/x-www-form-urlencoded'). See also this answer stackoverflow.com/a/7769356/1744164 — Sir Rufo
– Sir Rufo, Commented Oct 27, 2015 at 10:54
@SirRufo: The TStrings version of TIdHTTP.Post() automatically sets the Request.ContentType to application/x-www.webform-urlencoded if it has not already been set. — Remy Lebeau
– Remy Lebeau, Commented Oct 27, 2015 at 18:31

Remy Lebeau · Accepted Answer · 2015-10-28 01:26:36Z

2

~~I see no problem with your TIdHTTP code (although the /? on the end of the URL for the Post() is not necessary).~~ There are missing steps in your TIdHTTP code. See further below for details.

I also see a problem with your StringReplace() code:

sGET := http.Get('http://apply.dataprocessors.com.au');
sGET_count := StringReplace(sGET, '$', ' ', [rfReplaceAll, rfIgnoreCase]);
sGET_count := StringReplace(sGET, 'unfilled', '$', [rfReplaceAll, rfIgnoreCase]);

Notice the first parameter of the second StringReplace(). In that second call, you are searching the original HTML that is returned by TIdHTTP.Get(). You are not searching the result of the first StringReplace(). And then you are discarding the result of the first StringReplace() altogether when you re-assign sGET_count, effectively making the first call a no-op. You probably meant to do this instead:

sGET := http.Get('http://apply.dataprocessors.com.au');
sGET_count := StringReplace(sGET, '$', ' ', [rfReplaceAll, rfIgnoreCase]);
sGET_count := StringReplace({sGET}sGET_count, 'unfilled', '$', [rfReplaceAll, rfIgnoreCase]);

Update: when I visit the URL in my webbrowser, I found a couple of things that your code is not taking into account. One, the webform includes a jobref field that you are not including in your POST submission. But more importantly, the webpage in question invokes a JavaScript function when the webpage is retrieved by a real webbrowser, but not when retrieved by your current TIdHTTP setup. That Javascript is located at http://apply.dataprocessors.com.au/funcs1.js and contains a single command to dynamically add a cookie to the webbrowser:

function init() {
  document.cookie = 'pc3o=d2r22; path=';
}

This command is invoked by the HTML:

<html>
  <head>
    ...
    <script type="text/javascript" src="funcs1.js"></script>
  </head>
<body onLoad="init()">
  ...
  </html>

That extra cookie gets sent back to the webserver when the webbrowser submits the webform, in addition to a normal PHP cookie that is present in the HTTP headers when the HTML is retrieved. You need to set the TIdHTTP.AllowCookies property to True (it is False by default) so TIdHTTP.Post() can send back the cookies, however Post() will only send the PHP cookie by default, not the Javascript cookie, for obvious reasons (TIdHTTP does not invoke client-side scripts).

If you change the TIdHTTP.Request.UserAgent property to mimic a real webbrowser, instead of using Indy's default UserAgent, the downloaded HTML will include that JavaScript reference. This is optional, as you can manually add the Javascript cookie (fortunately it is static, so you don't need to actually parse the Javascript, but you can if you want to) to the TIdHTTP.CookieManager regardless of the UserAgent, and Post() will still send it, and the server doesn't seem to care if the HTML included the JavaScript reference or not.

With that said, the following code works for me (I got the server to send a successful response to the POST):

procedure TForm1.Button1Click(Sender: TObject);
var
  sGET, sPOST, sGET_count: String;
  countCircles: integer;
  parameters: TStringList;
begin
  parameters := TStringList.Create;
  try
    http.AllowCookies := True;
    // optional: http.Request.UserAgent := 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko';

    sGET := http.Get('http://apply.dataprocessors.com.au');
    http.CookieManager.AddServerCookie('pc3o=d2r22; path=', http.URL);

    sGET_count := StringReplace(sGET, '$', '', [rfReplaceAll, rfIgnoreCase]);
    sGET_count := StringReplace(sGET_count, '"filled_O.gif"', '$', [rfReplaceAll, rfIgnoreCase]);
    countCircles := sGET_count.CountChar('$');

    parameters.Add('title=submit');
    parameters.Add('jobref=');
    parameters.Add('value=' + countCircles.ToString());

    http.Request.ContentType := 'application/x-www-form-urlencoded';
    http.Request.Referer := 'http://apply.dataprocessors.com.au';

    sPOST := http.Post('http://apply.dataprocessors.com.au', parameters);
    ShowMessage(sPOST);
  finally
    parameters.Free;
  end;
end;

edited Oct 28, 2015 at 1:26

answered Oct 27, 2015 at 18:30

Remy Lebeau

610k36 gold badges516 silver badges875 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Raoni Over a year ago

That's true, the second StringReplace() was wrong, what I meant is exactly what you explained. But the first StringReplace() is just for safety, just in case there is "$" in the HTML that would spoil the math. As expected, even fixing this I'm still getting the same message. Anyway, thanks for your help.

fantaghirocco Over a year ago

really interesting. Referring to "If you change the TIdHTTP.Request.UserAgent property to mimic a real webbrowser, instead of using Indy's default UserAgent, the downloaded HTML will include that JavaScript reference." would you be kind enough to explain why? Thank you in advance.

Remy Lebeau Over a year ago

Some webservers send different content to different user agents. Some browsers have quirks/behaviors that differ from other browsers, so sometimes webpage authors create browser-specific content, browser-specific scripts, etc. In this case, apply.dataprocessors.com.au is apparently useragent-aware, because it tweaks the HTML based on different User-Agent header values.

fantaghirocco Over a year ago

apparently too much useragent-aware :D So obvious that I couldn't figure it out. Thank you again sir

Raoni Over a year ago

Allowing and adding the cookie worked. Thanks for your help and excellent explanation about the solution.

mjn · Accepted Answer · 2015-10-27 11:07:27Z

0

Use F12 (or a HTTP proxy) to capture your browser communication and to see how a "working" request looks like.

Does the working request use cookies? If yes, enable cookie support for IdHttp.

edited Oct 27, 2015 at 11:07

answered Oct 27, 2015 at 10:25

mjn

36.7k30 gold badges185 silver badges392 bronze badges

2 Comments

Raoni Over a year ago

I checked the method POST on google chrome and I'm getting the same response already, but just while sending the wrong answer for the puzzle. When I send the right answer I get: Problem with submission. Please try again.

Remy Lebeau Over a year ago

Actually, you are sending the right answer, you are just not sending everything that the server is expecting you to send. See my answer for details.

Collectives™ on Stack Overflow

Posting Form Data With idHTTP in Delphi

2 Answers 2

5 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related