Powershell filedownload from https

Question

I am trying to download several files using Powershell and its Invoke-WebRequest method.

I'm basically looping through several filenames (I know that they are available on the server) and download them.

My problem is that my script works for the first file and fails for every file that follows. When I open one of the later files (.csv`s) there is just some html code in it).

I already read a lot about passing session cookings but I am not sure if this is my problem or how I can do that.

My script so far looks like this:

$httpsUser = 'XXX'
$httpsPass = 'YYY'

foreach ($instrument in 'ivv','ijh','ijr','iwm') {

$Source = 'https://***', `
$instrument, '-en_us.csv' -join ""

$Target = 'C:\User\', `
$instrument, '-en_us.csv' -join ""

$uri = New-Object “System.Uri” “$Source”
$WebClient = [System.Net.HttpWebRequest]::Create($uri) 
$webclient.Proxy.Credentials =
[System.Net.CredentialCache]::DefaultNetworkCredentials
$webclient.Credentials =
New-Object System.Net.NetworkCredential($httpsUser,$httpsPass)

Invoke-WebRequest -Uri $Source -OutFile $Target
}

Thank you all and let me know what you think :)

What error message do you get? Can you validate, that the $Target path is valid (directory exisists) — Martin Brandl
– Martin Brandl, Commented Apr 8, 2016 at 6:59
Hi, yes targetpath is dynamic (in the foreach loop) and exists. Files get saved to the folder but only the first one has the correct content. The other files have as content: <html xmlns="w3.org/1999/xhtml" xml:lang="en"> <body onload="document.forms[0].submit()"> <noscript> <p> <strong>Note:</strong> Since your browser does not support JavaScript you must press the Continue button once to proceed. </noscript> </form> </body> </html> I had to shorten the html code... — Lennart
– Lennart, Commented Apr 8, 2016 at 7:04

Martin Brandl · Accepted Answer · 2016-04-08 07:15:05Z

1

It seems like you don't using the HttpWebRequest you created to download the file. Anyway, I would recommend using System.Net.WebClient:

$wc = New-Object System.Net.WebClient
$wc.Credentials =  New-Object System.Net.NetworkCredential($httpsUser,$httpsPass)
$wc.DownloadFile($Source, $target)

answered Apr 8, 2016 at 7:15

Martin Brandl

59.4k16 gold badges151 silver badges190 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Lennart Over a year ago

I tried the webclient method but it didn't work files still habe this crappy html code as content.. What bothers me is that it works for the first file but not for the following files - it seems to me as the server is not accepting any other webrequest after the first one (maybe I have to pass the session id or smth??)

Martin Brandl Over a year ago

hm, you could try to move the first two lines of my code outside the foreach loop so that you are using the same webclient for each file download.

Martin Brandl Over a year ago

hm. sorry then. Can you try to change the foreach loop order so that ijh is the first file to get downloaded?

Martin Brandl Over a year ago

you could also try to download the file using a browser where you disable your java script and grab the url of the button you mentioned in your comment within your question. You may have to invoke a request to that - I don't know what the server is doing there...

Kirill Pashkov · Accepted Answer · 2016-04-08 07:16:30Z

0

Try using webclient methods DownloadFile(src,dst). Should be something like this:

$httpsUser = 'XXX'
$httpsPass = 'YYY'

foreach ($instrument in 'ivv','ijh','ijr','iwm') {

$Source = 'https://***', `
$instrument, '-en_us.csv' -join ""

$Target = 'C:\User\', `
$instrument, '-en_us.csv' -join ""

$webclient = New-Object -TypeName Net.WebClient
$webclient.Encoding = [System.Text.Encoding]::UTF8
$webclient.UseDefaultCredentials = $true
$webclient.Proxy.Credentials = New-Object System.Net.NetworkCredential($httpsUser,$httpsPass)
$webclient.DownloadFile($Source,$Target)
}

answered Apr 8, 2016 at 7:16

Kirill Pashkov

3,2461 gold badge19 silver badges22 bronze badges

1 Comment

Lennart Over a year ago

webclient method produces the same result just tried it

Lennart · Accepted Answer · 2016-04-12 11:04:06Z

Thank you all for your answers. I figured out now that the site generates a security token while logging in. This token needs to be passed in every webrequest. I was not yet able to figure out how to do that with powershell but know that perl has a build in function (called $merch) for exactly this problem.

To solve my problem I had to automatize the IE :( (I know this is not the most sophisticated way but right now it is the quickest solution. If anyone is interested here is the code for that:

$ie = new-object -ComObject 'InternetExplorer.Application'
$requestUri = 'https://www.trololo.com'
$userIdFragment = "userName";
$passwordIdFragment = "password";
$buttonIdFragment = "submitLogin";
$ie.visible = 'false'


$ie.navigate($requestUri)
while($ie.Busy) { Start-Sleep -Milliseconds 100 }


$doc1 = $ie.Document
$doc1.getElementsByTagName("input") | % {
    if ($_.id -ne $null){
        if ($_.id.Contains($buttonIdFragment)) { $btn = $_ }
        if ($_.id.Contains($passwordIdFragment)) { $pwd = $_ }
        if ($_.id.Contains($userIdFragment)) { $user = $_ }
    }
}

$user.value = "XXXX"
$pwd.value = "YYYY
$btn.disabled = $false
$btn.click()
while($ie.Busy) { Start-Sleep -Milliseconds 5000 }
$ie.navigate($requestUri)
while($ie.Busy) { Start-Sleep -Milliseconds 200 }
$doc1 = $ie.Document


$link = $doc1.getElementsByTagName("a") | where-object {$_.href -match "Your String"}
$link.click()
Start-Sleep -Milliseconds 1000
$wshell = new-object -com wscript.shell
$wshell.appactivate("Internet Explorer")
$wshell.sendkeys("%s")

Collectives™ on Stack Overflow

Powershell filedownload from https

3 Answers 3

4 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related