C# HttpClient.SendAsync throw "An error occurred while sending the request" exception when testing some URLs

Question

I am developing a C# console application for testing whether a URL is valid or not. It works well for most of URLs and can get response with HTTP Status Code from the target website. But when testing some other URLs, the application throws an exception saying "An error occurred while sending the request" when running HttpClient.SendAsync method. So I can't get any response or HTTP Status Code although this URL actually works in the browser. I am desperate to find out how to handle this case. If the URL doesn't work or the server rejects my request, it should at least give me the corresponding HTTP Status code.

Here is the simplified code of my test application:

using System;
using System.Net.Http;
using System.Threading.Tasks;

namespace TestUrl
{
    class Program
    {
        static void Main(string[] args)
        {
           // var urlTester = new UrlTester("http://www.sitename.com/wordpress"); // works well and get 404
           // var urlTester = new UrlTester("http://www.fc.edu/"); // Throw exception and the URL doesn't work
           var urlTester = new UrlTester("http://www.ntu.edu.tw/english/"); // Throw exception and the URL works actually

            Console.WriteLine("Test is started");

            Task.WhenAll(urlTester.RunTestAsync());
            
            Console.WriteLine("Test is stoped");
            Console.ReadKey();
        }


        public class UrlTester
        {
            private HttpClient _httpClient;
            private string _url;

            public UrlTester(string url)
            {
                _httpClient = new HttpClient();
                _url = url;
            }

            public async Task RunTestAsync()
            {
                var httpRequestMsg = new HttpRequestMessage(HttpMethod.Head, _url);

                try
                {
                    using (var response = await _httpClient.SendAsync(httpRequestMsg, HttpCompletionOption.ResponseHeadersRead))
                    {
                        Console.WriteLine("Response: {0}", response.StatusCode);
                    }
                }
                catch (Exception e) 
                { 
                }
            }
        }

    }
}

Tamir Vered · Accepted Answer · 2015-11-08 09:35:57Z

41

If you look at the InnerException you will see that:

"The remote name could not be resolved: 'www.fc.edu'"

This URL does not work on my browser either.

In order to get an HTTP response you need the client to be able to communicate with the server (even in order to get error 404) and in your case the error occurred at the DNS level.

Some browsers have auto-completion for this kind of cases where if a specific URL is not found, the browser retries with a different suffix/prefix, for example:

try "x"
if didn't work, try "www." + x
if this didn't work try "www." + x + ".com"
if this didn't work try "www." + x + ".net"
if this didn't work try "www." + x + "." + currentRegionSuffix.

But note that you can change your code from:

catch (Exception e)
{

}

To:

catch (HttpRequestException e)
{
    Console.WriteLine(e.InnerException.Message);
}

And you will be able to see what causes your error.

Also, You should never want to catch the generic Exception unless the thrower has thrown the generic Exception, and even than, never catch and do nothing with the exception, at least log it.

Notice than since you wait only for that one task you can use:

urlTester.RunTestAsync().Wait();

Instead of:

Task.WhenAll(urlTester.RunTestAsync());

Task.WhenAll creates a new Task when the given Tasks are completed. in your case you need Task.WaitAll or Task.WhenAll(...).Wait().

edited Nov 8, 2015 at 9:35

answered Oct 4, 2015 at 9:54

Tamir Vered

10.3k5 gold badges48 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Yuval Itzchakov Over a year ago

Generally, Task.WhenAll will return an awaitable which needs to be awaited. If he wants the blocking version, Task.WaitAll is what he should use.

codigube Over a year ago

Thank @TamirVered and Yuval Itzchakov help me with finding out the details of exception and improving my code. Now the the question becomes how can I know whether this URL works or not when this DNS level issue happens. Because in many cases, the URL actually works. Check out the tested URL part, I just added this case.

Tamir Vered Over a year ago

You can read the InnerException again to see the reason for the error and the InternalStatus to see the status of your connection after it.

Squazz Over a year ago

"This URL does not work on my browser either." - This one helped me, I had set the internal DNS op wrongly ;)

Huy · Accepted Answer · 2021-09-13 18:23:28Z

16

I solved the problem with:

System.Net.ServicePointManager.SecurityProtocol =
            SecurityProtocolType.Tls12 |
            SecurityProtocolType.Tls11 |
            SecurityProtocolType.Tls;

answered Sep 13, 2021 at 18:23

Huy

5265 silver badges15 bronze badges

1 Comment

Manjunath Patelappa Over a year ago

from security perspective this is wrong to allow lower TLS version or hardcoding. learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/…

Collectives™ on Stack Overflow

C# HttpClient.SendAsync throw "An error occurred while sending the request" exception when testing some URLs

2 Answers 2

4 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related