Multiple HTTP requests trigger HTTP Client timeout

Personally, I think attempting to issue 500 concurrent requests is always going to be error prone. You mention that you're doing it asynchronously, but in reality there's not a whole lot of asynchrony in your code as you fire-up 500 "hot" tasks then wait for them all to finish.

I would use a semaphore to control how many requests can be made at once. You may have to play with the numbers to find the sweet spot.

The following code works well in LINQPad (although bing quickly notices the odd number of requests and starts adding a CAPTCHA to the page):

// using System.Threading;
async Task Main()
{
    var httpClient = new HttpClient();
    var urls = Enumerable.Range(1, 500).Select(e => "https://www.bing.com/").ToList();
    
    // 10 concurrent requests - tweak this number
    var semaphore = new SemaphoreSlim(10, 10);
    
    var tasks = urls.Select(u => MakeRequest(u, semaphore, httpClient));
        
    var allResponses = await Task.WhenAll(tasks);
    
    // Do something with allResponses
}

private async Task<string> MakeRequest(string url, SemaphoreSlim semaphore, HttpClient httpClient)
{
    try
    {
        await semaphore.WaitAsync();
        var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url));
        var response = await httpClient.SendAsync(request);
        
        // Add an optional delay for further throttling:
        //await Task.Delay(TimeSpan.FromMilliseconds(100));
        
        return await response.Content.ReadAsStringAsync();
    }
    finally
    {
        semaphore.Release();
    }
}

There are a few problem of HttpClient.

For example:

1. HttpClient is disposable: Using HttpClient with the using statement is not the best choice because even when you dispose HttpClient object, the underlying socket is not immediately released and can cause a serious issue named ‘sockets exhaustion’. For more information about this issue, You're using httpclient wrong and it is destabilizing your software

2. HttpClient is intended to be instantiated once and reused throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. That issue will result in SocketException errors. Possible approaches to solve that problem are based on the creation of the HttpClient object as singleton or static.

3. HttpClient that you can have when you use it as singleton or static object. In this case, a singleton or static HttpClient doesn't respect DNS changes. For more information :Singleton HttpClient doesn't respect DNS changes

To address those mentioned issues and make the management of HttpClient instances easier, .NET Core 2.1 introduced a new HttpClientFactory.

What is HttpClientFactory:

1. Provide a central location for naming and configuring logical HttpClient objects. For example, you may configure a client (Service Agent) that's pre-configured to access a specific microservice.
2. Codify the concept of outgoing middleware via delegating handlers in HttpClient and implementing Polly-based middleware to take advantage of Polly’s policies for resiliency.
3. HttpClient already has the concept of delegating handlers that could be linked together for outgoing HTTP requests. You register HTTP clients into the factory and you can use a Polly handler to use Polly policies for Retry, CircuitBreakers, and so on.
4. Manage the lifetime of HttpClientMessageHandlers to avoid the mentioned problems/issues that can occur when managing HttpClient lifetimes yourself.

For more details please visit this link: Use HttpClientFactory to implement resilient HTTP requests

You can also use RestSharp library for send any requests. For more http://restsharp.org/

Referenced resources:

1. https://josefottosson.se/you-are-probably-still-using-httpclient-wrong-and-it-is-destabilizing-your-software/
2. https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
3. https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests
4. https://github.com/dotnet/runtime/issues/18348

A few things I would be checking:

• Rather than use a new HttpClient each time, use an HttpClientFactory to avoid the overhead of HttpClient construction, and the exhaustion of ports.

• Are you sure that you're not being rate-limited by Bing? Try targeting a remote server under your control.

• Consider limiting the concurrency of your requests with a SemaphoreSlim or a parallelism library.

Try to increase default connections limit:

ServicePointManager.UseNagleAlgorithm = true;
ServicePointManager.Expect100Continue = true;
ServicePointManager.DefaultConnectionLimit = <number>;

Seems you are sending request to bing search engine right? If I'm right you have another problem. Bing search may be doesn't allow bot requests.hence bing block requests.check further about that.You can try with some time delays instead of async requests.your app seems working fine.You can verify my theory by sending requests to another service or API.