GETting a URL with an url-encoded slash

Question

I want to send a HTTP GET to http://example.com/%2F. My first guess would be something like this:

using (WebClient webClient = new WebClient())
{
  webClient.DownloadData("http://example.com/%2F");
}

Unfortunately, I can see that what is actually sent on the wire is:

GET // HTTP/1.1
Host: example.com
Connection: Keep-Alive

So http://example.com/%2F gets translated into http://example.com// before transmitting it.

Is there a way to actually send this GET-request?

The OCSP-protocol mandates sending the url-encoding of a base-64-encoding when using OCSP over HTTP/GET, so it is necessary to send an actual %2F rather than an '/' to be compliant.

EDIT:

Here is the relevant part of the OCSP protocol standard (RFC 2560 Appendix A.1.1):

An OCSP request using the GET method is constructed as follows:

GET {url}/{url-encoding of base-64 encoding of the DER encoding of the OCSPRequest}

I am very open to other readings of this, but I cannot see what else could be meant.

Sounds like a bug in the OCSP protocol to me (or, alternatively, a misinterpretation of it). — Julian Reschke
– Julian Reschke, Commented Apr 23, 2009 at 11:38
@knocte: It has been reported and fixed. See Bradley Gaingers answer. — Rasmus Faber
– Rasmus Faber, Commented Oct 31, 2012 at 19:59
actually, either the link is broken, or the bug report is private :( — knocte
– knocte, Commented Oct 31, 2012 at 20:43

Rasmus Faber · Accepted Answer · 2012-06-04 10:10:10Z

53

This is a terrible hack, bound to be incompatible with future versions of the framework and so on.

But it works!

(on my machine...)

Uri uri = new Uri("http://example.com/%2F");
ForceCanonicalPathAndQuery(uri);
using (WebClient webClient = new WebClient())
{
  webClient.DownloadData(uri);
}

void ForceCanonicalPathAndQuery(Uri uri){
  string paq = uri.PathAndQuery; // need to access PathAndQuery
  FieldInfo flagsFieldInfo = typeof(Uri).GetField("m_Flags", BindingFlags.Instance | BindingFlags.NonPublic);
  ulong flags = (ulong) flagsFieldInfo.GetValue(uri);
  flags &= ~((ulong) 0x30); // Flags.PathNotCanonical|Flags.QueryNotCanonical
  flagsFieldInfo.SetValue(uri, flags);
}

edited Jun 4, 2012 at 10:10

answered Apr 24, 2009 at 8:15

Rasmus Faber

49.8k25 gold badges149 silver badges193 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

David Over a year ago

+1. Perfect! I needed this to get the Google Webmaster Tools API to work.

Patrick Klug Over a year ago

works indeed. I wonder what the reason is for them not allowing the 'dontEscape' parameter?

Matt Sherman Over a year ago

For the love of Atwood this answer saved me hours. Thanks!

Bradley Grainger Over a year ago

@user1473484: Yes, you can work around this by changing your app.config; see my answer to a similar question here: stackoverflow.com/a/10415482

Michael Brown Over a year ago

To get this to work under .Net Standard (and possibly recent versions of .Net Framework) you will need to add flags &= ~((ulong)0xC30); and change m_Flags to _flags

|

Community · Accepted Answer · 2021-10-07 05:46:31Z

31

By default, the Uri class will not allow an escaped / character (%2f) in a URI (even though this appears to be legal in my reading of RFC 3986).

Uri uri = new Uri("http://example.com/%2F");
Console.WriteLine(uri.AbsoluteUri); // prints: http://example.com//

(Note: don't use Uri.ToString to print URIs.)

According to the bug report for this issue on Microsoft Connect, this behaviour is by design, but you can work around it by adding the following to your app.config or web.config file:

<uri>
  <schemeSettings>
    <add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes" />
  </schemeSettings>
</uri>

(Reposted from https://stackoverflow.com/a/10415482 because this is the "official" way to avoid this bug without using reflection to modify private fields.)

Edit: The Connect bug report is no longer visible, but the documentation for <schemeSettings> recommends this approach to allow escaped / characters in URIs. Note (as per that article) that there may be security implications for components that don't handle escaped slashes correctly.

edited Oct 7, 2021 at 5:46

CommunityBot

11 silver badge

answered Aug 29, 2012 at 2:32

Bradley Grainger

28.3k4 gold badges95 silver badges117 bronze badges

1 Comment

r590 Over a year ago

Be sure to use name="https" if the URL you are calling is HTTPS.

Community · Accepted Answer · 2017-05-23 12:10:05Z

13

Update on this: It looks like the default behavior of the Uri class was actually changed in .NET 4.5, and you can now use escaped slashes and they will not be touched.

I ran the following code in .NET 3.5, .NET 4.0, .NET 4.5/4.5.1

static void Main(string[] args)
{
    var uri = new Uri("http://www.yahooo.com/%2F");
    var client = new WebClient();
    client.DownloadString(uri);
}

In .NET 3.5/4.0 the trace shows that the %2F was in fact unescaped as expected.

Fiddler trace

However, In .NET 4.5/4.5.1 you can see the %2F was not unescaped (notice the GET /%2F)

Fiddler trace

You can even use ToString() now on the Uri and you'll get the same result.

So in conclusion, it appears if you are using .NET >= .NET 4.5 then things will behave as they should inline with the RFC.

I just did an exploration of trying to get the same approach working on Mono. I posted my question on the approach here: Getting a Uri with escaped slashes on mono

edited May 23, 2017 at 12:10

CommunityBot

11 silver badge

answered Dec 22, 2013 at 20:15

Glenn Block

8,4551 gold badge35 silver badges34 bronze badges

1 Comment

aKzenT Over a year ago

I would like to add, that the change only takes effect when changing the "Target Framework" in Visual Studio to .NET 4.5. Just having .NET 4.5 installed on the machine will not change the behaviour.

Michael Brown · Accepted Answer · 2019-10-13 05:18:59Z

As mentioned in my comment on the answer posted by Ramus, the following is required for .Net Standard (and possibly later versions of .Net Framework) to get this hack working:

Uri uri = new Uri("http://example.com/%2F");
ForceCanonicalPathAndQuery(uri);
using (WebClient webClient = new WebClient())
{
  webClient.DownloadData(uri);
}

void ForceCanonicalPathAndQuery(Uri uri){
  string paq = uri.PathAndQuery; // need to access PathAndQuery
  FieldInfo flagsFieldInfo = typeof(Uri).GetField("_flags", BindingFlags.Instance | BindingFlags.NonPublic);
  ulong flags = (ulong) flagsFieldInfo.GetValue(uri);
  flags &= ~((ulong) 0xC30); // Flags.PathNotCanonical|Flags.QueryNotCanonical
  flagsFieldInfo.SetValue(uri, flags);
}

dr. evil · Accepted Answer · 2009-04-23 10:55:52Z

-7

Double encode it : %252F

But also if you use HttpWebRequest you can actually tell not to encode the URL, either way it should work.

Also If WebClient accepts URI then you can create a new URI and you can set it to not encode.

answered Apr 23, 2009 at 10:55

dr. evil

27.4k37 gold badges136 silver badges203 bronze badges

2 Comments

Rasmus Faber Over a year ago

If I try to get example.com/%252F, it actually sends GET /%252F, so that does not work. The URI-constructor with dontEscape is deprecated since 2.0 and according to the documentation, the dontEscape-parameter is ignored. Was that what you meant about using HttpWebRequest?

dr. evil Over a year ago

Check if the problem is in URI constructor? or the actual sending process? this can help to diagnose to exact problem.

Collectives™ on Stack Overflow

GETting a URL with an url-encoded slash

5 Answers 5

10 Comments

1 Comment

1 Comment

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

10 Comments

1 Comment

1 Comment

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related