What is the proper method to parse and send HTTP requests using sockets in Java?

Question

I'm creating a basic local proxy server, the goal is to accept http and https traffic from my web browser, parse it for information, send and receive the requests to the proper host, and then return it to the web browser.

I currently have an open socket to my web browser. I am receiving both http and https requests from the browser like so:

HTTP:

GET http://example.com/ HTTP/1.1 
Host: example.com User-Agent:
Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5 
Accept-Encoding: gzip, deflate
Connection: keep-alive 
Upgrade-Insecure-Requests: 1

HTTPS:

CONNECT example.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: example.com:443

I open a socket to the "Host:" from the above with the following code:

public void sendRequest() throws IOException{
        Socket socket = new Socket(host, port);
        //socket.getInputStream.read();
        BufferedWriter out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream(), "UTF8"));
        BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
        for(int i = 0; i < lines.size(); i++){
            out.write(lines.get(i) + "\r\n");
        }
        out.flush();
        outputReturn(in);
    }

And I receive the reply like so:

public void outputReturn(BufferedReader in){
        try{
            System.out.println("\n * Response");
            String line;
            while ((line = in.readLine()) != null) {
                System.out.println(line);
            }
        }
        catch (IOException i){
            System.out.println(i);
        }
    }

The replies come back as so:

HTTP:

* Response
HTTP/1.1 200 OK
Content-Encoding: gzip
Accept-Ranges: bytes
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Wed, 10 Apr 2019 22:53:28 GMT
Etag: "1541025663+gzip"
Expires: Wed, 17 Apr 2019 22:53:28 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (ord/4C92)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 606

;�R�TA��0
         ��W�ri]��S�V @���1k��Z��$�6���q۽���@+���l�I�I��s�PzUe���Bf
                                                                   �'��+�>���+�OF   �I4h��^@^
�ЧA�p@�M���u����������*
<�|ԅߎP���P�-�6�O��$}�Jl)ǰ_,�4yU�rQazw�r���t
                                           .�s���3�
                                                   z�_������2�Mel
                                                                 ϋ5����%�t
                                                                          뫪R���t3

��:�|�Q��]���
             V-z�|�Y3*���rKp�5th��"��C���NH����v��OOyޣ�xs�����V��$��X�6�BR�b�C��PqE���K�<�  �G�כ7����E(17Vx2�US��
%   x��)�d�����e��O&�4/䤘���~��Oi�s�X�dW�7��#�u�"��y\$]j<�L�r�˻'�ɪ�Vg?Kr {=��΋]E��^x;�ƱX
                                                                                            TU��]�[�{��s+�e����9�g���]����H�4���#�KA��'�Z�����*r�
�$�G�   ��4�n�8���㊄+c���E�hA��X���������L��RIt�[4\����

HTTPS:

CONNECT getpocket.cdn.mozilla.net:443 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: getpocket.cdn.mozilla.net:443


 * Response
java.net.SocketException: Connection reset

Questions:

Why do I receive what seems like binary from the HTTP request?

Why so I receive nothing from my HTTPS request?

What SHOULD I be doing instead?

Thanks in advance.

You receive binary because you asked for it: Accept-Encoding: gzip, deflate, and the server told you you got it: Content-Encoding: gzip. — user207421
– user207421, Commented Apr 11, 2019 at 1:43
If you want to implement HTTP correctly, you should read the specs for it. Yes they are long and complicated -- and a correct proxy is long and complicated as a result. Shortcuts may seem to work, for a while, sometimes, but will eventually bite you. — dave_thompson_085
– dave_thompson_085, Commented Apr 11, 2019 at 6:48

Benjamin Urquhart · Accepted Answer · 2019-04-10 23:18:10Z

2

For your HTTP request, the Content-Encoding is gzip. The binary is the gzip-compressed data.

For your HTTPS request, you're not making an SSL/TLS handshake, so the server drops the connection.

For HTTP, I don't think you need to do anything, the browser should handle it for you. There's no feasible way to proxy an HTTPS/SSL/TLS using the method you described.

answered Apr 10, 2019 at 23:18

Benjamin Urquhart

1,64915 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Josh B Over a year ago

Thanks for the reply! Should I just send the HTTP data back through the socket I received the HTTP request from the browser?

Josh B Over a year ago

Also for https, are there any known methods?

Benjamin Urquhart Over a year ago

@JoshB you can send the HTTP data as-is to the browser, yes. Short of using your browser's private key (have fun with that) there's no way to do the same for HTTPS.

Josh B Over a year ago

So a proxy for https is unfeasible?

Benjamin Urquhart Over a year ago

@JoshB it really depends on what you want to accomplish. If you simply want to forward data without header parsing, you can definitely simply pass the HTTPS data back and forth. If you want to parse that data, however, you need your browser's private key.

|

Collectives™ on Stack Overflow

What is the proper method to parse and send HTTP requests using sockets in Java?

1 Answer 1

11 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Related