How to formulate curl -XHEAD request in java

Question

I am using elasticsearch Java API and trying to make a curl call in order to find out whether a document exists in my index or not. This is how it is done in the command line. As far as I can tell from the posts here, I should either use HttpURLConnection java class or apache httpclient to send the curl request in java. My request should be like :

curl -i -XHEAD http://localhost:9200/indexName/mappingName/docID.

There are actually many questions on how to send curl requests via java, but the answers are not that explanatory - thus I am not sure how to configure the request parameters for the curl head request. So far, I have reproduced this answer from Ashay and it doesn't work.

Does anybody send curl calls in elasticsearch's java API and can explain how to do it?

Here is my code and the error I get is "java.net.MalformedURLException: no protocol"

import org.apache.commons.codec.binary.Base64;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;

String encodedURL = URLEncoder.encode(new StringBuilder()
    .append("http://").append(serverName).append(":9200/") // elasticsearch port
    .append(indexName).append("/").append(mappingName).append("/")
    .append(url).toString(), "UTF-8"); // docID is a url
System.out.print("encodedURL : " + encodedURL + "\n");

URL url = new URL(new StringBuilder().append(encodedURL).toString());
System.out.print("url "+ url.toString() + "\n");

HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("X-Requested-With", "Curl");
connection.setRequestMethod("HEAD");

String userpass = new StringBuilder().append(username).append(":").append(password).toString();
String basicAuth = new StringBuilder().append("Basic ").append(new String(new Base64().encode(userpass.getBytes()))).toString();
connection.setRequestProperty("Authorization", basicAuth);
String inputLine;
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));

P.S. the document ids of this index are urls and that is why I need to encode them. On the other hand, I am not sure whether I should encode the full http request or not.

SubOptimal · Accepted Answer · 2016-05-07 09:15:01Z

0

Following snippet could be a starting point.

String serverName = "localhost";
String indexName = "index_name";
String mappingName = "mapping_name";
String docId = "FooBarId";

String username = "JohnDoe";
String password = "secret";

String requestURL = String.format("http://%s:9200/%s/%s/%s",
        serverName,
        indexName,
        mappingName,
        docId
);
System.out.println("requestURL: " + requestURL);

URL url = new URL(requestURL);
System.out.println("URL: " + url);

HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("X-Requested-With", "Curl");
connection.setRequestMethod("HEAD");

String credentials = username + ":" + password;

Base64.Encoder encoder = Base64.getEncoder();
String basicAuth = "Basic " + encoder.encodeToString(credentials.getBytes());

connection.setRequestProperty("Authorization", basicAuth);

connection.getHeaderFields()
        .entrySet()
        .forEach((Entry<String, List<String>> t) -> {
            System.out.printf("%-20s : %s%n", t.getKey(), t.getValue());
        });

Using requestURL = "http://localhost:9200"; with a default elasticsearch installation would return

requestURL: http://localhost:9200
URL: http://localhost:9200
null                 : [HTTP/1.1 200 OK]
Content-Length       : [0]
Content-Type         : [text/plain; charset=UTF-8]

add Maybe you can try something similar to following steps. Amend them to your needs. Maybe you can skip the first step.

index something

curl -XPUT "http://localhost:9200/books/book/1" -d'
{
    "title": "The Hitchhikers Guide to the Galaxy",
    "author": "Douglas Adams",
    "year": 1978
}'

query it from commandline

curl -X GET http://localhost:9200/books/book/1

output

{"_index":"books","_type":"book","_id":"1","_version":1,"found":true,"_source":
{
    "title": "The Hitchhikers Guide to the Galaxy",
    "author": "Douglas Adams",
    "year": 1978
}}

query using the above Java snippet

String serverName = "localhost";
String indexName = "books";
String mappingName = "book";
String docId = "1";
String requestURL = String.format("http://%s:9200/%s/%s/%s",
        serverName,
        indexName,
        mappingName,
        docId
);
System.out.println("requestURL: " + requestURL);
URL url = new URL(requestURL);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.getHeaderFields()
        .entrySet()
        .forEach((Entry<String, List<String>> t) -> {
            System.out.printf("%-20s : %s%n", t.getKey(), t.getValue());
        });
try (InputStream inputStream = connection.getInputStream()) {
    for (int i = inputStream.read(); i > -1; i = inputStream.read()) {
        System.out.print((char) i);
    }
}

output

requestURL: http://localhost:9200/books/book/1
null                 : [HTTP/1.1 200 OK]
Content-Length       : [184]
Content-Type         : [application/json; charset=UTF-8]
{"_index":"books","_type":"book","_id":"1","_version":1,"found":true,"_source":
{
    "title": "The Hitchhikers Guide to the Galaxy",
    "author": "Douglas Adams",
    "year": 1978
}}

The example use a default elasticsearch installation.

Depending on what you really want to achieve. You might be better use the elasticsearch TransportClient.

import java.net.InetAddress;
import org.elasticsearch.action.get.GetRequestBuilder;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.transport.InetSocketTransportAddress;

public class GetDemo {

    public static void main(String[] args) throws Exception {
        InetAddress hostAddr = InetAddress.getByName("localhost");
        InetSocketTransportAddress socketAddr =
                new InetSocketTransportAddress(hostAddr, 9300);
        try (Client client = TransportClient.builder().build()
                .addTransportAddress(socketAddr)) {
            GetRequestBuilder request = client.prepareGet("books", "book", "1");
            GetResponse response = request.execute().actionGet();
            response.getSource()
                    .forEach((k, v) -> System.out.printf("%-6s: %s%n", k, v));
        }
    }
}

output

...
year  : 1978
author: Douglas Adams
title : The Hitchhikers Guide to the Galaxy

edited May 7, 2016 at 9:15

answered May 2, 2016 at 15:19

SubOptimal

23.1k3 gold badges57 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

KLaz Over a year ago

Thanks for the reply. Your example code yields a "null=[HTTP/1.1 400 Bad Request] Content-Length=[145] Content-Type=[text/plain; charset=UTF-8]" error for all docIds. The only changes I made were to replace localhost with our server name and also use "Base64.encodeBase64(credentials.getBytes());" because Base64.Encoder is private

SubOptimal Over a year ago

@KonstantinaLazaridou Base64.Encoder is a public static class in Java 8. Have you checked your generated urland the basicAuth? As HTTP 400 is for The request could not be understood by the server due to malformed syntax.

KLaz Over a year ago

yes, the url seems fine and the user info are also correct. Is it possible that the server uses different encryption system than Base64? I also commended the authorization property (in case it was not necessary), but still the same error. I also don't know whether .setRequestProperty("X-Requested-With", "Curl"); is syntactically correct.

SubOptimal Over a year ago

@Try to break down the problem. 1) Enter the url you want to access in a browser. a) Is the url valid. b) Do you get a request pop-up for user/password. 2) If the url is working assign it as a fixed string to requestURL in the above example. Another question is your curl example working without an user/password? If so. Why do you believe you need it in the java code?

Collectives™ on Stack Overflow

How to formulate curl -XHEAD request in java

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related