TL;DR: I implemented a clumsy way to track a search request through various points in the network. Is there a better way?
I'm using the Elasticsearch SDK to perform searches. The search call traverses various points in the network, such as egress controllers and proxies. I'm seeing failures of search requests and wanted to see where they fail (i.e., how far does the request make it). I was studying the various logs from egress controllers, proxies, and Elasticsearch to trace the failed requests. I only see an entry like this:
/twitter,reddit,news/_search/template
It doesn't tell me what request this is. What I need is some type of trace ID that shows up in the logs. Only the URL and not the POST body is shown in those logs.
To be able to trace a call end-to-end, I decided to generate a trace ID on the client and add that trace ID as an index to the Elasticsearch request. The URL now looks something like this:
/twitter,reddit,news,2be8829d-8391-4014-b1c1-ffd8488ace0c/_search/template
I had to set the the ignoreUnavailable flag to true as per documentation.
With this, I can trace every request through all the interfaces and monitoring points. That's great! But I'm not sure if this is the right approach. I also don't know if it is a problem to always add a non-existing index to the search call. Could that become a problem?
Is there a better approach to this?
