dynamic path through json loaded in terraform

The Terraform language has no built-in functionality for this sort of arbitrary dynamic traversal.

As you noted in your question, it is possible in principle for a provider to offer this functionality. It wasn't clear to me whether you didn't want to use a provider at all or if you just didn't want to be the one to write it, and so just in case it was the latter I can at least offer a provider I already wrote and published which can potentially address this need, which is called apparentlymart/javascript and exposes a JavaScript interpreter into the Terraform language which you can use for arbitrary complex data manipulation:

terraform {
  required_providers {
    javascript = {
      source  = "apparentlymart/javascript"
      version = "0.0.1"
    }
  }
}

variable "traversal_path" {
  type = list(string)
}

data "javascript" "example" {
  source = <<-EOT
    for (var i = 0; i < path.length; i++) {
      data = data[path[i]]
    }
    data
  EOT

  vars = {
    data = jsondecode(file("${path.module}/file.json"))
    path = var.traversal_path
  }
}

output "result" {
  value = data.javascript.example.result
}

I can run this with different values of var.traversal_path to select different parts of the data structure in the JSON file:

$ terraform apply -var='traversal_path=["some1", "path1", "key1"]' -auto-approve
data.javascript.example: Reading...
data.javascript.example: Read complete after 0s

Changes to Outputs:
  + result = "value1"

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

result = "value1"

$ terraform apply -var='traversal_path=["some1", "path1", "key2"]' -auto-approve
data.javascript.example: Reading...
data.javascript.example: Read complete after 0s

Changes to Outputs:
  ~ result = "value1" -> "value2"

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

result = "value2"

$ terraform apply -var='traversal_path=["some1", "path1", "key3"]' -auto-approve
data.javascript.example: Reading...
data.javascript.example: Read complete after 0s

Changes to Outputs:
  - result = "value2" -> null

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

I included the final example above to be explicit that escaping into JavaScript for this problem means adopting some of JavaScript's behaviors rather than Terraform's, and JavaScript handles looking up a non-existing object property by returning undefined rather than returning an error as Terraform would, and the javascript data source translates that undefined into a Terraform null. If you want to treat that as an error as Terraform would then you'd need to write some logic into the loop to test whether data is defined after each step. You can use the JavaScript throw statement to raise an error from inside the given script.

Of course it's not ideal to embed one language inside another like this, but since the Terraform language is intended for relatively straightforward declarations rather than general computation I think it's reasonable to use an escape-hatch like this if the overall problem fits within the Terraform language but there is one small part of it that would benefit from the generality of a general-purpose language.

Bonus chatter: if you prefer a more functional style to the for loop I used above then you can alternatively make use of the copy of Underscore.js that's embedded inside the provider, using _.propertyOf to handle the traversal in a single statement:

  source = <<-EOT
    _.propertyOf(data)(path)
  EOT