Why static nested class?

Assuming that in your code snippet you are using a non-static inner class instead of a static nested class like this: private class Node, in that case, you will be trying to instantiate an Array which is not possible, we can't instantiate an Array in a generic class, because generics doesn't have any information regarding their type at runtime, while arrays creation expression specifies the element type.

So, the reason why using a Static Nested Class compiled, is that such classes are considered as a "top-level" class (in terms of behavior):

A static nested class interacts with the instance members of its outer class (and other classes) just like any other top-level class. In effect, a static nested class is behaviorally a top-level class that has been nested in another top-level class for packaging convenience.

Now, let's take all of this into consideration, and come back to the exact error displayed by the compiler:

Cannot create a generic array of TrieST<Value>.Node

That means that the type of the array you want to create is TrieST<Value>.Node whose runtime's type is not known, thus different types may be inserted into the next array. More clear and well explained examples could be found in Cannot Create Arrays of Parameterized Types

Whereas, a static Nested class is not behaving as an inner class of TrieST<Value> , thus instiating an array inside Node will not be illegal as it's not of the type TrieST<Value>.Node, its of the type Node (like if it's a top-level class) .

Because with static you create: Node[] next = new Node[R] and with non-static inner-class you create a Node that is associated with an instance of the outer-class, which has a generic type. And creation of generic arrays is forbidden.

But lets back up a few steps: the way to instantiate an inner-class (non-static) is as follows (example):

class TrieST<V> {
    private static final int R = 256;        

    private Node root;      // root of trie
    private int N;          // number of keys in trie
    private TrieST<String> inst = new TrieST<String>(); // must create an instance of the outer class first

    // R-way trie node
    private class Node {
        private Object val;
        private TrieST<String>.Node next =  inst.new Node(); //must use an instance of the outer class to instantiate an object of the inner class
    }

    public TrieST() {
    }
}

Now, if we'll try to change the implementation above from an instance of the inner class to an array, we'll get generic array creation because it's prohibited to create arrays with generic type due to the covariance nature of arrays (Shape[] is super of Triangle[]) which doesn't work well with the invariant nature of generics (List<Object> is not super of List<String>). In "Effective Java" Bloch provides a more detailed explanation if you want to dig in.

If you insist on using an inner-class, you can work around this restriction by using Array.newInstance() which can create array of a type known only at runtime as follows:

private Node[] next = (Node[]) Array.newInstance(Node.class, R);