Const
There are choices to be made regarding use of const with caches. We could consider lookup either to be a const operation (meaning we'd need to use mutable specifier for the LRU list) or to be a mutating operation, as we have done here. There are good arguments for both approaches, but it would be worth having a comment that shows why this choice was preferred here.
Strange assertion
Making the collection implement a Range (perhaps in LRU order) might be worthwhile, too. For a variant using plain std::map, we might consider two Range views - one in LRU order and one in the map's preferred order.