On doubt it's a good idea to check the documentation.
For example for std::set: insert
No iterators or references are invalidated. If the insertion is successful, pointers and references to the element obtained while it is held in the node handle are invalidated, and pointers and references obtained to that element before it was extracted become valid. (since C++17)
Similar can be read for erase:
References and iterators to the erased elements are invalidated. Other references and iterators are not affected.
So from this (and the docs for the other functions) it's safe to conclude that references (and pointers to the elements) are OK to be used as long as you don't erase the element from the set.
If we check the same for unordered_set: insert:
If rehashing occurs due to the insertion, all iterators are invalidated. Otherwise iterators are not affected. References are not invalidated. Rehashing occurs only if the new number of elements is greater than max_load_factor()*bucket_count(). If the insertion is successful, pointers and references to the element obtained while it is held in the node handle are invalidated, and pointers and references obtained to that element before it was extracted become valid. (since C++17)
And the same for erase:
References and iterators to the erased elements are invalidated. Other iterators and references are not invalidated.
So also here we can conclude that taking the pointer to the element should be safe as long as the element is stored inside of the unordered_set.
I'll leave the unordered_map to you to validate, however as std::unordered_set is a special case of std::unordered_map without having a value, I expect it to behave similar.
EDIT: wether it's a good idea or not is hard to say. The containers come with guarantees and requirements. If these match your use case, it's worth trying it. From my experience I would recommend encapsulating these kind of things in some class with a specific name and expose the things you need to use. That way, if you decide to change the implementation, you know what you provide and what ain't a requirement.
Aobjects are owned by thestd::set<A>. You're going to run into nightmares of dangling pointers and use after frees. What you should do is instead use astd::set<std::shared_ptr<A>>and create theAobjects separately. Then you can pass aroundstd::shared_ptr<A>as you need.