Here’s my opinion on the matter:

1. consider whether both variants can actually be used in your case. reference_wrapper is, by design, not default-constructible. That means you will not be able, for example, to call container.resize() when using the reference wrapper. A shared_ptr, on the other hand, is default-constructed to an invalid/NULL state. So, for some use cases, you can’t use a reference_wrapper. To give some examples:

   int i = 42;
   std::reference_wrapper<int> iref(i);             // OK
   std::reference_wrapper<int> uninitialized_iref;  // Not OK, but OK with shared_ptr
   
   iref = 23;                     // Not OK, but OK on a "normal" reference
   int& iref2 = iref; iref2 = 32; // OK
   static_cast<int&>(iref) = 23;  // OK
   iref.get() = 23;               // OK
   
   std::vector<std::reference_wrapper<int> > irefs; // OK
   irefs.resize(10);    // Not OK!
   irefs.resize(10, i); // OK
   

2. The above examples already hint at something else: Usability and readability. Depending on your use case, you have to decide whether you prefer pointer-style notation (shared_ptr) or more-or-less-reference-style notation (reference_wrapper) and whether that makes the code more maintainable. Personally I value this aspect quite highly.

3. As pointed out in a comment to your question, the object lifetime also plays a role. When the container’s lifetime is intended to exceed the lifetime of any of the referenced objects, you have to use shared_ptr (or maybe find a different system design). With references you need to be certain about the respective scopes. I also suspect that in your specific case, you should actually care about references becoming invalid — if that can happen, you’d be in trouble, and it would usually be difficult to debug.

4. Storing shared_ptr elements of course implies that the values you want to refer to are already stored in shared pointers. You can not create a shared pointer referencing an element created by some other means. So, if you need to refer to existing, non-shared elements, you can not use shared_ptr and would have to go with reference_wrapper or plain old pointers.

5. As far as performance goes, I would not expect much of a difference. reference_wrapper usually uses a pointer internally, and the overhead for a shared_ptr is usually negligible.

All in all, I would base my decision around (a) my concrete use case, and (b) usability/maintainability/readability of the code using the container.

It’s mostly a question of lifetime and ownership. A container using reference_wrapper doesn’t own its objects or manage their lifetime, a container using shared_ptr does.

If your container shouldn’t own its objects, then use reference_wrapper.

If it should own its objects, then I’d take a third option. Boost.Pointer Container has template classes such as boost::ptr_deque that store polymorphic types and manage ownership themselves, without the overhead of shared_ptr or the awkward syntax of having to dereference shared_ptr for each element.

The question is one of ownership

You make no mention of who owns the objects going into the container. Since the shared_ptr is an option, there is probably some form of shared ownership and dynamic storage allocation. A clear definition of who owns the objects and how they are observed (i.e. who can observe them) will frame much of the implementation.

In my specific case, the class containing the container actually wouldn’t care if a reference somehow became invalid.

The container is playing an observer role.

However there might be a more generalised answer and a ‘rule of thumb’, when one should use one approach over the other.

Given the semantics of the the reference_wrapper there is a reasonable expectation that the object being referred to is valid whilst the reference_wrapper is valid. In your case this may not apply, but in general this is true and any maintenance done on the code at a later date may well assume this.

Rule of thumb

Allow the lifetime of the shared ownership objects to be managed with a std::shared_ptr and use the “observer” std::weak_ptr in the container.

If the lifetime of the objects is not shared and somehow tied to the lifetime of container itself (via automatic storage duration), then the reference_wrapper is viable (in a similar way it could even be a raw pointer).