In my purview, Object is undefinitive, and belongs in a binary format, not in a human-readable text.

I don’t see the relationship between these two things; there are plenty of good reasons to choose text versus binary: this is not one of them. How it appears is an implementation detail only.

any derived type of Object

The biggest challenge, then, is deciding how you are going to encode the type data. In many serializers, the answer is simply: you don’t. This avoids an entire family of problems with refactoring and version (renaming types, assembly-names, etc). You could limit yourself to [Serializable] types, but by the time you’ve done that you might as well just use the (terrible, IMO) BinaryFormatter. Of course, some key questions here might be: why do you want to write your own serializer? What is the feature(s) you are after that aren’t satisfied by the existing platform tools and 3rd-party / open-source tools?

Then you have a range of protocol / formatting considerations; mapping serialized data back into types is … non-trivial. If the type is going to require configuration (think: [XmlElement(...)] etc), you could reasonably limit yourself to types that have the basic configuration evidence. This is kinda what DataContractSerializer does, except it falls back into a more basic field-level serializer if that configuration data is missing.

I’m also a big advocate for the view that entity models make lousy serialization types, so it is quite reasonable to expect there to exist a DTO model that broadly reflects your serialization needs; and if that is the case, it is not unreasonable to expect that the DTO type might be annotated in some way (or at least advertised to the serializer in some way – perhaps at runtime if attributes aren’t an option).