The methods I’ve seen most are 2 and 3.

The user supplied buffer is actually quite simple to use:

char buffer[128];
mytype_to_string(mt, buffer, 128);

Though most implementations will return the amount of buffer used.

Option 2 will be slower and is dangerous when using dynamically linked libraries where they may use different runtimes (and different heaps). So you can’t free what has been malloced in another library. This then requires a free_string(char*) function to deal with it.

I’d echo @ratchet_freak in most cases (maybe with a small tweak for sizeof buffer over 128) but I wanna jump in here with a weirdo response. How about being weird? Why not, besides the problems of getting weird looks from our colleagues and having to be extra persuasive? And I offer this:

// Note allocator parameter.
char* mytype_to_string(allocator* alloc, const mytype_t* t)
{
    char* buf = allocate(alloc, however_much_you_need);
    // fill out buf based on 't' contents
    return buf;
}

And example usage:

void func(my_type a, my_type b)
{
    allocator alloc = allocator_new();
    const char* str1 = mytype_to_string(&alloc, &a);
    if (!str1)
        goto oom;
    const char* str2 = mytype_to_string(&alloc, &b);
    if (!str2)
        goto oom
    // do something with str1 and str2
    goto finish;
oom:
    errno = ENOMEM;
finish:
    // Frees all memory allocated through `alloc`.
    allocator_purge(&alloc);
}

If you did it this way, you can make your allocator very efficient (more efficient than malloc both in terms of allocation/deallocation costs and also improved locality of reference for memory access). It can be an arena allocator that just involves incrementing a pointer in common cases for allocation requests and pool memory in a sequential fashion from large contiguous blocks (with the first block not even requiring a heap allocation — it can be allocated on the stack). It simplifies error handling. Also, and this might be the most debatable but I think it’s practically quite obvious in terms of how it makes it clear to the caller that it’s going to allocate memory to the allocator you pass in, requiring explicit freeing (allocator_purge in this case) without having to document such behavior in every single possible function if you use this style consistently. The acceptance of the allocator parameter makes it hopefully quite obvious.

I don’t know. I get counter-arguments here like implementing the simplest-possible arena allocator possible (just use maximum alignment for all requests) and dealing with it is too much work. My blunt thought is like, what are we, Python programmers? Might as well use Python if so. Please use Python if these details don’t matter. I’m serious. I have had many C programming colleagues who would very likely write not only more correct code but possibly even more efficient with Python since they ignore things like locality of reference while stumbling over bugs they create left and right. I don’t see what there is that’s so scary about some simple arena allocator here if we are C programmers concerned with things like data locality and optimal instruction selection, and this is arguably much less to think about than at least the types of interfaces that require callers to explicitly free every single individual thing that the interface can return. It offers bulk deallocation over individual deallocation of a sort that’s more error-prone. A proper C programmer challenges loopy calls to malloc as I see it, especially when it appears as a hotspot in their profiler. From my standpoint, there has to be more “oomph” to a rationale for still being a C programmer in 2020, and we can’t shy away from things like memory allocators anymore.

And this doesn’t have the edge cases of allocating a fixed-sized buffer where we allocate, say, 256 bytes and the resulting string is larger. Even if our functions avoid buffer overruns (like with sprintf_s), there’s more code to properly recover from such errors that’s required which we can omit with the allocator case since it doesn’t have those edge cases. We don’t have to deal with such cases in the allocator case, unless we truly exhaust the physical addressing space of our hardware (which the above code handles, but it doesn’t have to deal with “out of pre-allocated buffer” separately from “out of memory”).

Additional design idea for #3

When possible, also provide the maximum size needed for mytype in the same .h file as mytype_to_string().

#define MYTYPE_TO_STRING_SIZE 256

Now the user can code accordingly.

char buf[MYTYPE_TO_STRING_SIZE];
puts(mytype_to_string(mt, buf, sizeof buf));

Order

The size of arrays, when first, allows for VLA types.

char * mytype_to_string( const mytype_t *mt, size_t bufsize, char *buf[bufsize]); 

Not so important with single dimension, yet useful with 2 or more.

void matrix(size_t row, size_t col, double matrix[row][col]);

I recall reading having the size first is a preferred idiom in the next C. Need to find that reference….

As an addition to @ratchetfreak’s excellent answer, I would point out that alternative #3 follows a similar paradigm/pattern as standard C library functions.

For example, strncpy.

char * strncpy ( char * destination, const char * source, size_t num );

Following the same paradigm would help to reduce the cognitive load for new developers (or even your future self) when they need to use your function.

The only difference with what you have in your post would be that the destination argument in the C libraries tends to be listed first in the argument list. So:

char * mytype_to_string( char *buf, const mytype_t *mt, size_t buflen ); 

Besides the fact that what you’re proposing to do is a bad code smell, alternative 3 sounds best to me. I also think like @gnasher729 that you’re using the wrong language.

To be honest, you might want to switch to a different language where returning a string isn’t a complex, work intensive and error prone operation.

You could consider C++ or Objective-C where you could leave 99% of your code unchanged.