I have to update a service written in go to consume messages from a new avro kafka topic, before it was just JSON. I originally wrote the code pretty much following the documentation of hamba/avro defining the schema like:

var partialSchemaAccounts = avro.MustParse(`{
    "type": "record", 
    "name": "accounts", 
    "fields": [ 
        { "name": "created_at", "type": [ "null", "string" ] },
        { "name": "some_field", "type": "boolean" }
    ]
}`)

and then the account struct like:

type account struct {
    Created_at     *string `avro:"created_at"`
    some_field     bool    `avro:"some_field"`
}

then did all the createConsumer, subscribeTopics, readMessage stuff to then try and unmarshall like:

var account account
err := avro.Unmarshal(partialSchemaAccounts, payload, &account)

but instead of properly mapping the values I just get empty fields for created_at and some_field

I am now quite certain that the problem is related to this whole schema registry magic, as far as I understood, the first 5 bytes of all of the messages are 1 magic byte and then 4 bytes that contain the schema id. I don’t see the point in fetching the schema every time as this service really only need these two fields and that part of the schema shouldn’t ever really change.

My idea would be to ‘cut’ the first 5 bytes off of each message like message = msg.Value[5:] before trying the Unmarshall(). It feels a bit illegal though and I am also not sure if it will work as expected. What are your thoughts on this? I am also looking into ways of dealing with schema registry in go but I find it hard to find packages that fit my use case without making the code even more complex.

This: https://github.com/dangkaka/go-kafka-avro looks like it was originally what I needed, but it’s apparently not maintained anymore which makes me not want to use it here.