I should start with the fact that I am a relative newcomer to go, so apologies if this is a dumb question.
I’ve read / watched a few things that make the point that it is good practice (if not an idiom) to try and pass, say, a reader to a function, rather than, say, a byte slice. This feels like sound advice, as it makes functions more reusable, which is a good thing. So, I have been trying it out and I’ve got to a place where I think I am going astray.
The thing is, a lot of online code examples, make use of byte slices, not readers / writers etc. I am going to use base64 encoding as an example.
someEncodedString := base64.StdEncoding.EncodeToString([]byte("hello world"))
OK, so this means I can take the easy way out and write a function to base64 encode like this (and I appreciate that the example is somewhat trite) :
func Encode(p []byte) string {
return base64.StdEncoding.EncodeToString(p)
}
but if I want to follow the “pass readers not byte slices” rule, I’m now going to have to read from the reader, which needs a byte slice:
func Encode(r io.Reader) (string, err) {
p := make([]byte, 10)
for {
n, err := r.Read(p)
if err != nil {
if err == io.EOF {
...
and somewhere along the road I have to take all the iterations of reading into p and assemble them into something and then base64 encoding that and it all feels like a bit of a palaver.
Now, I know that, sometimes, it is useful to break something large into smaller chunks (for example, to avoid hoovering up all the memory in your system, or to pass to go routines to speed up processing of large amounts of data etc) but, let’s park that for a minute…
All I want to do is take a reader, encode the data in that reader and then (what seems the logical next step) return a reader for the next function that takes a reader. So that logic led me to write this:
func NewB64EncodedBuffer(r io.Reader) (*bytes.Buffer, error) {
var b bytes.Buffer
e := base64.NewEncoder(base64.StdEncoding, &b)
n, err := io.Copy(e, r)
if err != nil {
return &b, err
}
log.Printf("bytes written to buffer: [%v]n", n)
return &b, nil
}
…which all feels nice and tidy. I take a reader return a buffer (a nice readWriter, no less) this seems like a really flexible pattern. Now I can pass buffers, files, http requests etc and it all just works. The niggle is this line:
n, err := io.Copy(e, r)
The io.copy
function is either a Swiss Army Knife or it’s a dangerous hack and I cannot seem to work out which it is (it could be both!)
I can do things like this:
io.Copy(os.Stdout, encodedBuff)
and this:
func RequestEncoderHandlerFunc(w http.ResponseWriter, r *http.Request) {
b, e := NewB64EncodedBuffer(r.Body)
if e != nil {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte("something went wrong"))
}
w.WriteHeader(http.StatusOK)
io.Copy(w, b)
}
but suddenly I start to feel like I am just writing sloppy code and I need some advice to put me back on the straight and narrow.
It feels like I could just keep using io.Copy
all over the place but is this really the best method of passing data between readers and writers, or is the answer “this is fine, just be careful with memory usage etc”.
Steve Gall is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.