fmt: add Append, Appendf, Appendln

[seebs]

What version of Go are you using (go version)?

1.16

Does this issue reproduce with the latest release?

Sure.

What operating system and processor architecture are you using (go env)?

N/A

What did you do?

Profiled something that was using Sprintf in a minor way.

What did you expect to see?

Less Sprintf in the profile.

What did you see instead?

So much Sprintf.

The issue here is that most of what's happening is WAY cheaper than allocations. strconv has AppendInt, but that can't do formatting.

It makes sense, for most uses, that fmt.Sprintf is an allocator that produces a string, but there are times when you want formatted-output, but want to write into a []byte. And you can do that with bytes.Buffer and fmt.Fprintf, but creating a bytes.Buffer around a []byte is... an allocation. And fmt.Fprintf does an allocation. (Curiously, if I do fmt.Fprintf into a bytes.Buffer I just made for that purpose, I only get the one allocation.)

What I want: Something like Sprintf, but that can write into a []byte, and can fail gracefully if there's not enough space to write things.

Proposed name: fmt.Snprintf, because snprintf is what you call when you already have a buffer you want written to and you have a length limit in mind.

So, fmt.Snprintf(dest []byte, fmt string, args ...interface{}) (int, error), perhaps. The C standard's answer to "what if n isn't big enough" is "you report the n you would have used if n had been big enough", which exists to allow a single-pass process to figure out how much space you actually need. Alternatively, it could return number of bytes actually written, and if it didn't fit, an error with a concrete type that indicates space needed.

I note, browsing source, that fmt already has fmtBytes, although this doesn't do quite the thing this would need.

[ianlancetaylor]

Edited to add: this doesn't work, as the fmt package pools buffers and will reuse the byte slice.

You could do this instead of adding new API;

package x_test

import (
	"bytes"
	"fmt"
	"testing"
)

type sliceStealer struct {
	b []byte
}

func (ss *sliceStealer) Write(s []byte) (int, error) {
	if len(ss.b) > 0 {
		panic("multiple calls to Write")
	}
	ss.b = s
	return len(s), nil
}

func FprintfToBytes(format string, a ...interface{}) []byte {
	var ss sliceStealer
	fmt.Fprintf(&ss, format, a...)
	return ss.b
}

func TestFprintf(t *testing.T) {
	s := FprintfToBytes("%d", 10)
	if !bytes.Equal(s, []byte("10")) {
		t.Errorf(`FprintfToBytes("%%d", 10) = %q, want %q`, s, "10")
	}
}

func BenchmarkFprintf(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		FprintfToBytes("%d", 10)
	}
}

[DeedleFake]

~~It looks like an allocation can be eliminated by creating a simpler type locally instead of using bytes.Buffer. Does the allocation with bytes.Buffer happen because of the return of a pointer from inside of bytes.NewBuffer()?

Edit: Ah, darn. @ianlancetaylor beat me to it.~~

Edit 2: Never mind. See #47579 (comment) below.

[seebs]

Is there any guarantee that Fprintf does only one write? Also, while that allows me to steal the buffer fprintf used, it doesn't let me write into an existing buffer that I have, which is the really interesting case. For instance, to put a 0-padded value in filename[5:9] or something. The distinction being, with that, BenchmarkFprintf could report 0 allocs/op, theoretically.

So right now, I don't think it's possible to loop on Fprintf without at least one allocation per call...

[josharian]

Also, while that allows me to steal the buffer fprintf used, it doesn't let me write into an existing buffer that I have, which is the really interesting case.

The Write method of your io.Writer could copy the buffer's contents into your existing buffer.

It does mean an extra copy of the data, which is cheap but not free. In any case that should let you avoid the allocation.

Also, I'm not sure the slice stealer is safe. IIRC package fmt pools its buffers, so the buffer passed to Write will be used again by subsequent fmt work.

[ianlancetaylor]

Also, I'm not sure the slice stealer is safe. IIRC package fmt pools its buffers, so the buffer passed to Write will be used again by subsequent fmt work.

Good point.

[ianlancetaylor]

fmt.Snprintf(dest []byte, fmt string, args ...interface{}) (int, error)

I think we can simplify the issues about exceeding the size of the buffer by making this

fmt.Snprintf(dest []byte, fmt string, args ...interface{}) ([]byte, error)

That is, just return the byte slice, which will be a new slice if the original cap is not large enough. People who need to limit the size of the output, which I suspect is less common, can use Fprintf with a custom Writer. (But with this modification Snprintf is not the right name.)

As it happens, I think this might be fairly simple to implement: replace the pp.buf field with the passed in buffer, and then remove it and restore the original pp.buf field before releasing the pp.

[josharian]

If Sprintf is string-printf, then it seems to me that bytes-printf should be Bprintf.

[seebs]

Bprintf would also make sense to me. Hmm. "if cap isn't large enough, then allocate" seems like a reasonable choice, but I don't see a pretty/clean way to express "if this didn't fit in the space I provided". I guess something like if &provided[0] != &returned[0] maybe? (This isn't quite safe in all theoretical cases, because the returned len could be 0, but if the returned len is 0, I don't have to worry about the allocation, do I?)

I'd thought about creating a thing which just copies into an existing buffer, but then fmt's still allocating -- but if it's pooling those, I guess I care less about that than I thought I did.

got it! fmt.Apprendf

[josharian]

I don’t think the function would return an error; fmt functions only return errors when writes fail. So the signature would be

func Bprintf(dest []byte, format string, args ...interface{}) []byte

[josharian]

I don't see a pretty/clean way to express "if this didn't fit in the space I provided".

How about cap(provided) != cap(returned)? Can’t grow without modifying cap.

[seebs]

Yeah. I just dislike the lack of a way to express "is this a different thing" separately from "does this thing have different characteristics". And I suppose I could imagine a case where the desired outcome would be "if for some reason it doesn't fit, fail rather than allocating", but that feels like a different thing.