I bet our sizeHint is mostly accurate.  So lets use that to avoid copying.  But fall back into a copying behavior when its wrong:

  byte[] out = new byte[sizeHint];
  int pos = 0;
  while (pos < out.length) {
    int read = in.read(out, pos, out.length - pos);
    if (read < 0) {
      byte[] res = new byte[pos];
      System.arraycopy(out, 0, res, 0, pos);
      return res;
    }

    pos += read;
  }

  int last = in.read();
  if (last < 0)
    return out;

  TemporaryBuffer.Heap tmp = new TemporaryBuffer.Heap(Integer.MAX_VALUE);
  tmp.write(out);
  tmp.write(last);
  tmp.copy(in);
  return tmp.toByteArray();