I think things will be easier if inflate is defined as:

  private InputStream inflate(final Source src, final long inflatedSize)

Where it returns an InputStream that the caller can read to get the inflated data from.  For performance reasons we should try to use a single InputStream object, just like we use a single Inflater, and just reset it along with the Inflater when the method is invoked.

This way above in inflateAndReturn we can use IO.readFully() to read from the InputStream directly into the byte[] we have allocated and will return to the caller.  This reduces the number of allocations and copies involved.