Attachments stored in the S3 blob store are truncated when a page is deleted or exported as XAR

Problem

When the attachment content of a page comes from a stream that can return fewer bytes than requested before its end, which is the case of the S3 blob store, the XAR serialization of the page keeps only a prefix of each attachment. This affects:

• the recycle bin: deleting a page stores the page with its attachments as XAR XML, so the deleted page holds truncated attachments and restoring it restores the truncated attachments (data loss)
• the XAR export of pages with such attachments (same serializer, not tested on a real instance)

Found on a test cluster using the S3 blob store: after deleting pages, the recycle bin entries held 7761 of 102400 bytes, 7763 of 8192 bytes and 13140 of 12582912 bytes of the attachments, while the 4096-byte attachments were intact. Restoring the page with the 102400-byte attachment gave a 7761-byte attachment (its first 7761 bytes) on every cluster member.

Cause

XAROutputFilterStream#writeContent reads the attachment content in chunks of 4095 bytes with InputStream#read(byte[], int, int) and stops at the first read returning fewer than 4095 bytes, as if it was the end of the stream. The InputStream contract allows a read to return fewer bytes than requested before the end, and the stream of an S3 object (an HTTP response body) does so routinely, whereas a file stream fills the buffer in practice, which is why the filesystem store is not affected. The loop dates from the creation of the XAR output stream (5.2).

Fix

Fill each chunk with InputStream#readNBytes so that only the last one can be shorter. The chunk size stays a multiple of 3, so the Base64 output is unchanged for streams that already filled each read and has no padding between chunks (see XWIKI-9830).