/
opt
/
gsutil
/
gslib
/
__pycache__
/
up file
home
a +(Wg" � @ s� d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl m Z ddl m Z dd l mZ G d d � d e�ZdS ) z-Helper class for streaming resumable uploads.� )�absolute_import)�print_function)�division)�unicode_literalsN)�CommandException)�GetJsonResumableChunkSize)�UTF8c @ sV e Zd ZdZddd�Zedd� �Zddd �Zd d � Zd d � Z e j fdd�Z dd� Z dS )�#ResumableStreamingJsonUploadWrappera# Wraps an input stream in a buffer for resumable uploads. This class takes a non-seekable input stream, buffers it, and exposes it as a stream with limited seek capabilities such that it can be used in a resumable JSON API upload. max_buffer_size bytes of buffering is supported. Fc C sL || _ |s&|t� k r&td|t� f ��|| _t�� | _d| _d| _d| _ dS )av Initializes the wrapper. Args: stream: Input stream. max_buffer_size: Maximum size of internal buffer; should be >= the chunk size of the resumable upload API to ensure that at least one full chunk write can be replayed in the event of a server error. test_small_buffer: Skip check for buffer size vs. chunk size, for testing. z�Resumable streaming upload created with buffer size %s, JSON resumable upload chunk size %s. Buffer size must be >= JSON resumable upload chunk size to ensure that uploads can be resumed.r N) �_orig_fpr r �_max_buffer_size� collections�deque�_buffer� _buffer_start� _buffer_end� _position)�self�stream�max_buffer_sizeZtest_small_buffer� r �//opt/gsutil/gslib/resumable_streaming_upload.py�__init__) s � z,ResumableStreamingJsonUploadWrapper.__init__c C s t | jdd�S )z<Returns the mode of the underlying file descriptor, or None.�modeN)�getattrr �r r r r r C s z(ResumableStreamingJsonUploadWrapper.mode���c C sV |du p|dk }|r | j }n|}d}g }| j| jk r�| j}d}|t| j| � | jk rr|t| j| �7 }|d7 }q>|| jk r�|dkr�t| j| �}| j| } || } t| |�} |�| j| | | | � � || 8 }||7 }|d7 }| j| 7 _qr|�r>| j� |�} t| �} |�s| }n|�| � d� |�}| j| 7 _�n|�r"| j� |�} |�s\| }n|�| � d� |�}t| �} | �rR| j| 7 _| j�| � | j| 7 _d}| j| j | j k�r�| j� � }| jt|�7 _�q�|�rR| j | j| j }|�rR| j� || d� � | j|8 _n0t j�r>|�r>dd� |D �}|�rNd� |�nd}|S )z�"Reads from the wrapped stream. Args: size: The amount of bytes to read. If omitted or negative, the entire contents of the stream will be read and returned. Returns: Bytes from the wrapped stream. Nr � � c S s$ g | ] }t |t�r |�t�n|�qS r )� isinstance�str�encoder )�.0Zbdr r r � <listcomp>� s �z<ResumableStreamingJsonUploadWrapper.read.<locals>.<listcomp>)r r r r �lenr �min�appendr �read�join�popleft� appendleft�six�PY3)r �sizeZread_all_bytes�bytes_remaining�dataZ buffered_dataZ pos_in_bufferZ buffer_indexZ buffer_lenZoffset_from_positionZbytes_available_this_buffer� read_size�new_data�data_lenZ oldest_dataZ refill_amountr r r r& H s~ �� ��z(ResumableStreamingJsonUploadWrapper.readc C s | j S )z$Returns the current stream position.)r r r r r �tell� s z(ResumableStreamingJsonUploadWrapper.tellc C s dS )z/Returns true since limited seek support exists.Tr r r r r �seekable� s z,ResumableStreamingJsonUploadWrapper.seekablec C s� |t jkr<|| jk s || jkr4td|| j| jf ��|| _nT|t jkr�|| jkrbtd|| jf ��| �| j�rpqb| j|8 _ntd||f ��dS )z�Seeks on the buffered stream. Args: offset: The offset to seek to; must be within the buffer bounds. whence: Must be os.SEEK_SET. Raises: CommandException if an unsupported seek mode or position is used. z�Unable to resume upload because of limited buffering available for streaming uploads. Offset %s was requested, but only data from %s to %s is buffered.zHInvalid SEEK_END offset %s on streaming upload. Only %s can be buffered.z;Invalid seek mode on streaming upload. (mode %s, offset %s)N) �os�SEEK_SETr r r r �SEEK_ENDr r& )r �offset�whencer r r �seek� s" � � �z(ResumableStreamingJsonUploadWrapper.seekc C s | j �� S )N)r �closer r r r r: � s z)ResumableStreamingJsonUploadWrapper.closeN)F)r )�__name__� __module__� __qualname__�__doc__r �propertyr r& r2 r3 r4 r5 r9 r: r r r r r s e!r )r> � __future__r r r r r r4 r* �gslib.exceptionr �gslib.utils.boto_utilr �gslib.utils.constantsr �objectr r r r r �<module> s