/
opt
/
gsutil
/
gslib
/
__pycache__
/
up file
home
a +(Wg�9 � @ s� d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl m Z ddl m Z dd lmZ dd lmZ dd lmZ d ZG d d� de�Zejdd� �ZG dd� de�ZdS )z(Wrapper for use in daisy-chained copies.� )�absolute_import)�division)�print_function)�unicode_literalsN)�BadRequestException)�CloudApi)� constants)�parallelism_framework_util)�CryptoKeyWrapperFromKeyi @c @ s" e Zd ZdZddd�Zdd� ZdS ) � BufferWrapperz<Wraps the download file pointer to use our in-memory buffer.�bc C s$ || _ t|d�r|j| _n|| _dS )z�Provides a buffered write interface for a file download. Args: daisy_chain_wrapper: DaisyChainWrapper instance to use for buffer and locking. �modeN)�daisy_chain_wrapper�hasattrr )�selfr r � r �(/opt/gsutil/gslib/daisy_chain_wrapper.py�__init__, s zBufferWrapper.__init__c C s� | j j�. | j j| j jk r(W d � qRW d � n1 s<0 Y t�d� q t|�}|r�| j j�. | j j�|� | j j|7 _W d � n1 s�0 Y dS )z>Waits for space in the buffer, then writes data to the buffer.Nr ) r �lock�bytes_buffered�max_buffer_size�time�sleep�len�buffer�append)r �data�data_lenr r r �write9 s �, zBufferWrapper.writeN)r )�__name__� __module__� __qualname__�__doc__r r r r r r r ) s r c c s" | j |d�}|V |r | �� d S )N)�timeout)�acquire�release)r r# �resultr r r �AcquireLockWithTimeoutI s r' c @ sT e Zd ZdZddedfdd�Zddd�Zdd d �Zd d � Ze j fd d�Z dd� Z dS )�DaisyChainWrappera� Wrapper class for daisy-chaining a cloud download to an upload. This class instantiates a BufferWrapper object to buffer the download into memory, consuming a maximum of max_buffer_size. It implements intelligent behavior around read and seek that allow for all of the operations necessary to copy a file. This class is coupled with the XML and JSON implementations in that it expects that small buffers (maximum of constants.TRANSFER_BUFFER_SIZE) in size will be used. FNc C s� d| _ t�� | _d| _d| _|| _d| _d| _t � � | _ t � � | _ || _ || _|| _t|�| _|| _d| _d| _|| _t�� | _t�� | _| j| jd� | j�d�s�t d��d| _ dS )a` Initializes the daisy chain wrapper. Args: src_url: Source CloudUrl to copy from. src_obj_size: Size of source object. gsutil_api: gsutil Cloud API to use for the copy. compressed_encoding: If true, source object has content-encoding: gzip. progress_callback: Optional callback function for progress notifications for the download thread. Receives calls with arguments (bytes_transferred, total_size). download_chunk_size: Integer number of bytes to download per GetObjectMedia request. This is the upper bound of bytes that may be unnecessarily downloaded if there is a break in the resumable upload. decryption_key: Base64-encoded decryption key for the source object, if any. Raises: Exception: if the download thread doesn't start within 60 seconds r i N)�progress_callback�< z1Could not start download thread after 60 seconds.� ) �position� collections�dequer r r �_download_chunk_size� last_position� last_datar � CreateLockr �download_exception_lock� src_obj_size�src_url�compressed_encodingr �decryption_tuple� gsutil_api�download_exception�download_threadr) � threading�Event�download_started� stop_download�StartDownloadThread�wait� Exception� _unused_data_from_previous_read)r r5 r4 r8 r6 r) Zdownload_chunk_size�decryption_keyr r r r ^ s. zDaisyChainWrapper.__init__r c s. � fdd�}t j|||fd�� _� j�� dS )zCStarts the download thread for the source object (from start_byte).c s, � j �� z�| � j � jk r�� jj� jj� jjt � �� j | | � j d � jj � jt j j� jj|� jd� � j�� r�� j�� W dS | � j7 } q � jj� jj� jjt � �� j | � jj � jt j j� jj|� jd� W nV t�y& } z<� j� |� _� W d � n1 �s0 Y W Y d}~n d}~0 0 dS )a" Downloads the source object in chunks. This function checks the stop_download event and exits early if it is set. It should be set when there is an error during the daisy-chain upload, then this function can be called again with the upload's current position as start_byte. Args: start_byte: Byte from which to begin the download. progress_callback: Optional callback function for progress notifications. Receives calls with arguments (bytes_transferred, total_size). � ) r6 � start_byte�end_byte� generation� object_size�download_strategy�providerr) r7 N)r6 rE rG rH rI rJ r) r7 )r= �setr/ r4 r8 �GetObjectMediar5 � bucket_name� object_namer r6 rG r �DownloadStrategy�ONE_SHOT�schemer7 r> �is_set�clearrA r3 r9 )rE r) �e�r r r �PerformDownload� sL � � z>DaisyChainWrapper.StartDownloadThread.<locals>.PerformDownload)�target�argsN)r; �Threadr: �start)r rE r) rV r rU r r? � s :��z%DaisyChainWrapper.StartDownloadThreadc C sz | j | jks|dkrdS |du s*|tjkr<td|tjf ��| jrb| jd|� }| j|d� | _nvd}| j�P | jr�W d � q�t| j d�r�| j r�| j �n| j � � s�t d��W d � n1 s�0 Y t�d� qf| j�l |�s | j�� }|d|� }||d� | _| j | _|| _t|�}| j |7 _ | j|8 _W d � n1 �sR0 Y ||k�rvtd||f ��|S ) z9Exposes a stream from the in-memory buffer to the upload.r � NzGInvalid HTTP read size %s during daisy chain operation, expected <= %s.r+ � z Download thread died suddenly.zQInvalid read during daisy chain operation, got data of size %s, expected size %s.)r, r4 r �TRANSFER_BUFFER_SIZEr rB r r r' r3 r9 r: �is_aliverA r r �popleftr0 r1 r r )r �amtZ valid_datar r r r r �read� sN �� � & . ��zDaisyChainWrapper.readc C s0 | j � | jW d � S 1 s"0 Y d S )N)r r, rU r r r �tell" s zDaisyChainWrapper.tellc C s� d}|t jkr^|r td| ��| j�& | j| _d| _| j| _W d � n1 sP0 Y �nd|t jk�r�| j�\ || jkr~n@|| jkr�| j| _| jr�| j � | j� | j t | j�7 _ nd}W d � n1 s�0 Y |�r�| j �� | j�� �rP| j�4 | j �r$| j t | j �� �8 _ �q W d � n1 �s:0 Y t�d� q�| j�< || _t�� | _ d| _ d| _d| _| j �� W d � n1 �s�0 Y | j|| jd� n td| ��dS )z)Sets current read position of the stream.Fz_Invalid seek during daisy chain operation. Non-zero offset %s from os.SEEK_END is not supportedNTr )rE r) z:Daisy-chain download wrapper does not support seek mode %s)�os�SEEK_END�IOErrorr r, r0 r1 r4 �SEEK_SETr � appendleftr r r> rK r: r^ r_ r r r- r. rS r? r) )r �offset�whenceZrestart_downloadr r r �seek& sT ��* " < *��zDaisyChainWrapper.seekc C s dS )NTr rU r r r �seekable_ s zDaisyChainWrapper.seekable)r N)N) r r r! r" � _DEFAULT_DOWNLOAD_CHUNK_SIZEr r? ra rb rc rf rj rk r r r r r( Q s � H B :9r( )r" � __future__r r r r r- � contextlibrc r; r �gslib.cloud_apir r � gslib.utilsr r � gslib.utils.encryption_helperr rl �objectr �contextmanagerr' r( r r r r �<module> s&