/
opt
/
gsutil
/
gslib
/
utils
/
__pycache__
/
up file
home
a +(Wg�G � @ sD d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl m Z ddl m Z dd l mZ dd lmZ dd lmZ dd lmZ dd lmZ ddlmZ ddlmZ ddlmZ ddlmZ ddl m Z ddgZ g d�Z d*dd�Z dd� Z!dd � Z"d d � Z#d d � Z$d!d"� Z%d+d$d%�Z&d&d'� Z'G d(d)� d)e(�Z)dS ),zCUtility functions and class for listing commands such as ls and du.� )�absolute_import)�print_function)�division)�unicode_literalsN)�EncryptionException)�CommandException)�PluralityCheckableIterator)�GenerationFromUrlAndString)�S3_ACL_MARKER_GUID)�S3_DELETE_MARKER_GUID)�S3_MARKER_GUIDS)�UTF8)� IS_WINDOWS)�AclTranslation)� text_util)�StorageUrlFromString�md5Hash�crc32c)�acl� cacheControl�componentCount�contentDisposition�contentEncoding�contentLanguage� contentType� customTime� kmsKeyName�customerEncryption�etag�eventBasedHold� generation�metadata�metageneration�retentionExpirationTime�size� storageClass� temporaryHold� timeCreated� timeDeleted�timeStorageClassUpdated�updated� c C s" d� d| d | d �d�|�S )a� Returns a string with a vertically aligned label and value. Labels of the same indentation level will start at the same column. Values will all start at the same column (unless the combined left-indent and label length is excessively long). If a value spans multiple lines, indentation will only be applied to the first line. Example output from several calls: Label1: Value (default indent of 1 was used) Sublabel1: Value (used indent of 2 here) Label2: Value Args: label: The label to print in the first column. value: The value to print in the second column. indent: (4 * indent) spaces will be placed before the label. Returns: A string with a vertically aligned label and value. z{}{}� � �:� )�format�ljust)�label�value�indent� r5 �$/opt/gsutil/gslib/utils/ls_helper.py�MakeMetadataLineF s r7 c C s dS )z�Default function for printing headers for buckets. Header is printed prior to listing the contents of the bucket. Args: bucket_listing_ref: BucketListingRef of type BUCKET. Nr5 ��bucket_listing_refr5 r5 r6 �PrintBucketHeader] s r: c C s t �| j� dS )z�Default function for printing buckets or prefixes. Args: bucket_listing_ref: BucketListingRef of type BUCKET or PREFIX. N)r � print_to_fd� url_stringr8 r5 r5 r6 �PrintDirh s r= c C s dS )z�Off-by-default function for printing buckets or prefix size summaries. Args: num_bytes: Number of bytes contained in the directory. bucket_listing_ref: BucketListingRef of type BUCKET or PREFIX. Nr5 )� num_bytesr9 r5 r5 r6 �PrintDirSummaryr s r? c C s t �d�| j�� dS )z�Default function for printing headers for prefixes. Header is printed prior to listing the contents of the prefix. Args: bucket_listing_ref: BucketListingRef of type PREFIX. �{}:N)r r; r0 r< r8 r5 r5 r6 �PrintDirHeader| s rA c C s t �� dS )z<Default function for printing new lines between directories.N)r r; r5 r5 r5 r6 � PrintNewLine� s rB Tc C s | j }t|�}| j}|jr8t|jjv r8d}d}|d7 }n |j}d}t�d� |�� |j rpt�t d|j � d��� |j r�t�t d|j � d��� |jr�|j|j kr�t�t d|j� d��� |jr�t�t d |j�� |jr�t�t d d �� |jr�t�t d d �� |j�r t�t d |j� d��� |j�r6t�t d|j�� |j�rPt�t d|j�� |j�rjt�t d|j�� |j�r�t�t d|j�� |j�r�t�t d|j�� t�t d|j�� t�t d|j�� |j�r�t�t d|j�� |j�r�t�t d|j�� |j�rt�t d|j� d��� i }|j�r�|jj�r�g }|jjD ]*} | j t v�rT|� | � n | j || j < �q8|�r�t�t dd�� |D ]6} d� | j �} d� | j �} t | | dd �} t�| � �q~|j �r|j!�s�t�t d d �� |j"�s�t�t d d �� t�t d |j j#�� t�t d!|j j$�� |j!�r0t�t d |j!�� |j"�rJt�t d |j"�� t�t d"|j%�&d#��� |j'�r�t(||j'�}t�t d$|�� |j)�r�t�t d%|j)�� |�r�|j*�r�t�t d&t+�,|j*��� n0t-|v �r�t�t d&|t- �� nt�t d&d'�� ||fS )(a� Print full info for given object (like what displays for gsutil ls -L). Args: bucket_listing_ref: BucketListingRef being listed. Must have ref_type OBJECT and a populated root_object with the desired fields. incl_acl: True if ACL info should be output. Returns: Tuple (number of objects, object_length) Raises: Exception: if calling bug encountered. r z<DeleteMarker>r+ r@ z Creation timez%a, %d %b %Y %H:%M:%S GMTz Update timezStorage class update timez Storage classzTemporary Hold�EnabledzEvent-Based HoldzRetention ExpirationzKMS keyz Cache-ControlzContent-DispositionzContent-EncodingzContent-LanguagezContent-Lengthz Content-TypezComponent-Countz Custom-TimezNoncurrent timeZMetadata� z{}� )r4 z Hash (crc32c)� encryptedz Hash (md5)zEncryption algorithmzEncryption key SHA256�ETagz"'� GenerationZMetageneration�ACLz[]).r< r � root_objectr! r �additionalPropertiesr$ r r; r0 r' r7 �strftimer* r) r% r& r r# r r r r r r r r r( �keyr �appendr3 r r r �encryptionAlgorithm� keySha256r �stripr r r" r r �JsonFromMessager )r9 Zincl_acl�url_str� storage_url�objr> �num_objsZ marker_propsZnon_marker_propsZadd_propZapZap_keyZap_valueZmeta_data_line�generation_strr5 r5 r6 �PrintFullInfoAboutObject� s � �� �� � �� �� � � � � �� ���� �� �rX c C sJ zt �| j� W n4 tyD } z tr.|jdks0� W Y d}~n d}~0 0 dS )z�Default printing function for objects. Args: bucket_listing_ref: BucketListingRef of type OBJECT. Returns: (num_objects, num_bytes). r N)r+ r )r r; r< �IOErrorr �errno)r9 �er5 r5 r6 � PrintObject s r\ c @ sJ e Zd ZdZeeeeee dddddf dd�Z dd � Z dd d �Z d d � Z dS )�LsHelperzHelper class for ls and du.FN)�nameTc C sR || _ || _|| _|| _|| _|| _|| _|| _| | _| | _ | | _ | | _ | | _ dS )a� Initializes the helper class to prepare for listing. Args: iterator_func: Function for instantiating iterator. Inputs- url_string- Url string to iterate on. May include wildcards. all_versions=False- If true, iterate over all object versions. logger: Logger for outputting warnings / errors. print_object_func: Function for printing objects. print_dir_func: Function for printing buckets/prefixes. print_dir_header_func: Function for printing header line for buckets or prefixes. print_bucket_header_func: Function for printing header line for buckets or prefixes. print_dir_summary_func: Function for printing size summaries about buckets/prefixes. print_newline_func: Function for printing new lines between dirs. all_versions: If true, list all object versions. should_recurse: If true, recursively listing buckets/prefixes. exclude_patterns: Patterns to exclude when listing. fields: Fields to request from bucket listings; this should include all fields that need to be populated in objects so they can be listed. Can be set to None to retrieve all object fields. Defaults to short listing fields. list_subdir_contents: If true, return the directory and any contents, otherwise return only the directory itself. N) �_iterator_func�logger�_print_object_func�_print_dir_func�_print_dir_header_func�_print_bucket_header_func�_print_dir_summary_func�_print_newline_func� all_versions�should_recurse�exclude_patterns�bucket_listing_fields�list_subdir_contents)�selfZ iterator_funcr` �print_object_func�print_dir_func�print_dir_header_funcZprint_bucket_header_func�print_dir_summary_func�print_newline_funcrg rh ri �fieldsrk r5 r5 r6 �__init__' s ,zLsHelper.__init__c C s� d}d}d}d}|� � s | jr@|� � r0| �|� | j|jdd�S |�� rP|j}n|�� }t| j|| j d�j d| j d��}|� � }z |� � W n@ ty� t| j|jdd�| j d�j dtd��}|� � }Y n0 |D ]�} | �| �r�q�| �� �rd} | �| �\} } d}n�| �� �r�|�r | �� nd}|�r<| j�r<| �| � n|�rN| j�sNd}t| j�j| j�rdd ndd�} | �| �\} } } | �| | � ntd ��|| 7 }|| 7 }|| 7 }q�|||fS dS ) z�Iterates over the given URL and calls print functions. Args: url: StorageUrl to iterate over. Returns: (num_objects, num_bytes) total number of objects and bytes iterated. r F)�print_initial_newline�rg T��expand_top_level_bucketsrj N��wildcard_suffix�*z?Sub-level iterator returned a CsBucketListingRef of type Bucket)�IsBucketrh rd �_RecurseExpandUrlAndPrintr< � HasGeneration�CreatePrefixUrlr r_ rg �IterAllrj � HasPlurality� PeekExceptionr � UNENCRYPTED_FULL_LISTING_FIELDS�_MatchesExcludedPattern�IsObjectra �IsPrefixrf rk rc r re r )rl �url� num_objects�num_dirsr> Z print_newlineZ iteration_urlZtop_level_iteratorZ plurality�blr�nd�no�nb�expansion_url_strr5 r5 r6 �ExpandUrlAndPrinta s� ���� ��� ��� zLsHelper.ExpandUrlAndPrintc C s� d}d}d}| j d| | jd�jd| jd�D ]�}| �|�r<q,|�� rXd}| �|�\}} nx|�� r�| jr�|rt| � � nd}| � |� t |j �j dd�} | �| �\}}} | �| |� q�d\}}} | �|� ntd ��||7 }||7 }|| 7 }q,|||fS ) a� Iterates over the given URL string and calls print functions. Args: url_str: String describing StorageUrl to iterate over. Must be of depth one or higher. print_initial_newline: If true, print a newline before recursively expanded prefixes. Returns: (num_objects, num_bytes) total number of objects and bytes iterated. r z%sru Trv rz rx )r+ r r z=Sub-level iterator returned a bucketListingRef of type Bucket)r_ rg r rj r� r� ra r� rh rf rc r r< r~ r| re rb r ) rl rS rt r� r� r> r� r� r� r� r� r5 r5 r6 r| � sH �� �� � z"LsHelper._RecurseExpandUrlAndPrintc C s: | j r6t�|j�}| j D ] }t�|t�|��r dS qdS )z�Checks bucket listing reference against patterns to exclude. Args: blr: BucketListingRef to check. Returns: True if reference matches a pattern and should be excluded. TF)ri �six� ensure_strr< �fnmatch)rl r� Ztomatch�patternr5 r5 r6 r� � s z LsHelper._MatchesExcludedPattern)T)�__name__� __module__� __qualname__�__doc__r\ r= rA r: r? rB rs r� r| r� r5 r5 r5 r6 r] $ s � :N 2r] )r+ )T)*r� � __future__r r r r r� �sysr� �gslib.cloud_apir �gslib.exceptionr �"gslib.plurality_checkable_iteratorr �gslib.storage_urlr �gslib.utils.constantsr r r r Zgslib.utils.system_utilr � gslib.utils.translation_helperr � gslib.utilsr Zgslib.wildcard_iteratorr ZENCRYPTED_FIELDSr� r7 r: r= r? rA rB rX r\ �objectr] r5 r5 r5 r6 �<module> sB �