/
opt
/
gsutil
/
gslib
/
utils
/
__pycache__
/
up file
home
a +(Wg�l � @ s� d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlZddlZddl Z ddl Z ddl m Z ddl mZ dd lmZ dd lmZ dd lmZ dd lmZ dd lmZ G dd� dej�ZG dd� dej�Ze �d�Ze g d��Z e e g d��B Z e g d��Z e g d��Z e g d��Z!ddddd d �d d d d!d"d#d$d%d&d'd(� d)d*d+d,d-d.d/d0�d1d2d3d4d5�d6�Z"e d7d8g�Z#dPd:d;�Z$d<d=� Z%d>d?� Z&G d@dA� dAe'�Z(G dBdC� dCe'�Z)dDdE� Z*dFdG� Z+dHdI� Z,dJdK� Z-dLdM� Z.G dNdO� dOe'�Z/dS )QzCHelper for shim used to translate gsutil command to gcloud storage.� )�absolute_import)�print_function)�division)�unicode_literalsN)�config)� exception)� ApiSelector)�CommandException)� boto_util)� constants)� system_utilc @ s e Zd ZdZdZdZdS )�HIDDEN_SHIM_MODEZ no_fallback�dry_run�noneN)�__name__� __module__� __qualname__� NO_FALLBACK�DRY_RUN�NONE� r r �$/opt/gsutil/gslib/utils/shim_util.pyr % s r c @ s e Zd ZdZdZdS )�RepeatFlagTyper � N)r r r �LIST�DICTr r r r r + s r z(^decryption_key([1-9]$|[1-9][0-9]$|100$))�cp�mv�rsyncZsetmeta)Zls�rewrite�stat�cat�compose)r" r Zrm� retention)z cache-controlzcontent-dispositionzcontent-encoding� content-md5zcontent-languagez content-typez custom-time)zx-goog-generation-matchzx-goog-if-generation-matchzx-goog-metageneration-matchz x-goog-if-metageneration-match�AWS_ACCESS_KEY_ID�AWS_SECRET_ACCESS_KEYZ%CLOUDSDK_STORAGE_GS_XML_ACCESS_KEY_IDZ)CLOUDSDK_STORAGE_GS_XML_SECRET_ACCESS_KEYZ-CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE)�aws_access_key_id�aws_secret_access_key�gs_access_key_id�gs_secret_access_key�use_client_certificateZCLOUDSDK_PROXY_ADDRESSZCLOUDSDK_PROXY_TYPEZCLOUDSDK_PROXY_PORTZCLOUDSDK_PROXY_USERNAMEZCLOUDSDK_PROXY_PASSWORDZCLOUDSDK_PROXY_RDNSZCLOUDSDK_CORE_HTTP_TIMEOUTZ"CLOUDSDK_CORE_CUSTOM_CA_CERTS_FILEZ!CLOUDSDK_STORAGE_BASE_RETRY_DELAYZ CLOUDSDK_STORAGE_MAX_RETRIES) �proxy� proxy_type� proxy_port� proxy_user� proxy_pass� proxy_rdns�http_socket_timeout�ca_certificates_file�max_retry_delay� num_retriesZ CLOUDSDK_STORAGE_CHECK_HASHESZCLOUDSDK_CORE_PROJECTZ%CLOUDSDK_CORE_DISABLE_USAGE_REPORTINGZ CLOUDSDK_STORAGE_USE_MAGICFILEZ4CLOUDSDK_STORAGE_PARALLEL_COMPOSITE_UPLOAD_THRESHOLDZ$CLOUDSDK_STORAGE_RESUMABLE_THRESHOLDZ&CLOUDSDK_STORAGE_RSYNC_LIST_CHUNK_SIZE)� check_hashes�default_project_id�disable_analytics_prompt� use_magicfile�#parallel_composite_upload_threshold�resumable_threshold�rsync_buffer_linesZCLOUDSDK_AUTH_CLIENT_IDZCLOUDSDK_AUTH_CLIENT_SECRETZCLOUDSDK_AUTH_AUTH_HOSTZCLOUDSDK_AUTH_TOKEN_HOST)� client_id� client_secretZprovider_authorization_uri�provider_token_uri)� Credentials�Boto�GSUtil�OAuth2Zstet_binary_pathZstet_config_pathFc C s� | � � }|tv r:| td�d� }|�d�s4d| }qL|}n|tv rH|}nd}|dur�|rt|tv sh|dkrldS d| S d�||�S dD ]>}|�|�r�| t|�d� }|r�d| S d �||� S q�dS ) a� Returns the gcloud storage flag for the given gsutil header. Args: raw_header_key: The header key. header_value: The header value unset: If True, the equivalent clear/remove flag is returned instead of the setter flag. This only applies to setmeta. Returns: A string representing the equivalent gcloud storage flag and value, if translation is possible, else returns None. Examples: >> get_flag_from_header('Cache-Control', 'val') --cache-control=val >> get_flag_from_header('x-goog-meta-foo', 'val') --update-custom-metadata=foo=val >> get_flag_from_header('x-goog-meta-foo', 'val', unset=True) --remove-custom-metadata=foo zx-goog-Nzif-r$ z--clear-z--{}={})z x-goog-meta-z x-amz-meta-z--remove-custom-metadata=z --update-custom-metadata={}={})�lower�PRECONDITIONS_HEADERS�len� startswith�DATA_TRANSFER_HEADERS�format)�raw_header_key� header_value�unset�lowercase_header_keyZproviderless_flagZ flag_name� header_prefixZ metadata_keyr r r �get_flag_from_header� s0 � rO c C s t jr dS dS )Nz^^^^�^�r � IS_WINDOWSr r r r �get_format_flag_caret� s rS c C s t jr dS dS )Nz^\^n� rQ r r r r �get_format_flag_newline� s rU c @ s e Zd Zddd�ZdS )�GcloudStorageFlagNFc C s || _ || _|| _dS )a Initializes GcloudStorageFlag. Args: gcloud_flag (str|dict): The name of the gcloud flag or a dictionary for when the gcloud flag depends on a gsutil value. gsutil "--pap off" -> gcloud "--no-public-access-prevention" repeat_type (RepeatFlagType|None): Gsutil sometimes handles list and dictionary inputs by accepting a flag multiple times. support_output_translation (bool): If True, this flag in gcloud storage supports printing gsutil formatted output. N)� gcloud_flag� repeat_type�supports_output_translation)�selfrW rX rY r r r �__init__� s zGcloudStorageFlag.__init__)NF)r r r r[ r r r r rV � s �rV c @ s e Zd ZdZddd�ZdS )�GcloudStorageMapzEMapping to translate gsutil command to its gcloud storage equivalent.Fc C s || _ || _|| _dS )a- Intalizes GcloudStorageMap. Args: gcloud_command (dict|str): The corresponding name of the command to be called in gcloud. If this command supports sub-commands, then this field must be a dict of sub-command-name:GcloudStorageMap pairs. flag_map (dict): A dict of str to GcloudStorageFlag. Mapping of gsutil flags to their equivalent gcloud storage flag names. supports_output_translation (bool): Indicates if the corresponding gcloud storage command supports the printing gsutil formatted output. N)�gcloud_command�flag_maprY )rZ r] r^ rY r r r r[ � s zGcloudStorageMap.__init__N)F)r r r �__doc__r[ r r r r r\ � s �r\ c C s t j�| dtjrdnd�S )N�binz gcloud.cmd�gcloud)�os�path�joinr rR )� cloudsdk_rootr r r �_get_gcloud_binary_path� s �rf c C s: t j�d�} | r| S t j�d�}|d u r2t�d��t|�S )NZGCLOUD_BINARY_PATHZCLOUDSDK_ROOT_DIRa� Requested to use "gcloud storage" but the gcloud binary path cannot be found. This might happen if you attempt to use gsutil that was not installed via Cloud SDK. You can manually set the `CLOUDSDK_ROOT_DIR` environment variable to point to the google-cloud-sdk installation directory to resolve the issue. Alternatively, you can set `use_gcloud_storage=False` to disable running the command using gcloud storage.)rb �environ�getr � GcloudStorageTranslationErrorrf )�gcloud_binary_pathre r r r �!_get_validated_gcloud_binary_path� s �rk c C sL | � dd�}|rH| � dd�}|r(d| nd}| � ddd�}d�|||�S d S ) Nr@ � gs_json_host� gs_json_port�:� �json_api_version�v1zhttps://{}{}/storage/{}�rh rI )r rl rm �portrp r r r �'_get_gcs_json_endpoint_from_boto_config s �rt c C s< | � dd�}|r8| � dd�}|r(d| nd}d�||�S d S )Nr@ �s3_host�s3_portrn ro z https://{}{}rr )r ru rv rs r r r �!_get_s3_endpoint_from_boto_config s rw c C s| g }t �t�}d}|t| �k �r| | |jvrF|�| | � |d7 }q|j| | }|sb|d7 }q|jr�|| �| |d � |d7 }qt|jt �r�|�|j� |d7 }q| |d |jvr�t d� | | | |d ���|j| |d }|r�|�|� |d7 }q|� � D ]l\}}|jt ju �r,d�|�}n4|jt ju �rPd�dd� |D ��}nt d� |j���|�d � |j|�� �q |S ) Nr r � z.Flag value not in translation map for "{}": {}�,c S s g | ]}d j |�dd�� �qS )�{}={}rn r )rI �split)�.0�sr r r � <listcomp>O � z2_convert_args_to_gcloud_values.<locals>.<listcomp>z'Shim cannot handle repeat flag type: {}rz )� collections� defaultdict�listrF r^ �appendrX � isinstancerW �str� ValueErrorrI �itemsr r rd r ) �args�gcloud_storage_mapZ gcloud_argsZrepeat_flag_data�iZgcloud_flag_objectZtranslated_flag_and_value�valuesZcondensed_flag_valuesr r r � _convert_args_to_gcloud_values$ sR �� �� � r� c @ sz e Zd ZdZdZdd� Zdd� Zdd� Zd d � Zd d d �Z dd� Z d dd�Z d dd�Z dd� Z dd� Zdd� Zdd� ZdS ) �GcloudStorageCommandMixinz�Provides gcloud storage translation functionality. The command.Command class must inherit this class in order to support converting the gsutil command to it's gcloud storage equivalent. Nc C s d | _ d | _d | _d S �N)�"_translated_gcloud_storage_command�_translated_env_variables�_gcloud_has_active_account)rZ r r r r[ b s z"GcloudStorageCommandMixin.__init__c C st t � }| jdu rntj|dddgtjtjdd�}|jrFt�d�|j ���|j � � dk| _| j � d �|j |j �� | jS ) z8Returns True if gcloud has an active account configured.Nr rh �accountzutf-8)�stdout�stderr�encodingzJError occurred while trying to retrieve gcloud's active account. Error: {}ro zHResult for "gcloud config get account" command: STDOUT: {}. STDERR: {})rk r� � subprocess�run�PIPE� returncoder ri rI r� r� �strip�logger�debug)rZ Z gcloud_path�processr r r �gcloud_has_active_accountg s$ ��� �z3GcloudStorageCommandMixin.gcloud_has_active_accountc C s� |d u rt �d�| j���g }t|jt�r8|�|j� njt|jt�r�|j rZt d�| j���|d }| j |dd � dd�\}}| � |||j� |��S t d�| j���|r�|D ]@\}}||j vr�t �d�|���q�|�|� |d kr�|�|� q�t|| |�S ) Nz_Command "{}" cannot be translated to gcloud storage because the translation mapping is missing.zcFlags mapping should not be present at the top-level command if a sub-command is used. Command: {}.r r F)r� � should_update_sub_opts_and_argsz(Incorrect mapping found for "{}" commandz:Command option "{}" cannot be translated to gcloud storagero )r ri rI � command_namer� r] r� �extend�dictr^ r� � ParseSubOpts�_get_gcloud_storage_argsrh r� r� ) rZ �sub_optsZ gsutil_argsr� r� Z sub_command� parsed_args�option�valuer r r r� { sR �� �� � �� �� �z2GcloudStorageCommandMixin._get_gcloud_storage_argsc C s� g }ddd�}| j dkr&|�ddg� | j dkr:|�d� | jrJ|�d � | jr`|�d | j � | jrv|�d | j � tjr�|�d tj � | jd ko�t � dd�dko�t � dd�dk }| j s�|s�d|d< d|d< ||fS )a� Translates gsutil's top level flags. Gsutil specifies the headers (-h) and boto config (-o) as top level flags as well, but we handle those separately. Returns: A tuple. The first item is a list of top level flags that can be appended to the gcloud storage command. The second item is a dict of environment variables that can be set for the gcloud storage command execution. Z gsutil_shim�True)Z CLOUDSDK_METRICS_ENVIRONMENTZ#CLOUDSDK_STORAGE_RUN_BY_GSUTIL_SHIM� z --verbosityr� � z --log-httpz--no-user-output-enabledz--billing-project=z--trace-token=z --impersonate-service-account=r rB �parallel_process_count�1Zthread_process_countZ CLOUDSDK_STORAGE_THREAD_COUNTZ CLOUDSDK_STORAGE_PROCESS_COUNT) r� r� r� � quiet_mode� user_project� trace_tokenr �IMPERSONATE_SERVICE_ACCOUNTr� r rh �parallel_operations)rZ �top_level_flags� env_variablesZshould_use_rsync_overrider r r �_translate_top_level_flags� s4 � � � z4GcloudStorageCommandMixin._translate_top_level_flagsFc C s� g }|dur|n| j }g }|�� D ]�\}}|�� }|dkr<q"t|||d�} | jtv rd| r�|�| � n | jtv r�|tv r�|�| � | s"| j � d� ||�� |�d� ||�� q"|r�|�dd� |� � |S )ak Translates gsutil headers to equivalent gcloud storage flags. Args: headers (dict|None): If absent, extracts headers from command instance. unset (bool): Yield metadata clear flags instead of setter flags. Returns: List[str]: Translated flags for gcloud. Raises: GcloudStorageTranslationError: Could not translate flag. Nzx-goog-api-version)rL zzHeader {}:{} cannot be translated to a gcloud storage equivalent flag. It is being treated as an arbitrary request header.rz z--additional-headers=ry ) �headersr� rD rO r� � COMMANDS_SUPPORTING_ALL_HEADERSr� �$PRECONDITONS_ONLY_SUPPORTED_COMMANDSrE r� �warnrI rd ) rZ r� rL �flagsZheaders_to_translate�additional_headersrJ rK rM �flagr r r �_translate_headers� s0 � �z,GcloudStorageCommandMixin._translate_headersc C sb g }i }t t�}|r ||d< tt�}|r0||d< g }t�� D �] \}}|�� D ]�\}} |dkrx| jtv rx|�d| � qNt�|�r�| jtv r�|�| � qN|dkr�| jt v r�|�d| � qN|t v r�| j � d� ||�� qN|dkr�| s�d |d < qN|d v �rt�� �s| j �d � ||�� qNt�|i ��|d �} | d urN| || < qNq<|�rZ|�dd�|� � ||fS )a Translates boto config options to gcloud storage properties. Returns: A tuple where first element is a list of flags and the second element is a dict representing the env variables that can be set to set the gcloud storage properties. Z'CLOUDSDK_API_ENDPOINT_OVERRIDES_STORAGEZ CLOUDSDK_STORAGE_S3_ENDPOINT_URL�encryption_keyz--encryption-key=�content_languagez--content-language=zNThe boto config field {}:{} cannot be translated to gcloud storage equivalent.�https_validate_certificatesTZ$CLOUDSDK_AUTH_DISABLE_SSL_VALIDATION)r) r* zxThe boto config field {}:{} skipped translation to the gcloud storage equivalent as it would have been unused in gsutil.Nz--decryption-keys=ry )rt r rw r� r� � ENCRYPTION_SUPPORTED_COMMANDSr� �DECRYPTION_KEY_REGEX�matchr� �'_REQUIRED_BOTO_CONFIG_NOT_YET_SUPPORTEDr� �errorrI r � UsingGsHmacr� �_BOTO_CONFIG_MAPrh rd ) rZ r� Zenv_varsZgcs_json_endpointZ s3_endpointZdecryption_keys� section_name�section�keyr� �env_varr r r �_translate_boto_config� sR � � � � � � z0GcloudStorageCommandMixin._translate_boto_configc C s | � | j| j|p| j�S )a� Translates the gsutil command flags to gcloud storage flags. It uses the command_spec.gcloud_storage_map field that provides the translation mapping for all the flags. Args: gcloud_storage_map (GcloudStorageMap|None): Command surface may pass a custom translation map instead of using the default class constant. Useful for when translations change based on conditional logic. Returns: A list of all the options and arguments that can be used with the equivalent gcloud storage command. Raises: GcloudStorageTranslationError: If a flag or command cannot be translated. ValueError: If there is any issue with the mapping provided by GcloudStorageMap. )r� r� r� r� )rZ r� r r r �get_gcloud_storage_args* s �z1GcloudStorageCommandMixin.get_gcloud_storage_argsc C sV |r | j jn| j j}|d�d�|��� |rR|d� |�� D ]\}}|d||� q<d S )NzGcloud Storage Command: {}� z)Environment variables for Gcloud Storage:z%s=%s)r� �infor� rI rd r� )rZ r] r� r Z logger_func�k�vr r r �"_print_gcloud_storage_command_infoA s z<GcloudStorageCommandMixin._print_gcloud_storage_command_infoc C sN | � � \}}| �� }| �� \}}|�|� t� }|g| | | | }||fS r� )r� r� r� �updaterk ) rZ r� r� r� Z header_flagsZflags_from_botoZenv_vars_from_botorj �gcloud_storage_commandr r r �._get_full_gcloud_storage_execution_informationL s ��zHGcloudStorageCommandMixin._get_full_gcloud_storage_execution_informationc C sp | j dks| j dkrdS t�ddd�}ztt�ddd��}W n2 tyn t�d�d � d d � tD �����Y n0 |�rlz�| � | � � �\}}| � � s�d |d < |tj kr�| j||dd� nftj�d�s�t�d�t� ���nFt�� r�tj| jjvr�td�| j ���n | �||� || _|| _W dS W nL tj�yj } z0|tjk�rHt�|��| j � d|� W Y d}~n d}~0 0 dS )a\ Translates the gsutil command to gcloud storage equivalent. The translated commands get stored at self._translated_gcloud_storage_command. This command also translate the boto config, which gets stored as a dict at self._translated_env_variables Returns: True if the command was successfully translated, else False. �version�testFrB �use_gcloud_storage�hidden_shim_moder zYInvalid option specified for GSUtil:hidden_shim_mode config setting. Should be one of: {}z | c S s g | ] }|j �qS r )r� )r| �xr r r r~ s r zVGcloudStorageCommandMixin.translate_to_gcloud_storage_if_requested.<locals>.<listcomp>r� Z!CLOUDSDK_AUTH_DISABLE_CREDENTIALST)r �(CLOUDSDK_CORE_PASS_CREDENTIALS_TO_GSUTILz�Requested to use "gcloud storage" but gsutil is not using the same credentials as gcloud. You can make gsutil use the same credentials by running: {} config set pass_credentials_to_gsutil Truez�Requested to use "gcloud storage" with Cloud Storage XML API HMAC credentials but the "{}" command can only be used with the Cloud Storage JSON API.zYCannot translate gsutil command to gcloud storage. Going to run gsutil command. Error: %sN) r� r �getboolr rh r� r r rI rd r� r� r� r r� rb rg ri rk r r� r �XML� command_spec�gs_api_supportr� r� r r� r� )rZ r� r� r� r� �er r r �(translate_to_gcloud_storage_if_requestedY sj � �� � � �� ���� �zBGcloudStorageCommandMixin.translate_to_gcloud_storage_if_requestedc C s t j�� }|�| j� |S r� )rb rg �copyr� r� )rZ Zsubprocess_envsr r r �'_get_shim_command_environment_variables� s zAGcloudStorageCommandMixin._get_shim_command_environment_variablesc C s t j| j| �� d�}|jS )N)�env)r� r� r� r� r� )rZ r� r r r �run_gcloud_storage� s �z,GcloudStorageCommandMixin.run_gcloud_storage)NF)N)F)r r r r_ r� r[ r� r� r� r� r� r� r� r� r� r� r� r r r r r� Y s +' '6 � Dr� )F)0r_ � __future__r r r r r� �enumrb �rer� �botor �gslibr �gslib.cs_api_mapr �gslib.exceptionr � gslib.utilsr r r �Enumr r �compiler� � frozensetr� r� r� rH rE r� r� rO rS rU �objectrV r\ rf rk rt rw r� r� r r r r �<module> s� �� � ����1� : 5