apluslms_file_transfer.client

Submodules

apluslms_file_transfer.client.action

apluslms_file_transfer.client.action.publish(publish_url, init_headers, pid_file)[source]

Publish uploaded files into use

Parameters:
  • publish_url (str) – the url sent to the server to publish files
  • init_headers (dict) – the initial headers in the request being sent
  • pid_file (str) – the local file path that stores the process id of the file deployment process
apluslms_file_transfer.client.action.upload(get_files_url, upload_url, init_headers, target_dir, pid_file)[source]

Select and then upload files to the server

Parameters:
  • get_files_url (str) – the url sent to the server to get the updated files
  • upload_url (str) – the url sent to the server to upload files
  • init_headers (dict) – the initial headers in the request being sent
  • target_dir (str) – the path of the local directory to upload
  • pid_file (str) – the local file path that stores the process id of the file deployment process

apluslms_file_transfer.client.fileinfo

apluslms_file_transfer.client.fileinfo.get_files_manifest_in_folder(directory)[source]

Get the manifest of files in a folder

Parameters:directory (str) – the path of the directory
Returns:a nested dict {rel_file_name: {“mtime”:, “checksum”:}}
Return type:dict
apluslms_file_transfer.client.fileinfo.get_files_to_upload(url, headers, target_dir)[source]

Send request to the server to get the collection of files to upload

Parameters:
  • url (str) – the url posted to the server
  • headers (dict) – the headers in the posted request
  • target_dir (str) – the directory path to upload
Returns:

  • files_upload (list) - a list of uploaded files (tuple (file_path, file_size))
  • pid (str) - a unique id of this file deployment process
apluslms_file_transfer.client.fileinfo.get_manifest(file)[source]

Get the manifest (modification time, checksum) of a file

Parameters:file (str) – path of the file
Returns:the dict of the file manifest (key:’mtime’, ‘checksum’)
Return type:dict

apluslms_file_transfer.client.upload_utils

apluslms_file_transfer.client.upload_utils.upload_fbuffer_by_chunk(buffer, whether_last_file, upload_url, headers, data, file_index)[source]

Upload a BytesIO buffer of a file by chunk. The function is used to upload big files.

Parameters:
  • object buffer (BytesIO) – buffer to upload
  • whether_last_file (bool) – whether the file is the last file to upload in this file employment process
  • upload_url (str) – the url for uploading files
  • headers (dict) – the headers in the request
  • data (dict) – the data in the request
  • file_index (int) – the index of the file in the (big) file list
apluslms_file_transfer.client.upload_utils.upload_files_by_tar(file_list, last_file, basedir, buff_size_threshold, upload_url, headers, data)[source]

Compress a list of files and then upload. The function is used to upload small files.

If the buffer of the compression file smaller than buff_size_threshold, then it is upload. Otherwise the file list will be divided as two subsets. For each subset repeat the above process
Parameters:
  • list file_list (list) – a list of uploaded files (tuple(file_path, file_size))
  • last_file (str) – the path of the last file in the complete file_list
  • basedir (str) – the base directory of the relative file path
  • buff_size_threshold (int|float) – the threshold of buffer size to determine division action
  • upload_url (str) – the url for uploading files
  • headers (dict) – headers of requests
  • data (dict) – data of requests
apluslms_file_transfer.client.upload_utils.upload_files_to_server(files_and_sizes, basedir, upload_url, request_data)[source]

Upload a collection of files to the server, using different uploading methods based on the fie size:

  1. the files bigger than 50MB are compressed one by one,
    and the smaller files are collected to fill a quota (50MB) and then compressed
  2. the compression file smaller than 4MB is posted directly, otherwise posted by chunks
Parameters:
  • files_and_sizes (list) – a list of files to upload (tuple(file_path, file_size))
  • basedir – the base directory of the relative file path
  • upload_url – the url for uploading files
  • request_data (dict) – the data in the request

apluslms_file_transfer.client.utils

apluslms_file_transfer.client.utils.examine_env_var()[source]
apluslms_file_transfer.client.utils.iter_read_chunks(buffer, chunk_size=4194304)[source]

An iterator of read_in_chunks function.

Parameters:
  • buffer (BytesIO) – a BytesIO object
  • | int chunk_size (float) – the chunk size of each read
apluslms_file_transfer.client.utils.read_in_chunks(buffer, chunk_size=4194304.0)[source]

Read a buffer in chunks.

Parameters:
  • buffer (BytesIO) – a BytesIO object
  • | int chunk_size (float) – the chunk size of each read
apluslms_file_transfer.client.utils.store_process_id(process_id, file)[source]

Store the id of this process.

Parameters:
  • process_id (str) – the id of this file employment process
  • file (str) – the file path to store the id
apluslms_file_transfer.client.utils.tar_files_buffer(files, basedir)[source]

Generate a buffer of a compression file.

Parameters:
  • files (list) – a list of files to upload (tuple(file_path, file_size))
  • basedir (str) – the base directory of the relative file path
Returns:

a BytesIO object
apluslms_file_transfer.client.utils.validate_directory(directory, file_type)[source]

Check whether the static directory and the index.html file exist.

Parameters:
  • directory (str) – the path of a static directory
  • file_type (str) – the file type to upload
Returns:

a dictionary {“target_dir”: the directory where the files are uploaded}
Return type:

dict