Contents:

Modules:

pythonbasictools package

pythonbasictools package¶

Submodules¶

pythonbasictools.decorators module¶

pythonbasictools.decorators.log_func(_func=None, *, box_length=50, box_char='-', logging_func=<function info>)¶

Log the function call and the time it took to execute.

Parameters:

_func (Callable) – The function to decorate.
box_length (int) – The length of the box to use to log the function call.
box_char (str) – The character to use to log the function call.
logging_func (Callable) – The logging function to use to log the function call.

Returns:

The decorated function.

pythonbasictools.device module¶

class pythonbasictools.device.DeepLib(value)¶

Bases: Enum

Enumerate the different deep learning libraries.

Null = -1¶

Pytorch = 0¶

SkLearn = 2¶

Tensorflow = 1¶

pythonbasictools.device.log_device_setup(deepLib: DeepLib = DeepLib.Null, level=20)¶

Log the device setup.

Parameters:

deepLib (DeepLib) – The deep learning library to use.
level (int) – The logging level to use.

Returns:

None

pythonbasictools.device.log_pytorch_device_setup(level=20)¶

Log the Pytorch device setup.

Parameters:: level (int) – The logging level to use.
Returns:: None

pythonbasictools.device.log_sklearn_device_setup(level=20)¶

Log the SkLearn device setup.

Parameters:: level (int) – The logging level to use.
Returns:: None

pythonbasictools.device.log_tensorflow_device_setup(level=20)¶

Log the Tensorflow device setup.

Parameters:: level (int) – The logging level to use.
Returns:: None

pythonbasictools.device.set_tf_loglevel(level=20)¶

Set the Tensorflow log level.

Parameters:: level (int) – The logging level to use.
Returns:: None

pythonbasictools.discord module¶

pythonbasictools.discord.send_files_to_discord(webhook_urls: str, file_paths: str | List[str])¶

Send the logs file to Discord.

Parameters:

webhook_urls (str) – The webhook urls to use to send the files.
file_paths (Union[str, List[str]]) – The file paths to send.

Returns:

The result of the request.

pythonbasictools.discord.send_logs_file_to_discord(webhook_urls: str)¶

Send the logs file to Discord.

Parameters:: webhook_urls (str) – The webhook urls to use to send the logs file.
Returns:: The result of the request.

pythonbasictools.docstring module¶

pythonbasictools.docstring.format_field(field_name: str, field_doc: str) → str¶

pythonbasictools.docstring.get_bases_docs(prop, bases: Type | List[Type] | None = None) → Dict[str, str]¶

Get the docstring of the parent class.

Parameters:

prop (Any) – The object to find the parent class of.
bases (Optional[Union[Type, List[Type]]]) – The list of base classes to inherit the docstring from.

Returns:

A dictionary of the docstring of the parent class. The key is the name of the parent class and the value is the docstring of the parent class.

Return type:

Dict[str, str]

pythonbasictools.docstring.get_field_from_docstring(doc: str, field_name: str) → List[str]¶

Get all field body associated to a field name in a docstring.

Parameters:

doc (str) – The docstring to parse.
field_name (str) – The field name to look for.

Returns:

The list of field body associated to the field name.

Return type:

List[str]

pythonbasictools.docstring.inherit_docstring(_prop=None, *, sep: str = '\n', bases: Type | List[Type] | None = None)¶

Decorator to add the docstring of the parent class to the child class.

Parameters:

_prop – The object to decorate.
sep (str) – The separator to use between the docstring of the parent and the child.
bases (Optional[Union[Type, List[Type]]]) – The list of base classes to inherit the docstring from.

Returns:

The decorated object.

pythonbasictools.docstring.inherit_fields_docstring(_prop=None, *, sep: str = '\n', bases: Type | List[Type] | None = None, fields: List[str] | str | None = None)¶

Inherit the fields of the parents docstrings and add it to the child fields.

Parameters:

_prop (Any) – The object to decorate.
sep (str) – The separator to use between the docstring of the parent and the child.
bases (Optional[Union[Type, List[Type]]]) – The list of base classes to inherit the docstring from.
fields (Optional[Union[List[str], str]]) – The list of fields to inherit. If the fields is a string, it will be converted to a list by splitting the string by the ‘,’ character. If the fields is None, it will be set to all found fields.

Returns:

The decorated object.

Return type:

Any

pythonbasictools.docstring.walk_docstring(doc: str) → Dict[str, List[str]]¶

pythonbasictools.google_drive module¶

class pythonbasictools.google_drive.GoogleDriveDownloader(file_id: str, dest_path: str, *, chunk_size: int | None = 32768, skip_existing: bool = True, verbose: bool = True)¶

Bases: object

Object used to download a file from Google Drive.

Exemple:

>>> gdd = GoogleDriveDownloader(file_id='[file id]', dest_path='data.zip')
>>> gdd.download()

DOWNLOAD_URL = 'https://docs.google.com/uc?export=download'¶

__init__(file_id: str, dest_path: str, *, chunk_size: int | None = 32768, skip_existing: bool = True, verbose: bool = True)¶

Create a new GoogleDriveDownloader object.

Parameters:

file_id – The ID of the file to download. This is the part of the URL after the “/d/” and before the “/view” in the URL of the file in Google Drive.
dest_path – The path to save the downloaded file to.
chunk_size – The chunk size to use when downloading the file.
skip_existing – If True, the file will not be downloaded if it already exists at the destination path.
verbose – If True, the download progress will be printed to the console.

download(session_params: Dict[str, Any] | None = None)¶

Download the file.

Returns:: None

static get_confirm_token(response)¶

save_response_content(response)¶

pythonbasictools.lock module¶

class pythonbasictools.lock.FileLock(lock_path: str = './lock.lck', wait_time: float = 0.1, process_name: str = 'process')¶

Bases: object

Class used to lock a process across multiple ones. This can be very useful when you want to use a file across multiple different processes or different applications.

Attributes:

lock_path (str): The path to the lock file.
wait_time (float): The time to wait between each check of the lock file in seconds.
process_name (str): The name of the process.

Methods:

acquire(): Acquire the lock.
release(): Release the lock.
__enter__(): Acquire the lock.
__exit__(): Release the lock.

Example:

>>> with FileLock():
>>>     # Do something

>>> lock = FileLock()
>>> lock.acquire()
>>> # Do something
>>> lock.release()

>>> data_path = "data.txt"
>>> lock = FileLock()
>>> with lock:
>>>     with open(data_path, "r") as f:
>>>             data = f.read()
>>>     # Do something with data
>>>     with open(data_path, "w") as f:
>>>             f.write(data)

__init__(lock_path: str = './lock.lck', wait_time: float = 0.1, process_name: str = 'process')¶

Constructor of the class.

Parameters:

lock_path (str) – The path to the lock file.
wait_time (float) – The time to wait between each check of the lock file in seconds.
process_name (str) – The name of the process.

acquire()¶

Acquire the lock.

Returns:: None

release()¶

Release the lock.

Returns:: None

pythonbasictools.logging module¶

pythonbasictools.logging.logs_file_setup(file: str, level=20, root_logs_dir: str = './', add_stdout: bool = True)¶

Set up the logs file.

Parameters:

file (str) – The logs file name.
level (int) – The level of the logs.
root_logs_dir (str) – The root directory of the logs.
add_stdout (bool) – Whether to add the stdout handler.

Returns:

The logs file path.

Return type:

str

pythonbasictools.multiprocessing module¶

pythonbasictools.multiprocessing.apply_func_main_process(func, iterable_of_args: List[Tuple], iterable_of_kwargs: List[Dict] | None = None, **kwargs)¶

Apply a function to a list of arguments in the main process.

Parameters:

func (Callable) – The function to apply.
iterable_of_args (List[Tuple]) – The list of arguments to apply the function to.
iterable_of_kwargs (Optional[List[Dict]]) – The list of keyword arguments to apply the function to.
kwargs – The additional arguments.

Keyword Arguments:

desc (str) – The description of the function to apply. See tqdm.tqdm for more details.
unit (str) – The unit of the function to apply. See tqdm.tqdm for more details.
verbose (bool) – Whether to print the progress bar or not. Default to True.

Returns:

The list of results.

Raises:

ValueError – If the length of iterable_of_args and iterable_of_kwargs are not the same.

Example:

>>> from pythonbasictools.multiprocessing import apply_func_main_process
>>> def func(x, y):
...     return x + y
>>> apply_func_main_process(func, [(1, 2), (3, 4)])
>>> [3, 7]

pythonbasictools.multiprocessing.apply_func_multiprocess(func, iterable_of_args: List[Tuple], iterable_of_kwargs: List[Dict] | None = None, nb_workers=-2, **kwargs)¶

Apply a function to a list of arguments in parallel.

Parameters:

func (Callable) – The function to apply.
iterable_of_args (List[Tuple]) – The list of arguments to apply the function to.
iterable_of_kwargs (Optional[List[Dict]]) – The list of keyword arguments to apply the function to.
nb_workers (int) – The number of workers to use. If -1, use all the logical available CPUs. If -2, use all the available CPUs. If 0, use the main process. If greater than 0, use the specified number of workers. Default to -2.
kwargs – The additional arguments.

Keyword Arguments:

desc (str) – The description of the function to apply. See tqdm.tqdm for more details.
unit (str) – The unit of the function to apply. See tqdm.tqdm for more details.
verbose (bool) – Whether to print the progress bar or not. Default to True.

Returns:

The list of results.

Raises:

ValueError – If the length of iterable_of_args and iterable_of_kwargs are not the same.
ValueError – If the number of workers is less than -2.

Example:

>>> from pythonbasictools.multiprocessing import apply_func_multiprocess
>>> def func(a, b):
...     return a + b
>>> apply_func_multiprocess(func, [(1, 2), (3, 4), (5, 6)])
>>> [3, 7, 11]

pythonbasictools.multiprocessing.multiprocess_logger_init()¶

pythonbasictools.multiprocessing.worker_init(q)¶

pythonbasictools.progress_bar module¶

pythonbasictools.progress_bar.printProgressBar(iteration, total, prefix='', suffix='', decimals=0, length=100, fill='█', printEnd='', current_elapse_seconds=None, log_func=<built-in function print>) → str¶

Call in a loop to create terminal progress bar

Parameters:

iteration (int) – Current iteration
total (int) – Total iterations
prefix (str) – Prefix string
suffix (str) – Suffix string
decimals (int) – Positive number of decimals in percent complete
length (int) – Character length of bar
fill (str) – Bar fill character
printEnd (str) – end parameter of the print function
current_elapse_seconds (float) – The current elapsed time in seconds.
log_func (Callable) – The logging function to use to log the function call.

Returns:

The progress bar string.

pythonbasictools.slurm module¶

class pythonbasictools.slurm.SlurmHostServer(value)¶

Bases: Enum

Enum of the different Slurm host server.

BELUGA = 'beluga.computecanada.ca'¶

CEDAR = 'cedar.computecanada.ca'¶

GRAHAM = 'graham.sharcnet.ca'¶

HELIOS = 'helios.calculquebec.ca'¶

pythonbasictools.slurm.generate_slurm_cmd(repository_root: str, credential: Dict[str, str] | None = None, bash_file_to_run: str | None = None, run_count: int = 1, job_to_cancel: str | None = None) → List[str]¶

Generate a command to run on a host.

Parameters:

repository_root (str) – The root directory of the repository.
credential (Optional[Dict[str, str]]) – The credential to use. If provided, must contain the following key: “username”.
bash_file_to_run (str) – The bash file to run.
run_count (int) – The number of times to run the bash file.
job_to_cancel (str) – The job to cancel.

Returns:

The command to run.

Return type:

List[str]

Exemple:

>>> generate_slurm_cmd(
...     repository_root="./GitHub/pythonbasictools",
...     credential={
...         "username": "user",
...         "password": "password"
...     },
...     bash_file_to_run="./GitHub/pythonbasictools/jobs/test.sh",
...     run_count=1,
...     job_to_cancel="123456"
... )

pythonbasictools.slurm.send_slurm_cmd(hostnames, port, username, password, cmd_to_execute)¶

Send a command to a host.

Parameters:

hostnames (str) – The hostname of the host to send the command to.
port (int) – The port to connect to.
username (str) – The username to connect with.
password (str) – The password to connect with.
cmd_to_execute (str) – The command to execute.

Returns:

The output of the command.

Return type:

str

pythonbasictools package¶

Submodules¶

pythonbasictools.decorators module¶

pythonbasictools.device module¶

pythonbasictools.discord module¶

pythonbasictools.docstring module¶

pythonbasictools.google_drive module¶

pythonbasictools.lock module¶

pythonbasictools.logging module¶

pythonbasictools.multiprocessing module¶

pythonbasictools.progress_bar module¶

pythonbasictools.slurm module¶

Module contents¶