pythonbasictools package¶

Subpackages¶

Submodules¶

pythonbasictools.collections_tools module¶

pythonbasictools.collections_tools.dict_of_lists_to_list_of_dicts(dict_of_lists: Dict) → List[dict]¶

Convert a dictionary of lists to a list of dictionaries. The keys of the dictionary are used as keys in the dictionaries. The values of the dictionary are inserted into the dictionaries at the same index as the list.

Example:

>>> dict_of_lists = {
...    "a": [1, 2, 3],
...    "b": [9, 8, 7],
...    "c": [True, False, True],
...}
>>> list_of_dicts = dict_of_lists_to_list_of_dicts(dict_of_lists)
>>> print(list_of_dicts)
[{'a': 1, 'b': 9, 'c': True}, {'a': 2, 'b': 8, 'c': False}, {'a': 3, 'b': 7, 'c': True}]

Note:

If the lists in the dictionary have different lengths, the last directories will have less keys than the first ones.

Parameters:

dict_of_lists (dict) – The dictionary of lists to convert.

Returns:

The list of dictionaries.

Return type:

List[dict]

pythonbasictools.collections_tools.dict_of_lists_to_product_list_of_dicts(dict_of_lists: Dict) → List[dict]¶

Convert a dictionary of lists to a list of dictionaries as the Cartesian product of the lists.

Example:

>>> dict_of_lists = {
...    "a": [1, 2],
...    "b": [9, 8],
...}
>>> list_of_dicts = dict_of_lists_to_product_dict_of_lists(dict_of_lists)
>>> print(list_of_dicts)
[{'a': 1, 'b': 9}, {'a': 1, 'b': 8}, {'a': 2, 'b': 9}, {'a': 2, 'b': 8}]

Parameters:

dict_of_lists (dict) – The dictionary of lists to convert.

Returns:

The list of dictionaries.

Return type:

List[dict]

pythonbasictools.collections_tools.list_insert_replace_at(__list: List, idx: int, value: Any, default: Any = None) → List¶

Insert a value at a specific index. If there is already a value at this index, replace it.

Parameters:

• __list (List) – The list to modify.

• idx (int) – The index to insert the value.

• value (Any) – The value to insert.

• default (Any) – The default value to insert if the index is out of bounds.

Returns:

The new list.

Return type:

List

pythonbasictools.collections_tools.list_of_dicts_to_dict_of_lists(list_of_dicts: List[dict], default: Any = None) → dict¶

Convert a list of dictionaries to a dictionary of lists. The keys of the dictionaries are used as keys in the dictionary of lists. The values of the dictionaries are inserted into the lists at the same index as the dictionary. This means that every list in the dictionary of lists will have the same length as the list of dictionaries.

Parameters:

• list_of_dicts (List[dict]) – The list of dictionaries to convert.

• default (Any) – The default value to insert if a key is not present in a dictionary.

Returns:

The dictionary of lists.

Return type:

dict

pythonbasictools.collections_tools.maybe_unpack_singleton_dict(x: dict | Any, default: Any = None) → Any¶

Accept a dict or any other type. If x is a dict with one key and value, the singleton is unpacked. Otherwise, x is returned without being changed.

Parameters:

• x (Union[dict, Any]) – The value to unpack to maybe unpack.

• default (Any) – The default value to return if the dictionary is empty.

Returns:

The unpacked value or the original value.

Return type:

Any

pythonbasictools.collections_tools.ravel_dict(d: dict, key_sep: str = '.') → dict¶

Ravel a dictionary into a single level dictionary.

Parameters:

• d (dict) – The dictionary to ravel.

• key_sep (str) – The separator to use between keys in the raveled dictionary.

Returns:

The raveled dictionary.

Return type:

dict

pythonbasictools.collections_tools.sequence_get(sequence: Sequence, idx: int, default: Any = None) → Any¶

Get an item from a sequence at a specific index. If the index is out of bounds, return a default value.

Parameters:

• sequence (Sequence) – The sequence to get the item from.

• idx (int) – The index to get the item from.

• default (Any) – The default value to return if the index is out of bounds.

Returns:

The item at the index or the default value.

Return type:

Any

pythonbasictools.collections_tools.unpack_singleton_dict(x: dict, default: Any = None) → Any | None¶

Unpack a dictionary with a single key and value. If the dict has more than one key, a ValueError is raised. If the dict is empty, the default value is returned.

Parameters:

• x (dict) – The dictionary to unpack.

• default (Any) – The default value to return if the dictionary is empty.

Returns:

The value of the single key in the dictionary.

Return type:

Any

pythonbasictools.decorators module¶

pythonbasictools.decorators.log_func(_func=None, *, box_length=50, box_char='-', logging_func=<function info>)¶

Log the function call and the time it took to execute.

Parameters:

• _func (Callable) – The function to decorate.

• box_length (int) – The length of the box to use to log the function call.

• box_char (str) – The character to use to log the function call.

• logging_func (Callable) – The logging function to use to log the function call.

Returns:

The decorated function.

pythonbasictools.decorators.try_func_n_times(_func=None, *, n: int = 32, delay: float = 0.1)¶

pythonbasictools.device module¶

class pythonbasictools.device.DeepLib(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)¶

Bases: Enum

Enumerate the different deep learning libraries.

Null = -1¶

Pytorch = 0¶

SkLearn = 2¶

Tensorflow = 1¶

pythonbasictools.device.log_device_setup(deepLib: DeepLib = DeepLib.Null, level=20)¶

Log the device setup.

Parameters:

• deepLib (DeepLib) – The deep learning library to use.

• level (int) – The logging level to use.

Returns:

None

pythonbasictools.device.log_pytorch_device_setup(level=20)¶

Log the Pytorch device setup.

Parameters:

level (int) – The logging level to use.

Returns:

None

pythonbasictools.device.log_sklearn_device_setup(level=20)¶

Log the SkLearn device setup.

Parameters:

level (int) – The logging level to use.

Returns:

None

pythonbasictools.device.log_tensorflow_device_setup(level=20)¶

Log the Tensorflow device setup.

Parameters:

level (int) – The logging level to use.

Returns:

None

pythonbasictools.device.set_tf_loglevel(level=20)¶

Set the Tensorflow log level.

Parameters:

level (int) – The logging level to use.

Returns:

None

pythonbasictools.google_drive module¶

class pythonbasictools.google_drive.GoogleDriveDownloader(file_id: str, dest_path: str, *, chunk_size: int | None = 32768, skip_existing: bool = True, verbose: bool = True)¶

Bases: object

Object used to download a file from Google Drive.

Exemple:

>>> gdd = GoogleDriveDownloader(file_id='[file id]', dest_path='data.zip')
>>> gdd.download()

DOWNLOAD_URL = 'https://docs.google.com/uc?export=download'¶

__init__(file_id: str, dest_path: str, *, chunk_size: int | None = 32768, skip_existing: bool = True, verbose: bool = True)¶

Create a new GoogleDriveDownloader object.

Parameters:

• file_id – The ID of the file to download. This is the part of the URL after the “/d/” and before the “/view” in the URL of the file in Google Drive.

• dest_path – The path to save the downloaded file to.

• chunk_size – The chunk size to use when downloading the file.

• skip_existing – If True, the file will not be downloaded if it already exists at the destination path.

• verbose – If True, the download progress will be printed to the console.

download(session_params: Dict[str, Any] | None = None)¶

Download the file.

Returns:

None

static get_confirm_token(response)¶

save_response_content(response)¶

pythonbasictools.hash_tools module¶

pythonbasictools.hash_tools.hash_dict(d: dict) → str¶

Hash a dictionary using the SHA256 algorithm.

Parameters:

d (dict) – The dictionary to hash.

Returns:

The hash of the dictionary.

Return type:

str

pythonbasictools.lock module¶

class pythonbasictools.lock.FileLock(lock_path: str = './lock.lck', wait_time: float = 0.1, process_name: str = 'process')¶

Bases: object

Class used to lock a process across multiple ones. This can be very useful when you want to use a file across multiple different processes or different applications.

Attributes:

• lock_path (str): The path to the lock file.

• wait_time (float): The time to wait between each check of the lock file in seconds.

• process_name (str): The name of the process.

Methods:

• acquire(): Acquire the lock.

• release(): Release the lock.

• __enter__(): Acquire the lock.

• __exit__(): Release the lock.

Example:

>>> with FileLock():
>>>         # Do something

>>> lock = FileLock()
>>> lock.acquire()
>>> # Do something
>>> lock.release()

>>> data_path = "data.txt"
>>> lock = FileLock()
>>> with lock:
>>>         with open(data_path, "r") as f:
>>>                 data = f.read()
>>>         # Do something with data
>>>         with open(data_path, "w") as f:
>>>                 f.write(data)

__init__(lock_path: str = './lock.lck', wait_time: float = 0.1, process_name: str = 'process')¶

Constructor of the class.

Parameters:

• lock_path (str) – The path to the lock file.

• wait_time (float) – The time to wait between each check of the lock file in seconds.

• process_name (str) – The name of the process.

acquire()¶

Acquire the lock.

Returns:

None

release()¶

Release the lock.

Returns:

None

pythonbasictools.logging_tools module¶

pythonbasictools.logging_tools.logs_file_setup(file: str, level=20, root_logs_dir: str = './', add_stdout: bool = True)¶

Set up the logs file.

Parameters:

• file (str) – The logs file name.

• level (int) – The level of the logs.

• root_logs_dir (str) – The root directory of the logs.

• add_stdout (bool) – Whether to add the stdout handler.

Returns:

The logs file path.

Return type:

str

pythonbasictools.multiprocessing_tools module¶

pythonbasictools.multiprocessing_tools.apply_func_main_process(func, iterable_of_args: List[Tuple], iterable_of_kwargs: List[Dict] | None = None, tqdm_options: Dict | None = None, **kwargs)¶

Apply a function to a list of arguments in the main process.

Parameters:

• func (Callable) – The function to apply.

• iterable_of_args (List[Tuple]) – The list of arguments to apply the function to.

• iterable_of_kwargs (Optional[List[Dict]]) – The list of keyword arguments to apply the function to.

• tqdm_options (Optional[Dict]) – The options for tqdm.tqdm.

• kwargs – The additional arguments.

Keyword Arguments:

• desc (str) – The description of the function to apply. See tqdm.tqdm for more details.

• unit (str) – The unit of the function to apply. See tqdm.tqdm for more details.

• verbose (bool) – Whether to print the progress bar or not. Default to True.

• callbacks (List[Callable]) – The list of callbacks to call after each iteration.

Returns:

The list of results.

Raises:

ValueError – If the length of iterable_of_args and iterable_of_kwargs are not the same.

Example:

>>> from pythonbasictools.multiprocessing import apply_func_main_process
>>> def func(x, y):
...     return x + y
>>> apply_func_main_process(func, [(1, 2), (3, 4)])
>>> [3, 7]

pythonbasictools.multiprocessing_tools.apply_func_multiprocess(func, iterable_of_args: List[Tuple], iterable_of_kwargs: List[Dict] | None = None, nb_workers=-2, tqdm_options: Dict | None = None, **kwargs)¶

Apply a function to a list of arguments in parallel.

Parameters:

• func (Callable) – The function to apply.

• iterable_of_args (List[Tuple]) – The list of arguments to apply the function to.

• iterable_of_kwargs (Optional[List[Dict]]) – The list of keyword arguments to apply the function to.

• nb_workers (int) – The number of workers to use. If -1, use all the logical available CPUs. If -2, use all the available CPUs. If 0, use the main process. If greater than 0, use the specified number of workers. Default to -2.

• tqdm_options (Optional[Dict]) – The options for tqdm.tqdm.

• kwargs – The additional arguments.

Keyword Arguments:

• desc (str) – The description of the function to apply. See tqdm.tqdm for more details.

• unit (str) – The unit of the function to apply. See tqdm.tqdm for more details.

• verbose (bool) – Whether to print the progress bar or not. Default to True.

• callbacks (List[Callable]) – The list of callbacks to call after each iteration. See multiprocessing.Pool.apply_async for more details.

Returns:

The list of results.

Raises:

• ValueError – If the length of iterable_of_args and iterable_of_kwargs are not the same.

• ValueError – If the number of workers is less than -2.

Example:

>>> from pythonbasictools.multiprocessing import apply_func_multiprocess
>>> def func(a, b):
...     return a + b
>>> apply_func_multiprocess(func, [(1, 2), (3, 4), (5, 6)])
>>> [3, 7, 11]

pythonbasictools.multiprocessing_tools.multiprocess_logger_init()¶

pythonbasictools.multiprocessing_tools.worker_init(q)¶

Module contents¶