clowder.util package

Submodules

clowder.util.clowder_utils module

Clowder command line utilities

clowder.util.clowder_utils.existing_branch_groups(groups, branch, is_remote)

Checks if given branch exists in any project

Parameters:
  • groups (list[Group]) – Groups to check
  • branch (str) – Branch to check for
  • is_remote (bool) – Check for remote branch
Returns:

True, if at least one branch exists
Return type:

bool
clowder.util.clowder_utils.existing_branch_projects(projects, branch, is_remote)

Checks if given branch exists in any project

Parameters:
  • projects (list[Project]) – Projects to check
  • branch (str) – Branch to check for
  • is_remote (bool) – Check for remote branch
Returns:

True, if at least one branch exists
Return type:

bool
clowder.util.clowder_utils.filter_groups(groups, names)

Filter groups based on given group names

Parameters:
  • groups (list[Group]) – Groups to filter
  • names (list[str]) – Group names to match against
Returns:

List of groups in groups matching given names
Return type:

list[Group]
clowder.util.clowder_utils.filter_projects(groups, group_names=None, project_names=None)

Filter projects based on given project or group names

Parameters:
  • groups (list[Group]) – Groups to filter
  • group_names (list[str]) – Group names to match against
  • project_names (list[str]) – Project names to match against
Returns:

List of projects in groups matching given names
Return type:

list[Project]
clowder.util.clowder_utils.get_clowder_yaml_import_path(import_name)

Return path to imported clowder.yaml file

Parameters:import_name (str) – Name of imported clowder.yaml
Returns:Path to imported clowder.yaml file
Return type:str
clowder.util.clowder_utils.get_saved_version_names()

Return list of all saved versions

Returns:List of all saved version names
Return type:list[str]

Create symlink pointing to clowder.yaml file

Parameters:version (Optional[str]) – Version name of clowder.yaml to link
Raises:ClowderExit –
clowder.util.clowder_utils.options_help_message(options, message)

Help message for groups option

Parameters:
  • options (list[str]) – List of options
  • message (str) – Help message
Returns:

Formatted options help message
Return type:

str
clowder.util.clowder_utils.print_parallel_groups_output(groups, skip)

Print output for parallel group command

Parameters:
  • groups (list[Group]) – Groups to print output for
  • skip (list[str]) – Project names to skip
clowder.util.clowder_utils.print_parallel_projects_output(projects, skip)

Print output for parallel project command

Parameters:
  • projects (list[Project]) – Projects to print output for
  • skip (list[str]) – Project names to skip
clowder.util.clowder_utils.run_group_command(group, skip, command, *args, **kwargs)

Run group command and print output

Parameters:
  • group (Group) – Group to run command for
  • skip (list[str]) – Project names to skip
  • command (str) – Name of method to invoke
  • args – List of arguments to pass to method invocation
  • kwargs – Dict of arguments to pass to method invocation
clowder.util.clowder_utils.run_project_command(project, skip, command, *args, **kwargs)

Run project command and print output

Parameters:
  • project (Praject) – Project to run command for
  • skip (list[str]) – Project names to skip
  • command (str) – Name of method to invoke
  • args – List of arguments to pass to method invocation
  • kwargs – Dict of arguments to pass to method invocation
clowder.util.clowder_utils.validate_groups(groups)

Validate status of all projects for specified groups

Parameters:groups (list[Group]) – Groups to validate
Raises:ClowderExit –
clowder.util.clowder_utils.validate_print_output(clowder, group_names, **kwargs)

Validate projects/groups and print output

clowder.util.clowder_utils.validate_print_output(clowder, group_names, project_names=None, skip=[])
Parameters:
  • clowder (ClowderController) – ClowderController instance
  • group_names (list[str]) – Group names to validate/print
Keyword Arguments:
 
  • project_names (list[str]) – Project names to validate/print
  • skip (list[str]) – Project names to skip
clowder.util.clowder_utils.validate_projects(projects)

Validate status of all projects

Parameters:projects (list[Project]) – Projects to validate
Raises:ClowderExit –
clowder.util.clowder_utils.validate_projects_exist(clowder)

Validate existence status of all projects for specified groups

Parameters:clowder (ClowderController) – ClowderController instance
Raises:ClowderExit –

clowder.util.connectivity module

Network connectivity

clowder.util.connectivity.is_offline(host='8.8.8.8', port=53, timeout=3)

Returns True if offline, False otherwise

Service: domain (DNS/TCP)

Note

Implementation source https://stackoverflow.com/a/33117579

Parameters:
  • host (str) – Host to check. Default is 8.8.8.8 (google-public-dns-a.google.com)
  • port (int) – Port number. Default is 53/tcp
  • timeout (int) – Seconds to wait until timeout
Returns:

True, if offline
Return type:

bool
Raises:

ClowderExit –

clowder.util.connectivity.network_connection_required(func)

If no network connection, print offline message and exit

clowder.util.decorators module

clowder.yaml validation

clowder.util.decorators.clowder_required(func)

If no clowder repo, print clowder not found message and exit

clowder.util.decorators.valid_clowder_yaml_required(func)

If clowder.yaml is invalid, print invalid yaml message and exit

clowder.util.execute module

Subprocess execution utilities

clowder.util.execute.execute_command(command, path, **kwargs)

Execute command via thread

clowder.util.execute.execute_command(command, path, shell=True, env=None, print_output=True)
Parameters:
  • command (str or list[str]) – Command to run
  • path (str) – Path to set as cwd
Keyword Arguments:
 
  • shell (bool) – Whether to execute subprocess as shell
  • env (dict) – Enviroment to set as env
  • print_output (bool) – Whether to print output
Returns:

Command return code
Return type:

int
Raises:

ClowderError –

clowder.util.execute.execute_forall_command(command, path, forall_env, print_output)

Execute forall command with additional environment variables and display continuous output

Parameters:
  • command (str or list[str]) – Command to run
  • path (str) – Path to set as cwd
  • forall_env (dict) – Enviroment to set as env
  • print_output (bool) – Whether to print output
Returns:

Command return code
Return type:

int
clowder.util.execute.execute_subprocess_command(command, path, **kwargs)

Execute subprocess command

clowder.util.execute.execute_subprocess_command(command, path, shell=True, env=None, stdout=None, stderr=None)
Parameters:
  • command (str or list[str]) – Command to run
  • path (str) – Path to set as cwd
Keyword Arguments:
 
  • shell (bool) – Whether to execute subprocess as shell
  • env (dict) – Enviroment to set as env
  • stdout (int) – Value to set as stdout
  • stderr (int) – Value to set as stderr
Returns:

Subprocess return code
Return type:

int
Raises:

ClowderError –

clowder.util.execute.subprocess_exit_handler(process)

terminate subprocess

Parameters:process (Popen) – Popen subprocess instance

clowder.util.file_system module

File system utilities

Force symlink creation

Parameters:
  • file1 (str) – File to create symlink pointing to
  • file2 (str) – Symlink location
Raises:

ClowderExit –

clowder.util.file_system.remove_directory(path)

Remove directory at path

Parameters:path (str) – Path to remove
Raises:ClowderExit –

Returns target path if input is a symlink, otherwise returns original path

Parameters:path (str) – Path of file or symlink
Returns:Target path if input is a symlink, otherwise original path
Return type:str

clowder.util.formatting module

String formatting utilities

clowder.util.formatting.clowder_command(cmd)

Return formatted clowder command name

Parameters:cmd (str) – Clowder command name
Returns:Formatted clowder command name
Return type:str
clowder.util.formatting.command(cmd)

Return formatted command name

Parameters:cmd (str or list[str]) – Clowder command name
Returns:Formatted clowder command name
Return type:str
clowder.util.formatting.command_failed_error(cmd)

Format error message for failed command

Parameters:cmd (str or list[str]) – Clowder command name
Returns:Formatted clowder command name
Return type:str
clowder.util.formatting.depth_error(depth, yml)

Return formatted error string for invalid depth

Parameters:
  • depth (int) – Git clone depth. 0 indicates full clone, otherwise must be a positive integer
  • yml (str) – Path to yaml file
Returns:

Formatted depth error
Return type:

str
clowder.util.formatting.empty_yaml_error(yml)

Return formatted error string for empty clowder.yaml

Parameters:yml (str) – Path to yaml file
Returns:Formatted empty yaml error
Return type:str
clowder.util.formatting.error(err)

Return error message for generic error

Parameters:err (Exception) – Generic error
Returns:Formatted generic error
Return type:str
clowder.util.formatting.file_exists_error(path)

Format error message for already existing file

Parameters:path (str) – File path name
Returns:Formatted file exists error
Return type:str
clowder.util.formatting.fork_string(name)

Return formatted fork name

Parameters:name (str) – Fork name
Returns:Formatted fork name
Return type:str
clowder.util.formatting.get_path(path)

Return formatted path

Parameters:path (str) – Path name
Returns:Formatted path name
Return type:str
clowder.util.formatting.group_name(name)

Print formatted group name

Parameters:name (str) – Group name
Returns:Formatted group name
Return type:str
clowder.util.formatting.invalid_protocol_error(protocol, yml)

Return formatted error string for incorrect protocol

Parameters:
  • protocol (str) – Git protocol
  • yml (str) – Path to yaml file
Returns:

Formatted invalid ref error
clowder.util.formatting.invalid_ref_error(ref, yml)

Return formatted error string for incorrect ref

Parameters:
  • ref (str) – Git reference
  • yml (str) – Path to yaml file
Returns:

Formatted invalid ref error
clowder.util.formatting.invalid_yaml_error()

Return error message for invalid clowder.yaml

Returns:Formatted yaml error
Return type:str
clowder.util.formatting.missing_entries_error(name, yml)

Return formatted error string for invalid entry in collection

Parameters:
  • name (str) – Entry name
  • yml (str) – Path to yaml file
Returns:

Formatted missing entries error
Return type:

str
clowder.util.formatting.missing_entry_error(entry, name, yml)

Return formatted error string for missing entry in dictionary

Parameters:
  • entry (str) – Name of entry to check
  • name (str) – Name of entry to print if missing
  • yml (str) – Path to yaml file
Returns:

Formatted missing entry in dictionary error
Return type:

str
clowder.util.formatting.missing_imported_yaml_error(path, yml)

Return formatted error string for missing imported clowder.yaml

Parameters:
  • path (str) – File path
  • yml (str) – Path to yaml file
Returns:

Formatted missing YAML error
Return type:

str
clowder.util.formatting.missing_yaml_error()

Format error message for missing clowder.yaml

Returns:Formatted missing YAML error
Return type:str
clowder.util.formatting.offline_error()

Return error message for no internet connection

Returns:Offline error message
Return type:str
clowder.util.formatting.open_file_error(path)

Format error message for failing to open file

Parameters:path (str) – File path
Returns:Formatted file error
Return type:str
clowder.util.formatting.parallel_exception_error(path, *args)

Return formatted error string for parallel error

Parameters:
  • path (str) – Clowder file path
  • args – Method arguments
Returns:

Formatted parallel exception error
Return type:

str
clowder.util.formatting.recursive_import_error(depth)

Format error message for too many recursive imports

Parameters:depth (int) – Git clone depth. 0 indicates full clone, otherwise must be a positive integer
Returns:Formatted too many recursive imports error
Return type:str
clowder.util.formatting.ref_string(ref)

Return formatted ref name

Parameters:ref – Git reference
Returns:Formatted ref name
Return type:str
clowder.util.formatting.remote_already_exists_error(remote_name, remote_url, actual_url)

Format error message when remote already exists with different url

Parameters:
  • remote_name (str) – Remote name
  • remote_url (str) – Remote URL
  • actual_url (str) – Actual URL
Returns:

Formatted remote exists error
Return type:

str
clowder.util.formatting.remote_name_error(fork, project, remote)

Return formatted error string for fork with same remote as project

Parameters:
  • fork (str) – Fork name
  • project (str) – Project name
  • remote (str) – Remote name
Returns:

Formatted duplicate remote fork name error
Return type:

str
clowder.util.formatting.remote_string(remote)

Return formatted remote name

Parameters:remote (str) – Remote branch name
Returns:Formmatted remote branch name
Return type:str
clowder.util.formatting.remove_prefix(text, prefix)

Remove prefix from a string

Parameters:
  • text (str) – String to remove prefix from
  • prefix (str) – Prefix to remove
Returns:

Text with prefix removed
Return type:

str
clowder.util.formatting.save_default_error(name)

Format error message for trying to save ‘default’ version

Parameters:name (str) – Version name
Returns:Formatted default version error
Return type:str
clowder.util.formatting.save_file_error(path)

Format error message for failing to save file

Parameters:path (str) – File path
Returns:Formatted save failure error
Return type:str
clowder.util.formatting.save_version(version_name, yml)

Format message for saving version

Parameters:
  • version_name (str) – Clowder version name
  • yml (str) – Path to yaml file
Returns:

Formatted version name
Return type:

str
clowder.util.formatting.save_version_exists_error(version_name, yml)

Format error message previous existing saved version

Parameters:
  • version_name (str) – Version name
  • yml (str) – Path to yaml file
Returns:

Formatted version exists error
Return type:

str
clowder.util.formatting.skip_project_message()

Return skip project message

Returns:Skip project message
Return type:str
clowder.util.formatting.type_error(name, yml, type_name)

Return formatted error string for value with wrong type

Parameters:
  • name (str) – Value name
  • yml (str) – Path to yaml file
  • type_name (str) – Type name
Returns:

Formatted incorrect type error
Return type:

str
clowder.util.formatting.unknown_entry_error(name, collection, yml)

Return formatted error string for unknown entry in collection

Parameters:
  • name (str) – Entry name
  • collection (dict) – Entries collection
  • yml (str) – Path to yaml file
Returns:

Formatted unknown entry error
Return type:

str
clowder.util.formatting.version(version_name)

Return formatted string for clowder.yaml version

Parameters:version_name (str) – Clowder version name
Returns:Formatted clowder version name
Return type:str
clowder.util.formatting.yaml_file(yml)

Return formatted string for clowder.yaml file

Parameters:yml (str) – Path to yaml file
Returns:Formatted YAML string
Return type:str
clowder.util.formatting.yaml_path(yml)

Returns formatted yaml path

Parameters:yml (str) – Path to yaml file
Returns:Formatted YAML path
Return type:str
clowder.util.formatting.yaml_string(yaml_output)

Return yaml string from python data structures

Parameters:yaml_output (dict) – YAML python object
Returns:YAML as a string
Return type:str
Raises:ClowderExit –

clowder.util.parallel_commands module

Clowder parallel commands

clowder.util.parallel_commands.async_callback(val)

Increment async progress bar

Parameters:val – Dummy parameter to satisfy callback interface
clowder.util.parallel_commands.forall_parallel(commands, skip, ignore_errors, projects)

Runs command or script for projects in parallel

Parameters:
  • commands (list[str]) – Command to run
  • skip (list[str]) – Project names to skip
  • ignore_errors (bool) – Whether to exit if command returns a non-zero exit code
  • projects (list[Project]) – Projects to run command for
clowder.util.parallel_commands.herd_parallel(clowder, group_names, **kwargs)

Clone projects or update latest from upstream in parallel

clowder.util.parallel_commands.herd_parallel(clowder, group_names, branch=None, tag=None, depth=0, rebase=False, project_names=None, skip=[], protocol=None)
Parameters:
  • clowder (ClowderController) – ClowderController instance
  • group_names (list[str]) – Group names to herd
Keyword Arguments:
 
  • branch (str) – Branch to attempt to herd
  • tag (str) – Tag to attempt to herd
  • depth (int) – Git clone depth. 0 indicates full clone, otherwise must be a positive integer
  • protocol (str) – Git protocol (‘ssh’ or ‘https’)
  • rebase (bool) – Whether to use rebase instead of pulling latest changes
  • project_names (list[str]) – Project names to herd
  • skip (list[str]) – Project names to skip
clowder.util.parallel_commands.herd_project(project, branch, tag, depth, rebase, protocol)

Herd command wrapper function for multiprocessing Pool execution

Parameters:
  • project (Project) – Project instance
  • branch (str) – Branch to attempt to herd
  • tag (str) – Tag to attempt to herd
  • depth (int) – Git clone depth. 0 indicates full clone, otherwise must be a positive integer
  • rebase (bool) – Whether to use rebase instead of pulling latest changes
  • protocol (str) – Git protocol (‘ssh’ or ‘https’)
clowder.util.parallel_commands.pool_handler(count)

Pool handler for finishing parallel jobs

Parameters:count (int) – Total count of projects in progress bar
Raises:ClowderExit –
clowder.util.parallel_commands.reset_parallel(clowder, group_names, **kwargs)

Reset project branches to upstream or checkout tag/sha as detached HEAD in parallel

clowder.util.parallel_commands.reset_parallel(clowder, group_names, timestamp_project=None, project_names=None, skip=[])
Parameters:
  • clowder (ClowderController) – ClowderController instance
  • group_names (list[str]) – Group names to reset
Keyword Arguments:
 
  • timestamp_project (str) – Reference project to checkout commit timestamps of other projects relative to
  • project_names (list[str]) – Project names to reset
  • skip (list[str]) – Project names to skip
clowder.util.parallel_commands.reset_project(project, timestamp)

Reset command wrapper function for multiprocessing Pool execution

Parameters:
  • project (Project) – Project instance
  • timestamp (str) – If not None, reset to commit at timestamp, or closest previous commit
clowder.util.parallel_commands.run_project(project, commands, ignore_errors)

Run command wrapper function for multiprocessing Pool execution

Parameters:
  • project (Project) – Project instance
  • commands (list[str]) – Commands to run
  • ignore_errors (bool) – Whether to exit if command returns a non-zero exit code
clowder.util.parallel_commands.sync_parallel(projects, protocol, rebase=False)

Sync projects in parallel

Parameters:
  • projects (list[Project]) – Projects to sync
  • protocol (str) – Git protocol, ‘ssh’ or ‘https’
  • rebase (Optional[bool]) – Whether to use rebase instead of pulling latest changes
clowder.util.parallel_commands.sync_project(project, protocol, rebase)

Sync command wrapper function for multiprocessing Pool execution

Parameters:
  • project (Project) – Project instance
  • protocol (str) – Git protocol, ‘ssh’ or ‘https’
  • rebase (bool) – Whether to use rebase instead of pulling latest changes
clowder.util.parallel_commands.worker_init()

Process pool terminator

Note

Implementation source https://stackoverflow.com/a/45259908

clowder.util.progress module

Progress bar wrapper class

class clowder.util.progress.Progress

Bases: object

Class wrapping progress bar

close()

Close progress bar

complete()

Complete progress bar

start(count)

Start progress bar

Parameters:count (int) – Initial count
update()

Update progress bar

Module contents