datacatalog package¶
Subpackages¶
- datacatalog.compositions package
- datacatalog.definitions package
- datacatalog.filetypes package
- Submodules
- datacatalog.filetypes.anytype module
- datacatalog.filetypes.filetype module
- datacatalog.filetypes.infer module
- datacatalog.filetypes.listing module
- datacatalog.filetypes.mime module
- datacatalog.filetypes.rules module
- datacatalog.filetypes.ruleset module
- datacatalog.filetypes.schemas module
- datacatalog.filetypes.unknown module
- datacatalog.filetypes.validate module
- datacatalog.formats package
- Subpackages
- Submodules
- datacatalog.formats.classify module
- datacatalog.formats.common module
- datacatalog.formats.converter module
- datacatalog.formats.runner module
- datacatalog.formats.schemas module
- datacatalog.hashable package
- datacatalog.identifiers package
- Subpackages
- Submodules
- datacatalog.identifiers.datacatalog_uuid module
- datacatalog.identifiers.identifier module
- datacatalog.identifiers.interestinganimal module
- datacatalog.identifiers.schemas module
- datacatalog.jsonschemas package
- Subpackages
- Submodules
- datacatalog.jsonschemas.encoders module
- datacatalog.jsonschemas.exceptions module
- datacatalog.jsonschemas.formatchecker module
- datacatalog.jsonschemas.schema module
- datacatalog.jsonschemas.schemas module
- datacatalog.jsonschemas.validate module
- datacatalog.jsonschemas.version module
- datacatalog.linkages package
- datacatalog.linkedstores package
- Subpackages
- datacatalog.linkedstores.annotation package
- datacatalog.linkedstores.annotations package
- datacatalog.linkedstores.association package
- datacatalog.linkedstores.basestore package
- Submodules
- datacatalog.linkedstores.basestore.admin module
- datacatalog.linkedstores.basestore.agaveclient module
- datacatalog.linkedstores.basestore.diff module
- datacatalog.linkedstores.basestore.documentschema module
- datacatalog.linkedstores.basestore.exceptions module
- datacatalog.linkedstores.basestore.extensible module
- datacatalog.linkedstores.basestore.heritableschema module
- datacatalog.linkedstores.basestore.linkmanager module
- datacatalog.linkedstores.basestore.managedfields module
- datacatalog.linkedstores.basestore.merge module
- datacatalog.linkedstores.basestore.mongomerge module
- datacatalog.linkedstores.basestore.ratelimit module
- datacatalog.linkedstores.basestore.record module
- datacatalog.linkedstores.basestore.schemas module
- datacatalog.linkedstores.basestore.softdelete module
- datacatalog.linkedstores.basestore.store module
- datacatalog.linkedstores.basestore.strategies module
- datacatalog.linkedstores.challenge_problem package
- Subpackages
- datacatalog.linkedstores.challenge_problem.mappings package
- Subpackages
- datacatalog.linkedstores.challenge_problem.mappings.gdrive package
- Submodules
- datacatalog.linkedstores.challenge_problem.mappings.gdrive.build_list module
- datacatalog.linkedstores.challenge_problem.mappings.gdrive.errors module
- datacatalog.linkedstores.challenge_problem.mappings.gdrive.gdrive module
- datacatalog.linkedstores.challenge_problem.mappings.gdrive.google_utils module
- datacatalog.linkedstores.challenge_problem.mappings.gdrive.quickstart_service module
- datacatalog.linkedstores.challenge_problem.mappings.gdrive package
- Subpackages
- datacatalog.linkedstores.challenge_problem.mappings package
- Submodules
- datacatalog.linkedstores.challenge_problem.schemas module
- datacatalog.linkedstores.challenge_problem.store module
- Subpackages
- datacatalog.linkedstores.experiment package
- Subpackages
- datacatalog.linkedstores.experiment.mappings package
- Subpackages
- datacatalog.linkedstores.experiment.mappings.gdrive package
- Submodules
- datacatalog.linkedstores.experiment.mappings.gdrive.build_list module
- datacatalog.linkedstores.experiment.mappings.gdrive.errors module
- datacatalog.linkedstores.experiment.mappings.gdrive.gdrive module
- datacatalog.linkedstores.experiment.mappings.gdrive.google_utils module
- datacatalog.linkedstores.experiment.mappings.gdrive.quickstart_service module
- datacatalog.linkedstores.experiment.mappings.gdrive package
- Subpackages
- datacatalog.linkedstores.experiment.mappings package
- Submodules
- datacatalog.linkedstores.experiment.schemas module
- datacatalog.linkedstores.experiment.store module
- Subpackages
- datacatalog.linkedstores.experiment_design package
- Subpackages
- datacatalog.linkedstores.experiment_design.mappings package
- Subpackages
- datacatalog.linkedstores.experiment_design.mappings.gdrive package
- Submodules
- datacatalog.linkedstores.experiment_design.mappings.gdrive.build_list module
- datacatalog.linkedstores.experiment_design.mappings.gdrive.errors module
- datacatalog.linkedstores.experiment_design.mappings.gdrive.gdrive module
- datacatalog.linkedstores.experiment_design.mappings.gdrive.google_utils module
- datacatalog.linkedstores.experiment_design.mappings.gdrive.quickstart_service module
- datacatalog.linkedstores.experiment_design.mappings.gdrive package
- Subpackages
- datacatalog.linkedstores.experiment_design.mappings package
- Submodules
- datacatalog.linkedstores.experiment_design.schemas module
- datacatalog.linkedstores.experiment_design.store module
- Subpackages
- datacatalog.linkedstores.file package
- datacatalog.linkedstores.fixity package
- datacatalog.linkedstores.measurement package
- datacatalog.linkedstores.pipeline package
- datacatalog.linkedstores.pipelinejob package
- Submodules
- datacatalog.linkedstores.pipelinejob.exceptions module
- datacatalog.linkedstores.pipelinejob.fsm module
- datacatalog.linkedstores.pipelinejob.graphfsm module
- datacatalog.linkedstores.pipelinejob.job module
- datacatalog.linkedstores.pipelinejob.mappings module
- datacatalog.linkedstores.pipelinejob.schema module
- datacatalog.linkedstores.pipelinejob.schemas module
- datacatalog.linkedstores.pipelinejob.store module
- datacatalog.linkedstores.pipelinejob.utils module
- datacatalog.linkedstores.process package
- datacatalog.linkedstores.product package
- datacatalog.linkedstores.reference package
- datacatalog.linkedstores.sample package
- datacatalog.linkedstores.structured_request package
- Subpackages
- datacatalog.logger package
- datacatalog.managers package
- Subpackages
- datacatalog.managers.annotations package
- datacatalog.managers.catalog package
- datacatalog.managers.pipelinejobs package
- Subpackages
- Submodules
- datacatalog.managers.pipelinejobs.config module
- datacatalog.managers.pipelinejobs.exceptions module
- datacatalog.managers.pipelinejobs.indexer module
- datacatalog.managers.pipelinejobs.indexrequest module
- datacatalog.managers.pipelinejobs.instanced module
- datacatalog.managers.pipelinejobs.jobmanager module
- datacatalog.managers.pipelinejobs.reactor module
- datacatalog.managers.pipelinejobs.store module
- datacatalog.managers.sampleset package
- Submodules
- datacatalog.managers.common module
- Subpackages
- datacatalog.mongo package
- datacatalog.pathmappings package
- datacatalog.settings package
- Submodules
- datacatalog.settings.callbacks module
- datacatalog.settings.crypt module
- datacatalog.settings.debug module
- datacatalog.settings.dockerfile module
- datacatalog.settings.helpers module
- datacatalog.settings.identifiers module
- datacatalog.settings.jsonschema module
- datacatalog.settings.mongo module
- datacatalog.settings.organization module
- datacatalog.slots package
- datacatalog.stores package
- datacatalog.tenancy package
- datacatalog.tokens package
- datacatalog.views package
- Subpackages
- datacatalog.views.challenge_problem_view package
- datacatalog.views.experiment_design_view package
- datacatalog.views.experiment_flow_incomplete_jobs_view package
- datacatalog.views.experiment_flow_jobs_view package
- datacatalog.views.experiment_pending_flow_jobs_view package
- datacatalog.views.experiment_rnaseq_jobs_view package
- datacatalog.views.experiment_view package
- datacatalog.views.file_view package
- datacatalog.views.jobs_view package
- datacatalog.views.measurement_view package
- datacatalog.views.metadatajob_view package
- datacatalog.views.pipelinejob_view package
- datacatalog.views.pipelinejobs_table package
- datacatalog.views.product_view package
- datacatalog.views.sample_view package
- datacatalog.views.science_table package
- datacatalog.views.science_view package
- datacatalog.views.upload_view package
- Submodules
- datacatalog.views.aggregations module
- Subpackages
Submodules¶
datacatalog.agavehelpers module¶
-
class
datacatalog.agavehelpers.
AgaveHelper
(client, storage_system='data-sd2e-community')[source]¶ Bases:
object
Uses an active API client to provide various utility functions
-
exists
(path, storage_system=None)[source]¶ Check if a path exists on an Agave storage resource
Parameters: Raises: AgaveHelperError
– The function has failed due an API errorReturns: Whether the path exists or not
Return type:
-
isdir
(path, storage_system=None)[source]¶ Check if a path on an Agave storage resource is a directory
Parameters: Raises: AgaveHelperError
– The function has failed due an API errorReturns: Whether the path is a directory or not
Return type:
-
isfile
(path, storage_system=None)[source]¶ Check if a path on an Agave storage resource is a file
Parameters: Raises: AgaveHelperError
– The function has failed due an API errorReturns: Whether the path is a file or not
Return type:
-
listdir
(path, recurse, storage_system=None, directories=True)[source]¶ Get the contents of a directory on an Agave storage resource
Parameters: Returns: Directory contents as a list of strings
Return type:
-
listdir_agave_native
(path, recurse, storage_system=None, directories=True, current_listing=[])[source]¶
-
mapped_posix_path
(path, storage_system=None)[source]¶ Resolve the absolute POSIX path for an Agave directory
Parameters: Returns: The path as a string
Return type:
-
mkdir
(dirName, systemId, basePath='/', sync=False, timeOut=60)[source]¶ Creates a directory dirName on a storage system at basePath
Like mkdir -p this is imdepotent. It will create the child path tree so long as paths are specified correctly, but will do nothing if all directories are already in place.
-
datacatalog.dicthelpers module¶
-
datacatalog.dicthelpers.
data_merge
(left, right, setkeys=['child_of'])[source]¶ Merge two mappings objects together, combining overlapping Mappings, and favoring right-values left: The left Mapping object. right: The right (favored) Mapping object. NOTE: This is not commutative (merge(a,b) != merge(b,a)).
-
datacatalog.dicthelpers.
data_merge_diff
(a, b, filters=('_id', 'uuid', 'properties', 'measurements_ids', 'measurements', 'files', 'samples'))[source]¶
-
datacatalog.dicthelpers.
dict_compare
(a, b)[source]¶ Lexically compare values of all primitives in a pair of dicts
-
datacatalog.dicthelpers.
dict_merge
(dct, merge_dct, add_keys=True)[source]¶ Recursive dict merge. Inspired by :meth:
dict.update()
, instead of updating only top-level keys, dict_merge recurses down into dicts nested to an arbitrary depth, updating keys. Themerge_dct
is merged intodct
. This version will return a copy of the dictionary and leave the original arguments untouched. The optional argumentadd_keys
, determines whether keys which are present inmerge_dict
but notdct
should be included in the new dict. :param dct: :type dct: dict :param merge_dct: dct merged into dct :type merge_dct: dict :param add_keys: whether to add new keys :type add_keys: boolReturns: updated dict Return type: dict
-
datacatalog.dicthelpers.
filter_dict
(target_dict, keys_to_filter)[source]¶ Filters key(s) from top level of a dict :param target_dict: the dictionary to filter :type target_dict: dict :param keys_to_filter: set of keys to filter :type keys_to_filter: list, tuple, str
Returns: A filtered copy of target_dict
-
datacatalog.dicthelpers.
is_primitive
(pyobj)[source]¶ Determine if pyobj is one of (what other languages would deem) a primitive
datacatalog.extensible module¶
datacatalog.githelpers module¶
-
datacatalog.githelpers.
get_remote_uri
(repo='/home/docs/checkouts/readthedocs.org/user_builds/python-datacatalog/checkouts/gh-pages/docs')[source]¶ Gets the remote origin for the current directory
Parameters: repo (str, optional) – The directory to inspect for a git reflog Returns: an SSH or HTTP git repository URL Return type: str
-
datacatalog.githelpers.
get_sha1
(repo='/home/docs/checkouts/readthedocs.org/user_builds/python-datacatalog/checkouts/gh-pages/docs')[source]¶ Get SHA-1 hash of given git repository
Inspects the specified directory, assumed to be a git repository, and returns the SHA1 hash from its HEAD-revision. The output is equivalent to calling git rev-parse HEAD.
Parameters: repo (str, optional) – The directory to inspect for a git reflog Returns: A hexadecimal string representing the SHA1 hash Return type: str
-
datacatalog.githelpers.
get_sha1_short
(repo='/home/docs/checkouts/readthedocs.org/user_builds/python-datacatalog/checkouts/gh-pages/docs')[source]¶ Return abbreviated SHA1 hash for the specified git repository
Returns the initial seven characters of the current repository’s most recent git commit.
Parameters: repo (str, optional) – The directory to inspect for a git reflog Returns: A hexadecimal string representing the SHA1 hash Return type: str
datacatalog.utils module¶
-
datacatalog.utils.
current_time
()[source]¶ Current UTC time :returns: A
datetime
object rounded to millisecond precision
-
datacatalog.utils.
dynamic_import
(module, package=None)[source]¶ Dynamically import a module by name at runtime
Parameters: Returns: The imported module
Return type:
-
datacatalog.utils.
import_submodules
(module, package=None)[source]¶ Dynamically discover and import submodules at runtime
-
datacatalog.utils.
safen_path
(file_path, no_unicode=False, no_spaces=False, url_quote=False, no_equals=False)[source]¶ Returns a safened version of a path
Trailing whitespace is removed, Unicode characters (sorry!) are transformed to ASCII equivalents, equals are replaced with a dash, and whitespaces are replaced with a dash character.
-
datacatalog.utils.
time_stamp
(dt=None, rounded=False)[source]¶ Get time in seconds :param dt: Optional datetime object. [current_time()] :type dt: datetime :param rounded: Whether to round respose to nearest int :type rounded: bool
Returns: Time expressed as a float
(orint
)
-
datacatalog.utils.
validate_file_to_schema
(file_path, schema_file='/schemas/default.jsonschema', permissive=False)[source]¶ Validate a JSON document against a specified JSON schema
Args: file_path (str): path to the file to validate schema_file (str): path to the requisite JSON schema file [/schemas/default.jsonschema] permissive (bool): swallow validation errors and return only boolean [False]
Returns: Boolean value - Error handling:
- Raises validation exceptions if ‘permissive’ is False.