datacatalog.linkedstores.pipeline package

Submodules

datacatalog.linkedstores.pipeline.exceptions module

exception datacatalog.linkedstores.pipeline.exceptions.DuplicatePipelineError[source]

Bases: datacatalog.linkedstores.basestore.exceptions.CatalogError

exception datacatalog.linkedstores.pipeline.exceptions.PipelineCreateFailure[source]

Bases: datacatalog.linkedstores.basestore.exceptions.CatalogError

exception datacatalog.linkedstores.pipeline.exceptions.PipelineUpdateFailure[source]

Bases: datacatalog.linkedstores.basestore.exceptions.CatalogError

datacatalog.linkedstores.pipeline.schema module

class datacatalog.linkedstores.pipeline.schema.PipelineDocument(inheritance=True, **kwargs)[source]

Bases: datacatalog.linkedstores.basestore.heritableschema.HeritableDocumentSchema

Defines a compute or ETL pipeline

datacatalog.linkedstores.pipeline.schemas module

datacatalog.linkedstores.pipeline.schemas.get_schemas()[source]

Get JSON schemas for PipelineDocument

Returns:Return the object and document JSON schema that define the store
Return type:dict

datacatalog.linkedstores.pipeline.serializer module

datacatalog.linkedstores.pipeline.serializer.INDENT = 0

Default indentation for JSON rendering

datacatalog.linkedstores.pipeline.serializer.SEPARATORS = (',', ':')

Default tuple of JSON separators

datacatalog.linkedstores.pipeline.serializer.SORT_KEYS = True

Default behavior for sorting keys when rendering to JSON

class datacatalog.linkedstores.pipeline.serializer.SerializedPipeline(component_list)[source]

Bases: object

Helper class for serializing a pipeline definition

Parameters:list – pipeline components expressed as dict objects
COMPONENT_KEYS = {'abaco_actor': (('id', True), ('image', True), ('options', False), ('uuid', False)), 'agave_app': (('id', True), ('modules', False), ('inputs', True), ('outputs', False), ('parameters', True), ('uuid', False)), 'agave_job': (('appId', True), ('id', True), ('uuid', False)), 'deployed_container': (('image', True), ('hostname', True), ('hash', False), ('options', False), ('uuid', False)), 'web_service': (('uri', True), ('identifier', False), ('options', False), ('uuid', False))}

Fields to include in component definitions. Key is named after jsonschema. Tuple is (field, required)

classmethod classify_component(component)[source]

Determine type for a ‘component’ based on whether it contains mandated keys

filter_actor_def(actor_def)[source]

Filter out extraneous keys in Abaco actor definitions

filter_app_job_def(app_job_def)[source]

Filter out extraneous keys in Agave app definitions

classmethod filter_component(component)[source]
to_json()[source]

Renders the pipeline as minified JSON

datacatalog.linkedstores.pipeline.store module

class datacatalog.linkedstores.pipeline.store.PipelineStore(mongodb, config={}, session=None, **kwargs)[source]

Bases: datacatalog.linkedstores.basestore.softdelete.SoftDelete, datacatalog.linkedstores.basestore.store.LinkedStore

Manage storage and retrieval of PipelineDocuments

get_typeduuid(payload, binary=False)[source]

Pipelines-specific method for getting a UUID

Parameters:payload (object) – A list or dict containing the pipeline definition
Returns:A UUID for this Pipeline
Return type:str
class datacatalog.linkedstores.pipeline.store.StoreInterface(mongodb, config={}, session=None, **kwargs)[source]

Bases: datacatalog.linkedstores.pipeline.store.PipelineStore