datacatalog.linkedstores.pipelinejob package¶
Submodules¶
datacatalog.linkedstores.pipelinejob.exceptions module¶
-
exception
datacatalog.linkedstores.pipelinejob.exceptions.
JobError
[source]¶ Bases:
Exception
A generic job error
-
exception
datacatalog.linkedstores.pipelinejob.exceptions.
JobCreateFailure
[source]¶ Bases:
datacatalog.linkedstores.pipelinejob.exceptions.JobError
Job was not created
-
exception
datacatalog.linkedstores.pipelinejob.exceptions.
JobUpdateFailure
[source]¶ Bases:
datacatalog.linkedstores.pipelinejob.exceptions.JobError
Job was not updated
-
exception
datacatalog.linkedstores.pipelinejob.exceptions.
DuplicateJobError
(error, code=None, details=None, max_wire_version=None)[source]¶ Bases:
pymongo.errors.DuplicateKeyError
Job was a duplicate which is not allowed
-
exception
datacatalog.linkedstores.pipelinejob.exceptions.
UnknownPipeline
[source]¶ Bases:
datacatalog.linkedstores.pipelinejob.exceptions.JobError
The referenced pipeline is not known
-
exception
datacatalog.linkedstores.pipelinejob.exceptions.
UnknownJob
[source]¶ Bases:
datacatalog.linkedstores.pipelinejob.exceptions.JobError
The referenced job is not known
datacatalog.linkedstores.pipelinejob.fsm module¶
-
class
datacatalog.linkedstores.pipelinejob.fsm.
JobStateMachine
(state='CREATED')[source]¶ Bases:
transitions.core.Machine
-
states
= ['CREATED', 'RUNNING', 'FAILED', 'FINISHED', 'INDEXING', 'VALIDATING', 'VALIDATED', 'REJECTED', 'FINALIZED', 'RETIRED', 'RESET']¶
-
transitions
= [{'trigger': 'create', 'source': 'CREATED', 'dest': 'CREATED'}, {'trigger': 'run', 'source': ['CREATED', 'RUNNING'], 'dest': 'RUNNING'}, {'trigger': 'update', 'source': ['CREATED', 'RUNNING', 'VALIDATING', 'INDEXING'], 'dest': '='}, {'trigger': 'resource', 'source': ['CREATED', 'RUNNING'], 'dest': '='}, {'trigger': 'fail', 'source': ['CREATED', 'RUNNING', 'VALIDATING', 'INDEXING'], 'dest': 'FAILED'}, {'trigger': 'finish', 'source': ['RUNNING', 'FINISHED'], 'dest': 'FINISHED'}, {'trigger': 'index', 'source': ['INDEXING', 'FINISHED'], 'dest': 'INDEXING'}, {'trigger': 'indexed', 'source': ['FINISHED', 'INDEXING'], 'dest': 'FINISHED'}, {'trigger': 'validate', 'source': ['FINISHED', 'VALIDATED'], 'dest': 'VALIDATING'}, {'trigger': 'validated', 'source': ['VALIDATED', 'VALIDATING'], 'dest': 'VALIDATED'}, {'trigger': 'reject', 'source': 'VALIDATING', 'dest': 'REJECTED'}, {'trigger': 'finalize', 'source': 'VALIDATED', 'dest': 'FINALIZED'}, {'trigger': 'retire', 'source': ['FAILED', 'FINALIZED', 'FINISHED', 'REJECTED', 'VALIDATING', 'VALIDATED'], 'dest': 'RETIRED'}, {'trigger': 'reset', 'source': '*', 'dest': 'RESET'}, {'trigger': 'ready', 'source': ['RESET'], 'dest': 'CREATED'}]¶
-
datacatalog.linkedstores.pipelinejob.graphfsm module¶
datacatalog.linkedstores.pipelinejob.job module¶
-
class
datacatalog.linkedstores.pipelinejob.job.
HistoryEntry
(entry)[source]¶ Bases:
datacatalog.linkedstores.basestore.extensible.ExtensibleAttrDict
-
class
datacatalog.linkedstores.pipelinejob.job.
PipelineJob
(job_document, agave=None)[source]¶ Bases:
datacatalog.linkedstores.basestore.extensible.ExtensibleAttrDict
,datacatalog.linkedstores.basestore.agaveclient.DocumentAgaveClient
-
exception
datacatalog.linkedstores.pipelinejob.job.
PipelineJobError
[source]¶ Bases:
datacatalog.linkedstores.basestore.exceptions.CatalogError
Error occured within scope of a PipelineJob
datacatalog.linkedstores.pipelinejob.mappings module¶
-
class
datacatalog.linkedstores.pipelinejob.mappings.
AgaveEvents
[source]¶ Bases:
object
Maps between Agave API and Pipeline job events
-
agavejobs
= {'ARCHIVING': 'resource', 'ARCHIVING_FAILED': 'fail', 'ARCHIVING_FINISHED': 'resource', 'CLEANING_UP': 'resource', 'CREATED': 'resource', 'DELETED': 'fail', 'FAILED': 'fail', 'FINISHED': 'finish', 'HEARTBEAT': 'update', 'KILLED': 'update', 'PAUSED': 'resource', 'PENDING': 'resource', 'PERMISSION_GRANT': 'resource', 'PERMISSION_REVOKE': 'resource', 'PROCESSING_INPUTS': 'resource', 'QUEUED': 'resource', 'RUNNING': 'run', 'STAGED': 'resource', 'STAGING_INPUTS': 'resource', 'STAGING_JOB': 'resource', 'STOPPED': 'fail', 'SUBMITTING': 'resource', 'UPDATED': 'updated'}¶
-
datacatalog.linkedstores.pipelinejob.schema module¶
-
class
datacatalog.linkedstores.pipelinejob.schema.
HistoryEventDocument
(inheritance=True, document='pipelinejob_event.json', filters='pipelinejob_event_filters.json', **kwargs)[source]¶ Bases:
datacatalog.linkedstores.basestore.heritableschema.HeritableDocumentSchema
-
class
datacatalog.linkedstores.pipelinejob.schema.
JobDocument
(inheritance=True, **kwargs)[source]¶ Bases:
datacatalog.linkedstores.basestore.heritableschema.HeritableDocumentSchema
datacatalog.linkedstores.pipelinejob.schemas module¶
datacatalog.linkedstores.pipelinejob.store module¶
-
class
datacatalog.linkedstores.pipelinejob.store.
PipelineJobStore
(mongodb, config={}, session=None, agave=None, **kwargs)[source]¶ Bases:
datacatalog.linkedstores.basestore.agaveclient.AgaveClient
,datacatalog.linkedstores.basestore.softdelete.SoftDelete
,datacatalog.linkedstores.basestore.store.LinkedStore
-
LINK_FIELDS
= ['child_of', 'acted_on', 'acted_using', 'generated_by']¶ Fields that should never be indexed
-
NEVER_INDEX_FIELDS
= 'data'¶
-
list_job_archive_path
(job_uuid, recurse=True, directories=False, **kwargs)[source]¶ Returns contents of a job’s archive_path on the job’s archive_system
Parameters: Notes
PipelineJobStore must be initialized with a valid Agave API client
Returns: Agave-canonical absolute filenames in job.archive_path Return type: list
-
-
class
datacatalog.linkedstores.pipelinejob.store.
StoreInterface
(mongodb, config={}, session=None, agave=None, **kwargs)[source]¶ Bases:
datacatalog.linkedstores.pipelinejob.store.PipelineJobStore