FTS3DB

Frontend to FTS3 MySQL DB. Written using sqlalchemy

class DIRAC.DataManagementSystem.DB.FTS3DB.FTS3DB(pool_size=15, url=None, parentLogger=None)

Bases: object

class RequestDB

db holding requests

__init__(pool_size=15, url=None, parentLogger=None)

c’tor

Parameters:
  • self – self reference

  • pool_size – size of the connection pool to the DB

cancelNonExistingJob(operationID, ftsGUID)

Cancel an FTS3Job with the associated FTS3Files. This is to be used when the job is not found on the server when monitoring.

The status of the job and files will be ‘Canceled’. The error is specifying that the job is not found. The ftsGUID of the file is released as to be able to pick it up again

Parameters:
  • operationID – guess

  • ftsGUID – guess

Returns:

S_OK() if successful, S_ERROR otherwise

createTables()

create tables

deleteFinalOperations(limit=20, deleteDelay=180)

deletes operation in final state that are older than given time

Parameters:
  • limit (int) – number of operations to treat

  • deleteDelay (int) – age of the lastUpdate in days

Returns:

S_OK/S_ERROR with number of deleted operations

getActiveJobs(limit=20, lastMonitor=None, jobAssignmentTag='Assigned')
Get the FTSJobs that are not in a final state, and are not assigned for monitoring

or has its operation being treated

By assigning the job to the DB:
  • it cannot be monitored by another agent

  • the operation to which it belongs cannot be treated

Parameters:
  • limit – max number of Jobs to retrieve

  • lastMonitor – jobs monitored earlier than the given date

  • jobAssignmentTag – if not None, block the Job for other queries, and use it as a prefix for the value in the operation table

Returns:

list of FTS3Jobs

getNonFinishedOperations(limit=20, operationAssignmentTag='Assigned')

Get all the non assigned FTS3Operations that are not yet finished, so either Active or Processed. An operation won’t be picked if it is already assigned, or one of its job is.

Parameters:
  • limit – max number of operations to retrieve

  • operationAssignmentTag – if not None, block the operations for other queries, and use it as a prefix for the value in the operation table

Returns:

list of Operations

getOperation(operationID)

read request

This does not set the assignment flag

Parameters:

operationID – ID of the FTS3Operation

getOperationsFromRMSOpID(rmsOpID)

Returns the FTS3Operations matching a given RMS OperationID

This does not set the assignment flag

Parameters:

rmsOpID – ID of the RMS Operation

kickStuckJobs(limit=20, kickDelay=2)
finds jobs that have not been updated for more than a given

time but are still assigned and resets the assignment

Parameters:
  • limit (int) – number of jobs to treat

  • kickDelay (int) – age of the lastUpdate in hours

Returns:

S_OK/S_ERROR with number of kicked jobs

kickStuckOperations(limit=20, kickDelay=2)
finds operations that have not been updated for more than a given

time but are still assigned and resets the assignment

Parameters:
  • limit (int) – number of operations to treat

  • kickDelay (int) – age of the lastUpdate in hours

Returns:

S_OK/S_ERROR with number of kicked operations

persistOperation(operation)
update or insert request into db

Also release the assignment tag

Parameters:

operation – FTS3Operation instance

updateFileStatus(fileStatusDict, ftsGUID=None)
Update the file ftsStatus and error

The update is only done if the file is not in a final state (To avoid bringing back to life a file by consuming MQ a posteriori)

TODO: maybe it should query first the status and filter the rows I want to update !

Parameters:
  • fileStatusDict – { fileID : { status , error, ftsGUID } }

  • ftsGUID – If specified, only update the rows where the ftsGUID matches this value. This avoids two jobs handling the same file one after another to step on each other foot. Note that for the moment it is an optional parameter, but it may turn mandatory soon.

updateJobStatus(jobStatusDict)
Update the job Status and error

The update is only done if the job is not in a final state The assignment flag is released

Parameters:

jobStatusDict – { jobID : { status , error, completeness } }