5.2. RMS objects

5.2.1. Requests

A Request is like a TODO list, each of the task being an Operation. A Request has the following attributes:

  • CreationTime: Time at which the Request was created

  • Error: Error if any at execution

  • JobID: The ID of the job that generated the request. If it comes from another source, it is 0

  • LastUpdate: Last time the request was touched

  • NotBefore: Time before which no execution will be attempted (automatic increments in case of failures, or unavailable SE)

  • OwnerDN: DN of the owner of the request

  • OwnerGroup: group of the owner of the request

  • RequestID: Unique identifier of the request

  • RequestName: Convenience name, does not need to be unique

  • SourceComponent: Who/what created the request (unused)

  • Status: Status of the request (see RMS state machine)

  • SubmitTime: Time at which the request was submitted to the database

And of course, it has an ordered list of Operations

5.2.2. Operations

An Operation is a task to execute. It is ordered within its Request

  • Arguments: Generic blob used as placeholder for some Operation types

  • Catalog: If an operation should target specific Catalogs only

  • CreationTime: Time at which the Operation was created

  • Error: Error if any at execution

  • LastUpdate: Last time the Operation was touched

  • Order: Execution order within the Request

  • SourceSE: Coma separated list of StorageElement used as source (used by some Operation types)

  • Status: Status of the Operation (see RMS state machine)

  • SubmitTime: Time at which the Operation was submitted to the database

  • TargetSE: Coma separated list of StorageElement used as target (used by some Operation types)

  • Type: Type of Operation (see Operation types)

In some cases, an Operation also has a list of `Files associated to it

5.2.3. Files

A File represents an LFN. Not all the Operations have Files. Files’ attributes are

  • Attempt: Number of time the Operation was attempted on that file

  • Checksum: Checksum of the file

  • ChecksumType: always Adler32

  • Error: Error if any at execution

  • GUID: file’s GUID

  • LFN: file’s LFN

  • PFN: file’s URL, unused in practice

  • Size: size of the file

  • Status: Status of the File (see RMS state machine)

5.2.4. RMS state machine

The objects in the RMS obey a state machine in their execution. Each of them can have different statuses. The status of a File is determined by the success of the action we attempt to perform. The status of the Operation is inferred from the Files (if it has any, otherwise from the success of the execution). The status of the Request is inferred from the Operations

5.2.4.1. Request

State machine for Request.

There are two special states for a Request:

  • Assigned: this means that it has been picked up by a RequestExecutingAgent for execution

  • Canceled: this means that we should stop trying. A Request can only be put manually in that state, and will remain as such (even if it was held by a RequestExecutingAgent, and set back)

5.2.4.2. Operation

State machine for operation.

5.2.4.3. File

State machine for File.

5.2.5. Operation types

Each of this Type correspond to what can be found in the Type field of an Operation. In order to be executed, they need to be entered in the CS under /Systems/RequestManagementSystem/Agents/RequestExecutingAgent/OperationHandlers. Each of the Type must have its own section named after the type (for example /Systems/RequestManagementSystem/Agents/RequestExecutingAgent/OperationHandlers/ReplicateAndRegister)

The OperationHandler sections share a few standard arguments:

  • Location: (mandatory) Path (without .py) in the pythonpath to the handler

  • LogLevel: self explanatory

  • MaxAttempts (default 1024): Maximum attempts to try an Operation, after what, it fails. Note that this only works for Operations with Files (the others are tried forever).

  • TimeOut: base timeout of the Operation

  • TimeOutPerFile: additional timeout per file

If TimeOut is not specified, the default timeout of the RequestExecutingAgent is used. Otherwise, the total timeout when executing an operation is calculated with TimeOut + NbOfFiles * TimeOutPerFile

For more information on how to add new Operation type, see Request Management System

5.2.5.1. DataManagement Operations

For these operations, the SourceSE, TargetSE and Catalog fields of an Operation are used

5.2.5.1.1. MoveReplica

This handler moves replicas from source SEs to target SEs.

Details: MoveReplica

No specific configuration options

5.2.5.1.2. PutAndRegister

Put a local file on an SE and registers it. This is very useful for example to move data from the experiment site to the grid world.

Details: PutAndRegister

No specific configuration options

5.2.5.1.3. RegisterFile

Register files in the FileCatalogs

Details: RegisterFile

5.2.5.1.4. RegisterReplica

Register a replica in the FileCatalogs

Details: RegisterReplica

5.2.5.1.5. RemoveFile

Remove a file from all SEs and FC

Details: RemoveFile

5.2.5.1.6. RemoveReplica

Remove the replica of a file at a given SE and from the FC

Details: RemoveReplica

5.2.5.1.7. ReplicateAndRegister

This Operation replicates a file to one or several SE. The source does not need to be specified, but can be. This is typically useful in case of failover: if a job tries to upload a file to its final destination and fails, it will upload it somewhere else, and creates a ReplicateAndRegister Operation as well as a RemoveReplica (from the temporary storage) Operation. The replication can be performed either locally, or delegating it to the FTS system (FTS3 support in DIRAC)

Details: ReplicateAndRegister

Extra configuration options:

  • FTSMode: If True, will use FTS to transfer files

  • FTSBannedGroups : list of groups for which not to use FTS

5.2.5.2. Others

5.2.5.2.1. ForwardDISET

The ForwardDISET operation is an operation allowing to execute a DISET RPC call on behalf of another user. Typically, when a datamanagement operation is performed, some accounting information are sent to the DataStore service. If this service turns out to be unavailable, a Request containing a ForwardDISET Operation will be created, that will just replay the exact same action.

Details: ForwardDISET

5.2.5.2.2. SetFileStatus

This Operation is used as a failover by jobs to set the status of a File in a Transformation.

SetFileStatus