FTS3Agent

FTS3Agent
{
  PollingTime = 120
  MaxThreads = 10
  # How many Operation we will treat in one loop
  OperationBulkSize = 20
  # How many Job we will monitor in one loop
  JobBulkSize = 20
  # Max number of files to go in a single job
  MaxFilesPerJob = 100
  # Max number of attempt per file
  MaxAttemptsPerFile = 256
  # days before removing jobs
  DeleteGraceDays = 180
  # Max number of deletes per cycle
  DeleteLimitPerCycle = 100
  # hours before kicking jobs with old assignment tag
  KickAssignedHours  = 1
  # Max number of kicks per cycle
  KickLimitPerCycle = 100
}
class DIRAC.DataManagementSystem.Agent.FTS3Agent.FTS3Agent(agentName, loadName, baseAgentName=False, properties={})

Bases: DIRAC.Core.Base.AgentModule.AgentModule

This Agent is responsible of interacting with the FTS3 services. Several of them can run in parallel. It first treats the Operations, by creating new FTS jobs and performing callback. Then, it monitors the current jobs.

CAUTION: This agent and the FTSAgent cannot run together.

__init__(agentName, loadName, baseAgentName=False, properties={})

Common __init__ method for all Agents. All Agent modules must define: __doc__ __RCSID__ They are used to populate __codeProperties

The following Options are used from the Configuration: - /LocalSite/InstancePath - /DIRAC/Setup - Status - Enabled - PollingTime default = 120 - MaxCycles default = 500 - WatchdogTime default = 0 (disabled) - ControlDirectory control/SystemName/AgentName - WorkDirectory work/SystemName/AgentName - shifterProxy ‘’ - shifterProxyLocation WorkDirectory/SystemName/AgentName/.shifterCred

It defines the following default Options that can be set via Configuration (above): - MonitoringEnabled True - Enabled True if Status == Active - PollingTime 120 - MaxCycles 500 - ControlDirectory control/SystemName/AgentName - WorkDirectory work/SystemName/AgentName - shifterProxy False - shifterProxyLocation work/SystemName/AgentName/.shifterCred

different defaults can be set in the initialize() method of the Agent using am_setOption()

In order to get a shifter proxy in the environment during the execute() the configuration Option ‘shifterProxy’ must be set, a default may be given in the initialize() method.

am_Enabled()
am_checkStopAgentFile()
am_createStopAgentFile()
am_disableMonitoring()
am_getBasePath()
am_getControlDirectory()
am_getCyclesDone()
am_getMaxCycles()
am_getModuleParam(optionName)
am_getOption(optionName, defaultValue=None)
am_getPollingTime()
am_getShifterProxyLocation()
am_getStopAgentFile()
am_getWatchdogTime()
am_getWorkDirectory()
am_go()
am_initialize(*initArgs)
am_monitoringEnabled()
am_removeStopAgentFile()
am_secureCall(functor, args=(), name=False)
am_setModuleParam(optionName, value)
am_setOption(optionName, value)
am_stopExecution()
beginExecution()

reload configurations before start of a cycle

deleteOperations()

delete final operations

endExecution()
execute()

one cycle execution

finalize()

finalize processing

getFTS3Context(username, group, ftsServer, threadID)

Returns an fts3 context for a given user, group and fts server

The context pool is per thread, and there is one context per tuple (user, group, server). We dump the proxy of a user to a file (shared by all the threads), and use it to make the context. The proxy needs a lifetime of at least 2h, is cached for 1.5h, and the lifetime of the context is 45mn

Parameters:
  • username – name of the user
  • group – group of the user
  • ftsServer – address of the server
Returns:

S_OK with the context object

initialize()

agent’s initialization

kickJobs()

kick stuck jobs

kickOperations()

kick stuck operations

monitorJobsLoop()
  • fetch the active FTSJobs from the DB
  • spawn a thread to monitor each of them
treatOperationsLoop()
  • Fetch all the FTSOperations which are not finished
  • Spawn a thread to treat each operation