TransformationCleaningAgent

TransformationCleaningAgent cleans up finalised transformations.

TransformationCleaningAgent options
TransformationCleaningAgent
{
  # MetaData key to use to identify output data
  TransfIDMeta=TransformationID

  # Location of the OutputData, if the OutputDirectories parameter is not set for
  # transformations only 'MetadataCatalog has to be used
  DirectoryLocations=TransformationDB,MetadataCatalog

  # From which SEs files will be removed.
  # If empty or not existing, only files in the Catalog will be removed
  ActiveSEs=

  # Enable or disable, default enabled
  EnableFlag=True

  # How many days to wait before archiving transformations
  ArchiveAfter=7

  # Shifter to use for removal operations, default is empty and
  # using the transformation owner for cleanup
  shifterProxy=

  # Which transformation types to clean
  # If not filled, transformation types are taken from
  #   Operations/Transformations/DataManipulation
  # and Operations/Transformations/DataProcessing
  TransformationTypes=

  #Time between cycles in seconds
  PollingTime = 3600
}
class DIRAC.TransformationSystem.Agent.TransformationCleaningAgent.TransformationCleaningAgent(*args, **kwargs)

Bases: DIRAC.Core.Base.AgentModule.AgentModule

class TransformationCleaningAgent
Parameters:
__init__(*args, **kwargs)

c’tor

am_Enabled()
am_checkStopAgentFile()
am_createStopAgentFile()
am_disableMonitoring()
am_getBasePath()
am_getControlDirectory()
am_getCyclesDone()
am_getMaxCycles()
am_getModuleParam(optionName)
am_getOption(optionName, defaultValue=None)
am_getPollingTime()
am_getShifterProxyLocation()
am_getStopAgentFile()
am_getWatchdogTime()
am_getWorkDirectory()
am_go()
am_initialize(*initArgs)
am_monitoringEnabled()
am_removeStopAgentFile()
am_secureCall(functor, args=(), name=False)
am_setModuleParam(optionName, value)
am_setOption(optionName, value)
am_stopExecution()
archiveTransformation(transID)

This just removes job from the jobDB and the transformation DB

Parameters:
  • self – self reference
  • transID (int) – transformation ID
beginExecution()
cleanCatalogContents(directory)

wipe out everything from catalog under folder :directory:

Parameters:self – self reference
Params str directory:
 folder name
cleanMetadataCatalogFiles(transID)

wipe out files from catalog

cleanStorageContents(directory)

delete lfn dir from all active SE

Parameters:
  • self – self reference
  • directory (sre) – folder name
cleanTransformation(transID)

This removes what was produced by the supplied transformation, leaving only some info and log in the transformation DB.

cleanTransformationLogFiles(directory)

clean up transformation logs from directory :directory:

Parameters:
  • self – self reference
  • directory (str) – folder name
cleanTransformationTasks(transID)

clean tasks from WMS, or from the RMS if it is a DataManipulation transformation

endExecution()
execute()

execution in one agent’s cycle

Parameters:self – self reference
finalize()
getTransformationDirectories(transID)
get the directories for the supplied transformation from the transformation system.
These directories are used by removeTransformationOutput and cleanTransformation for removing output.
Parameters:
  • self – self reference
  • transID (int) – transformation ID
initialize()

agent initialisation

reading and setting confing opts

Parameters:self – self reference
removeTransformationOutput(transID)

This just removes any mention of the output data from the catalog and storage