pegasus-wms 4.4.0+dfsg-7 / etc / pegasus / basic.properties

# TITLE "BASIC PROPERTIES"
#
# This is the reference guide to the basic properties regarding the
# Pegasus Workflow Planner, and their respective default values. Please refer
# to the advanced properties guide to know about all the properties that
# a user can use to configure the Pegasus Workflow Planner.
# Please note that the values rely on proper capitalization, unless explicitly 
# noted otherwise.
#
# Some properties rely with their default on the value of other
# properties. As a notation, the curly braces refer to the value of the
# named property. For instance, ${pegasus.home} means that the value depends
# on the value of the pegasus.home property plus any noted additions. You
# can use this notation to refer to other properties, though the extent
# of the subsitutions are limited. Usually, you want to refer to a set
# of the standard system properties. Nesting is not allowed.
# Substitutions will only be done once.
#
# There is a priority to the order of reading and evaluating properties.
# Usually one does not need to worry about the priorities. However, it
# is good to know the details of when which property applies, and how
# one property is able to overwrite another. The following is a mutually exclusive
# list ( highest priority first ) of property file locations.  
#
# <orderedlist>
# <listitem>--conf option to the tools. Almost all of the clients that use properties 
#  have a --conf option to specify the property file to pick up. 
# </listitem>
# <listitem> submit-dir/pegasus.xxxxxxx.properties file. All tools that work on the 
# submit directory ( i.e after pegasus has planned a workflow) pick up the 
# pegasus.xxxxx.properties file from the submit directory. The location for the 
# pegasus.xxxxxxx.propertiesis picked up from the braindump file.
# </listitem>
# <listitem>The properties defined in the user property file
# <emphasis>${user.home}/.pegasusrc</emphasis> have lowest priority. 
# </listitem>
# </orderedlist>
# 
# Commandline properties have the highest priority. These override any property loaded 
# from a property file. Each  commandline property is introduced by a -D argument. 
# Note that these arguments  are parsed by the shell wrapper, and thus the -D arguments 
# must be the first arguments to any command. Commandline properties are useful for debugging
# purposes.
#
# From Pegasus 3.1 release onwards, support has been dropped for the following
# properties that were used to signify the location of the properties file
#
# <itemizedlist>
#  <listitem>pegasus.properties</listitem>
#  <listitem>pegasus.user.properties</listitem>
# </itemizedlist>
#
# The following example provides a sensible set of properties to be set
# by the user property file. These properties use mostly non-default
# settings. It is an example only, and will not work for you:
#
# <screen>
# pegasus.catalog.replica              File
# pegasus.catalog.replica.file         ${pegasus.home}/etc/sample.rc.data
# pegasus.catalog.replica              Regex
# pegasus.catalog.replica.file         ${pegasus.home}/etc/sample.rc.data
# pegasus.catalog.transformation       Text
# pegasus.catalog.transformation.file  ${pegasus.home}/etc/sample.tc.text
# pegasus.catalog.site.file            ${pegasus.home}/etc/sample.sites.xml
# </screen>
#
# If you are in doubt which properties are actually visible, pegasus during the 
# planning of the workflow  dumps all properties after reading and prioritizing 
# in the submit directory in a file with the suffix properties.

# Property : pegasus.home
# Systems  : all
# Type     : directory location string
# Default  : "$PEGASUS_HOME"
#
# The property pegasus.home cannot be set in the property file. This property is 
# automatically set up by the pegasus clients internally by determining the installation
# directory of pegasus. Knowledge about this property is important for developers who 
# want to invoke PEGASUS JAVA classes without the shell wrappers. 
#
# pegasus.home		"$PEGASUS_HOME"


#
# SECTION "CATALOG PROPERTIES"
#

#
# SUBSECTION "REPLICA CATALOG"
#


# Property : pegasus.catalog.replica
# System   : Pegasus
# Since    : 2.0
# Type     : enumeration
# Value[0] : RLS
# Value[1] : LRC
# Value[2] : JDBCRC
# Value[3] : File
# Value[4] : Directory
# Value[5] : MRC
# Value[6] : Regex
# Default  : RLS
#
# Pegasus queries a Replica Catalog to discover the physical filenames
# (PFN) for input files specified in the DAX. Pegasus can interface
# with various types of Replica Catalogs. This property specifies
# which type of Replica Catalog to use during the planning process.
#
# <variablelist>
# <varlistentry>
# <term>RLS</term>
# <listitem> 
#      RLS (Replica Location Service) is a distributed replica
#      catalog, which ships with GT4. There is an index service called
#      Replica Location Index (RLI) to which 1 or more Local Replica
#      Catalog (LRC) report. Each LRC can contain all or a subset of
#      mappings. In this mode, Pegasus queries the central RLI to
#      discover in which LRC's the mappings for a LFN reside. It then
#      queries the individual LRC's for the PFN's.
#      To use RLS, the user additionally needs to set the property
#      pegasus.catalog.replica.url to specify the URL for the RLI to
#      query. 
#      Details about RLS can be found at
#      http://www.globus.org/toolkit/data/rls/  
# </listitem>
# </varlistentry>
# <varlistentry>
# <term>LRC</term>
# <listitem>
#      If the user does not want to query the RLI, but directly a
#      single Local Replica Catalog. 
#      To use LRC, the user additionally needs to set the property
#      pegasus.catalog.replica.url to specify the URL for the LRC to
#      query. 
#      Details about RLS can be found at
#      http://www.globus.org/toolkit/data/rls/  
# </listitem>
# </varlistentry>
# <varlistentry>
# <term>JDBCRC</term>
# <listitem> 
#      In this mode, Pegasus queries a SQL based replica catalog that
#      is accessed via JDBC. The sql schema's for this catalog can be
#      found at $PEGASUS_HOME/sql directory. 
#      To use JDBCRC, the user additionally needs to set the following
#      properties
#      <orderedlist>
#		<listitem>pegasus.catalog.replica.db.url</listitem>
#		<listitem>pegasus.catalog.replica.db.user</listitem>
#		<listitem>pegasus.catalog.replica.db.password</listitem>
#	   </orderedlist>		
# </listitem>
# </varlistentry>
# <varlistentry>
# <term>File</term>
# <listitem><para>In this mode, Pegasus queries a file based replica catalog. 
#     It is neither transactionally safe, nor advised to use for
#     production purposes in any way. Multiple concurrent access to
#     the File will end up clobbering the contents of the file.  The 
#     site attribute should be specified whenever possible. The attribute 
#     key for the site attribute is "pool".  
#
#     The LFN may or may not be quoted. If it contains linear
#     whitespace, quotes, backslash or an equality sign, it must be
#     quoted and escaped. Ditto for the PFN. The attribute key-value
#     pairs are separated by an equality sign  without any
#     whitespaces. The value may be in quoted. The LFN  sentiments about quoting apply.
#
#     <screen>
#     LFN PFN
#     LFN PFN a=b [..]
#     LFN PFN a="b" [..]
#     "LFN w/LWS" "PFN w/LWS" [..]
#     </screen>
#
#     To use File, the user additionally needs to specify
#     pegasus.catalog.replica.file property to specify the path to the
#     file based RC.
# </para></listitem>
# </varlistentry>
# <varlistentry>
# <term>Regex</term>
# <listitem><para>In this mode, Pegasus queries a file based replica catalog. 
#     It is neither transactionally safe, nor advised to use for
#     production purposes in any way. Multiple concurrent access to
#     the File will end up clobbering the contents of the file.  The 
#     site attribute should be specified whenever possible. The attribute 
#     key for the site attribute is "pool".  
#
#     The LFN may or may not be quoted. If it contains linear
#     whitespace, quotes, backslash or an equality sign, it must be
#     quoted and escaped. Ditto for the PFN. The attribute key-value
#     pairs are separated by an equality sign  without any
#     whitespaces. The value may be in quoted. The LFN  sentiments about quoting apply.
#
#     In addition users can specifiy regular expression based LFN's. A regular expression 
#     based entry should be qualified with an attribute named 'regex'. The attribute regex 
#     when set to true identifies the catalog entry as a regular expression based entry. 
#     Regular expressions should follow Java regular expression syntax.
#
#     For example, consider a replica catalog as shown below.
#
#     Entry 1 refers to an entry which does not use a resular expressions. This entry 
#     would only match a file named 'f.a', and nothing else.
#     Entry 2 referes to an entry which uses a regular expression. In this entry f.a 
#     referes to files having name as f[any-character]a i.e. faa, f.a, f0a, etc.
#
#     <screen>
#	1
#	f.a file:///Volumes/data/input/f.a pool="local"
#	2
#	f.a file:///Volumes/data/input/f.a pool="local" regex="true"
#     </screen>
#
#     Regular expression based entries also support substitutions. For example, 
#     consider the regular expression based entry shown below.
#    
#     Entry 3 will match files with name alpha.csv, alpha.txt, alpha.xml. 
#     In addition, values matched in the expression can be used to generate a PFN.
#
#     For the entry below if the file being looked up is alpha.csv, the PFN for the file 
#     would be generated as file:///Volumes/data/input/csv/alpha.csv. Similary if the 
#     file being lookedup was alpha.csv, the PFN for the file would be generated as 
#     file:///Volumes/data/input/xml/alpha.xml i.e. The section [0], [1] will be replaced. 
#     Section [0] refers to the entire string i.e. alpha.csv. Section [1] refers to a partial 
#     match in the input i.e. csv, or txt, or xml. Users can utilize as many sections as they wish.
#
#     3
#     <screen>
#     alpha\.(csv|txt|xml) file:///Volumes/data/input/[1]/[0] pool="local" regex="true"
#     </screen>
#
#     To use File, the user additionally needs to specify
#     pegasus.catalog.replica.file property to specify the path to the
#     file based RC.
# </para></listitem>
# </varlistentry>
# <varlistentry><term>Directory</term>
# <listitem><para>In this mode, Pegasus does a directory listing on an input 
#     directory to create the LFN to PFN mappings. The directory listing is 
#     performed recursively, resulting in deep LFN mappings. For example, if an
#     input directory $input is specified with the following structure
#     <screen>
#     $input
#     $input/f.1
#     $input/f.2
#     $input/D1
#     $input/D1/f.3
#     </screen>
#     Pegasus will create the mappings the following LFN PFN mappings internally
#     <screen>
#     f.1 file://$input/f.1  pool="local"
#     f.2 file://$input/f.2  pool="local"
#     D1/f.3 file://$input/D2/f.3 pool="local"
#     </screen>
#
#     pegasus-plan has --input-dir option that can be used to specify an input 
#     directory.
#
#     Users can optionally specify additional properties to configure the behvavior
#     of this implementation.
#
#     pegasus.catalog.replica.directory.site  to specify a site attribute other than
#     local to associate with the mappings. 
#
#     pegasus.catalog.replica.directory.url.prefix to associate a URL prefix for the PFN's
#     constructed. If not specified, the URL defaults to file://
# </para>
# </listitem></varlistentry>
# <varlistentry>
# <term>MRC</term>
# <listitem><para>In this mode, Pegasus queries multiple replica catalogs to
#     discover the file locations on the grid.  To use it set
# 
#     <screen>
#     pegasus.catalog.replica MRC
#     </screen>
#
#     Each associated replica catalog can be configured via properties
#     as follows. 
#
#     The user associates a variable name referred to as [value] for
#     each of the catalogs, where [value] is any legal identifier
#     (concretely [A-Za-z][_A-Za-z0-9]*) For each associated replica
#     catalogs the user specifies the following properties. 
#
#     <screen>	
#      pegasus.catalog.replica.mrc.[value]       specifies the type of replica catalog.
#      pegasus.catalog.replica.mrc.[value].key   specifies a property name key for a
#						 particular catalog
#     </screen>	 
#
#     For example, if a user wants to query two lrc's at the same time
#     he/she can specify as follows 
#
#     <screen>	
#     pegasus.catalog.replica.mrc.lrc1 LRC
#     pegasus.catalog.replica.mrc.lrc2.url rls://sukhna
#
#     pegasus.catalog.replica.mrc.lrc2 LRC
#     pegasus.catalog.replica.mrc.lrc2.url rls://smarty
#     </screen>
# 
#
#     In the above example, lrc1, lrc2 are any valid identifier names
#     and url is the property key that needed to be specified. 
# </para></listitem>
# </varlistentry>
# </variablelist>
#
#
#
# pegasus.catalog.replica			RLS


# Property : pegasus.catalog.replica.url
# System   : Pegasus
# Since    : 2.0
# Type     : URI string
# Default  : (no default)
#
# When using the modern RLS replica catalog, the URI to the Replica
# catalog must be  provided to Pegasus to enable it to look up
# filenames. There is no  default.
#
# pegasus.catalog.replica.url			(no default)





#
# SUBSECTION "SITE CATALOG"
#

# Property : pegasus.catalog.site.file
# System   : Site Catalog
# Since    : 2.0
# Type     : file location string
# Default  : ${pegasus.home.sysconfdir}/sites.xml
#
# Running things on the grid requires an extensive description of the
# capabilities of each compute cluster, commonly termed "site". This
# property describes the location of the file that contains such a site
# description. As the format is currently in flow, please refer to the
# userguide and Pegasus for details which format is expected.
#
# pegasus.catalog.site.file ${pegasus.home.sysconfdir}/sites.xml

#
# SUBSECTION "TRANSFORMATION CATALOG"
#


# Property : pegasus.catalog.transformation
# System   : Transformation Catalog
# Since    : 2.0
# Type     : enumeration
# Value[0] : Text
# Value[1] : File
# Default  : Text
# See also : pegasus.catalog.transformation.file
#
# 
# <variablelist>
# <varlistentry><term>Text</term>
# <listitem><para>In this mode, a multiline file based format is understood. The file
#     is read and cached in memory. Any modifications, as adding or
#     deleting, causes an update of the memory and hence to the file
#     underneath. All queries are done against the memory
#     representation. 
#     
#     The file sample.tc.text in the etc directory contains an example
#
#     Here is a sample textual format for transfomation catalog containing
#     one transformation on two sites
#
#     <screen>
#     tr example::keg:1.0 {
#
#       #specify profiles that apply for all the sites for the transformation
#       #in each site entry the profile can be overriden
#       profile env "APP_HOME" "/tmp/karan"
#       profile env "JAVA_HOME" "/bin/app"
# 
#       site isi {
#          profile env "me" "with"
#          profile condor "more" "test"
#          profile env "JAVA_HOME" "/bin/java.1.6"
#          pfn "/path/to/keg"
#          arch  "x86"
#          os    "linux"
#          osrelease "fc"
#          osversion "4"
#          type "INSTALLED"
#       }
#
#       site wind {
#          profile env "me" "with"
#          profile condor "more" "test"
#          pfn "/path/to/keg"
#          arch  "x86"
#          os    "linux"
#          osrelease "fc"
#          osversion "4"
#          type "STAGEABLE"
#       }
#     }
#     </screen>
# </para>
# </listitem></varlistentry>
# <varlistentry><term>File</term>
# <listitem>THIS FORMAT IS DEPRECATED. WILL BE REMOVED IN COMING VERSIONS.
#     USE pegasus-tc-converter to convert File format to Text Format.
#     In this mode, a file format is understood. The file is
#     read and cached in memory. Any modifications, as adding or
#     deleting, causes an update of the memory and hence to the file
#     underneath. All queries are done against the memory
#     representation. The new TC file format uses 6 columns:
#     <orderedlist>
#	<listitem>The resource ID is represented in the first column.</listitem> 
#	<listitem>The logical transformation uses the colonized format
#	    ns::name:vs.</listitem> 
#	<listitem>The path to the application on the system</listitem>
#	<listitem>The installation type is identified by one of the following
#	    keywords - all upper case: INSTALLED, STAGEABLE. 
#	    If not specified, or <command>NULL</command> is used, the type
#	    defaults to INSTALLED.</listitem> 
#	<listitem>The system is of the format ARCH::OS[:VER:GLIBC]. The
#	    following arch types are understood: "INTEL32", "INTEL64",
#	    "SPARCV7", "SPARCV9". 
#	    The following os types are understood: "LINUX", "SUNOS",
#	    "AIX". If unset or <command>NULL</command>, defaults to
#	    INTEL32::LINUX.</listitem> 	
#	<listitem>Profiles are written in the format
#	    NS::KEY=VALUE,KEY2=VALUE;NS2::KEY3=VALUE3 
#	    Multiple key-values for same namespace are seperated by a
#	    comma "," and multiple namespaces are seperated by a
#	    semicolon ";". If any of your profile values contains a
#	    comma  you must not use the namespace abbreviator.</listitem>
#    </orderedlist>
# </listitem></varlistentry>
# </variablelist>
#
#
# pegasus.catalog.transformation		Text


# Property : pegasus.catalog.transformation.file
# Systems  : Transformation Catalog
# Type     : file location string
# Default  : ${pegasus.home.sysconfdir}/tc.text | ${pegasus.home.sysconfdir}/tc.data
# See also : pegasus.catalog.transformation
#
# This property is used to set the path to the textual transformation 
# catalogs of type File or Text. If the transformation catalog is of type Text
# then tc.text file is picked up from sysconfdir, else tc.data
#
#
# pegasus.catalog.transformation.file	 ${pegasus.home.sysconfdir}/tc.text | ${pegasus.home.sysconfdir}/tc.data


#
# SECTION "DATA STAGING CONFIGURATION"
#



# Property : pegasus.data.configuration
# System   : Pegasus
# Since    : 3.1
# Type     : enumeration
# Value[0] : sharedfs
# Value[1] : nonsharedfs
# Value[2] : condorio
# Default  : sharedfs
#
# This property sets up Pegasus to run in different environments.
#
#
# <variablelist>
# <varlistentry><term>sharedfs</term>
# <listitem>If this is set, Pegasus will be setup to execute jobs on the shared
#    filesystem on the execution site. This assumes, that the head node of a cluster
#    and the worker nodes share a filesystem. The staging site in this case is 
#    the same as the execution site. Pegasus adds a create dir job to the executable
#    workflow that creates a workflow specific directory on the shared filesystem . 
#    The data transfer jobs in the executable workflow ( stage_in_ , stage_inter_ ,
#    stage_out_ ) transfer the data to this directory.The compute jobs in the 
#    executable workflow are launched in the directory on the shared  filesystem.
#    Internally, if this is set the following properties are set.
#     <screen>
#     pegasus.execute.*.filesystem.local   false
#     </screen>
# </listitem>
# </varlistentry>
# <varlistentry><term>condorio</term>
# <listitem>If this is set, Pegasus will be setup to run jobs in a pure condor pool, 
#     with the nodes not sharing a filesystem. Data is staged to the compute nodes from
#     the submit host using Condor File IO.
#     The planner is automatically setup to use the submit host ( site local ) as the
#     staging site. All the auxillary jobs added by the planner to the executable
#     workflow ( create dir, data stagein and stage-out, cleanup ) jobs refer to
#     the workflow specific directory on the local site.  The data transfer jobs in 
#     the executable workflow ( stage_in_ , stage_inter_ , stage_out_ ) transfer the 
#     data to this directory. When the compute jobs start, the input data for each 
#     job is shipped from the workflow specific directory on the submit host to
#     compute/worker node using Condor file IO. The output data for each job is 
#     similarly shipped back to the submit host from the compute/worker node.
#     This setup is particularly helpful when running workflows in the cloud 
#     environment where setting up a shared filesystem across the VM's may be
#     tricky. 
#     On loading this property, internally the following properies are set
#     <screen>
#     pegasus.transfer.lite.*.impl          Condor
#     pegasus.execute.*.filesystem.local   true
#     pegasus.gridstart 		   PegasusLite
#     pegasus.transfer.worker.package      true
#     </screen>
# </listitem>
# </varlistentry>
# <varlistentry><term>nonsharedfs</term>
# <listitem>If this is set, Pegasus will be setup to execute jobs on an execution site
#     without relying on a shared filesystem between the head node and the worker nodes.
#     You can specify staging site ( using --staging-site option to pegasus-plan) to
#     indicate the site to use as a central storage location for a workflow. The 
#     staging site is independant of the execution sites on which a workflow executes.
#     All the auxillary jobs added by the planner to the executable
#     workflow ( create dir, data stagein and stage-out, cleanup ) jobs refer to
#     the workflow specific directory on the staging site.  The data transfer jobs in 
#     the executable workflow ( stage_in_ , stage_inter_ , stage_out_ ) transfer the 
#     data to this directory. When the compute jobs start, the input data for each 
#     job is shipped from the workflow specific directory on the submit host to
#     compute/worker node using pegasus-transfer. The output data for each job is 
#     similarly shipped back to the submit host from the compute/worker node.
#     The protocols supported are at this time SRM, GridFTP, iRods, S3.
#     This setup is particularly helpful when running workflows on OSG where
#     most of the execution sites don't have enough data storage. Only a few
#     sites have large amounts of data storage exposed that can be used to place
#     data during a workflow run. This setup is also helpful when running workflows
#     in the cloud environment where setting up a shared filesystem across the VM's may be
#     tricky. 
#     On loading this property, internally the following properies are set
#     <screen>
#     pegasus.execute.*.filesystem.local   true
#     pegasus.gridstart 		   PegasusLite
#     pegasus.transfer.worker.package      true
#     </screen>
# </listitem></varlistentry>
# </variablelist>
#
# 
#
# pegasus.data.configuration		sharedfs