demoIterator.job

demoIterator                                # Total: 156 lines
|-- demoIterator.jdl                        #   79 lines
`-- demoIterator.policy                     #   77 lines

demoIterator.jdl

 1 # -----------------------------------------------------------------------------
 2 #  Copyright (C) 2010 Novell, Inc. All Rights Reserved.
 3 #
 4 #  NOVELL PROVIDES THE SOFTWARE "AS IS," WITHOUT ANY EXPRESS OR IMPLIED
 5 #  WARRANTY, INCLUDING WITHOUT THE IMPLIED WARRANTIES OF MERCHANTABILITY,
 6 #  FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGMENT.  NOVELL, THE AUTHORS
 7 #  OF THE SOFTWARE, AND THE OWNERS OF COPYRIGHT IN THE SOFTWARE ARE NOT LIABLE
 8 #  FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
 9 #  TORT, OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE
10 #  OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
11# -----------------------------------------------------------------------------
12 #  $Id: demoIterator.jdl 10344 2009-11-20 21:46:43Z jastin $
13 # -----------------------------------------------------------------------------
14 
15 import time, random
16 
17 #
18 # Add to the 'examples' group on deployment
19 #
20 if __mode__ == "deploy":
21    try:
22        jobgroupname = "examples"
23        jobgroup = getMatrix().getGroup(TYPE_JOB, jobgroupname)
24        if jobgroup == None:
25            jobgroup = getMatrix().createGroup(TYPE_JOB, jobgroupname)
26        jobgroup.addMember(__jobname__)
27    except:
28        exc_type, exc_value, exc_traceback = sys.exc_info()
29        print "Error adding %s to %s group: %s %s" % (__jobname__, jobgroupname, exc_type, exc_value)
30
31
32 class demoIteratorJob(Job):
33
34      def job_started_event(self):
35           print 'job_started_event'
36           self.completed = 0
37 
38           # Launch the joblets
39           numJoblets = self.getFact("jobargs.numJoblets")
40           print 'Launching ', numJoblets, ' joblets'
41
42           pspace = ParameterSpace()
43           i = 1
44           while i <= numJoblets:
45              pspace.appendRow({'name':'joblet'+str(i)})
46              i += 1
47          pspace.maxJobletSize = 1
48          self.schedule(demoIteratorJoblet,pspace,{})
49 
50      def joblet_completed_event(self, jobletnumber, node):
51          self.completed += 1
52          self.setFact("jobinstance.memo", "Tests run: %s" % (self.completed))
53
54
55 class demoIteratorJoblet(Joblet):
56
57      def joblet_started_event(self):
58          print "Hi from joblet ", self.getFact("joblet.number")
59          time.sleep(random.random() * 15)
60
61          cmd = self.getFact("jobargs.cmd")
62          if len(cmd) > 0:
63              system(cmd)
64
65
66
67              # Example of more sophisticated exec
68              # e.g.  e.signal("SIGUSR1")
69              """
70              e = Exec()
71              e.setCommand(cmd)
72              #e.setStdoutFile("cmd.out")
73              e.writeStdoutToLog()
74              e.writeStderrToLog()
75              #try:
76              e.execute()
77              #except:
78                  #self.retry("retryable example error")
79              """

demoIterator.policy

 1 <!--
 2  *=============================================================================
 3  * Copyright (C) 2010 Novell, Inc. All Rights Reserved.
 4  *
 5  * NOVELL PROVIDES THE SOFTWARE "AS IS," WITHOUT ANY EXPRESS OR IMPLIED
 6  * WARRANTY, INCLUDING WITHOUT THE IMPLIED WARRANTIES OF MERCHANTABILITY,
 7  * FITNESS FOR A PARTICULAR PURPOSE, AND NON INFRINGMENT.  NOVELL, THE AUTHORS
 8  * OF THE SOFTWARE, AND THE OWNERS OF COPYRIGHT IN THE SOFTWARE ARE NOT LIABLE
 9  * FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
10  * TORT, OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE
11  * OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
12  *=============================================================================
13  * $Id: demoIterator.policy 10344 2009-11-20 21:46:43Z jastin $
14  *=============================================================================
15  -->
16 
17 <policy>
18    <constraint type="accept" reason="Too busy for more work. Try again later!">
19       <or>
20          <lt fact="job.instances.queued" value="4" />
21          <contains fact="user.groups" value="superuser" />
22       </or>
23    </constraint>
24 
25    <constraint type="start" reason="Waiting on queue">
26      <or>
27        <lt fact="job.instances.active" value="2" />
28        <contains fact="user.groups" value="superuser" />
29      </or>
30    </constraint>
31 
32    <jobargs>
33      <fact name="numJoblets"
34            type="Integer"
35            description="joblets to run"
36            value="100"
37            visible="true" />
38 
39      <fact name="cmd"
40            type="String"
41           description="Simple command to execute"
42            value="" />
43
44      <fact name="os"
45            type="String"
46            description="Regular expression match for Operating System"
47            value=".*" />
48 
49      <fact name="cpu"
50            type="String"
51            description="Regular expression match for CPU architecture"
52            value=".*" />
53    </jobargs>
54 
55   <constraint type="resource" reason="Does not match">
56     <and>
57       <eq fact="resource.os.family" factvalue="jobargs.os" match="regexp" />
58       <eq fact="resource.cpu.architecture" factvalue="jobargs.cpu" match="regexp"/>
59 
60       <or>
61         <and>
62              <defined fact="env.VENDOR" />
63            <eq fact="resource.os.vendor" factvalue="env.VENDOR" match="regexp" />
64         </and>
65         <undefined fact="env.VENDOR" />
66       </or>
67     </and>
68   </constraint>
69 
70   <job>
71     <fact name="description"
72           type="String"
73           value="This example job is a reference for a simple test iterator. It is useful for demonstrating how policies and job args can be used to target the job to a particular resource." />
74   </job>
75 
76 </policy>
77

Definitions:

Job

A representation of a running job instance.

Joblet

Defines execution on the resource.

MatrixInfo

A representation of the matrix grid object, which provides operations for retrieving and creating grid objects in the system. MatrixInfo is retrieved using the built-in getMatrix() function. Write capability is dependent on the context in which getMatrix() is called. For example, in a joblet process on a resource, creating new grid objects is not supported.

GroupInfo

A representation of Group grid objects. Operations include retrieving the group member lists and adding/removing from the group member lists, and retrieving and setting facts on the group.

Exec

Used to manage command line execution on resources.

ParameterSpace

Defines a parameter space to be used by the scheduler to create a Joblet set. A parameter space might consist of rows of columns or a list of columns that is expanded and can be turned into a cross product.

zosadmin deploy

The deployment for the DemoIterator job is performed by lines 20-29 of demoIterator.jdl. When jobs are deployed into the grid, they can optionally be organized for grouping. In this case, the demoIterator job is added to the group named examples, and can be displayed in the PlateSpin Orchestrate Development Client in the Explorer view at the location:

/Orchestrate Servers/Grid_Name/Jobs/examples 

For a general overview of how jobs are added to groups during deployment, see Walkthrough: Deploying a Sample Job in the PlateSpin Orchestrate 2.6 Installation and Configuration Guide.

job_started_event

When the DemoIterator job receives a job_started_event, it creates a ParameterSpace JDL class and adds the number of rows as indicated by the value of the argument numJoblets (see lines 42-46 in demoIterator.jdl). A ParameterSpace object is like a spreadsheet, containing rows and columns of information that might all be given to one joblet or sliced up across many joblets at schedule time. In this case, the ParameterSpace is told that maxJobletSize is 1 (see line 47), meaning a joblet instance is created for each row in the ParameterSpace during job scheduling (see line 48).

Not shown in this example is the fact that a joblet can get access to this “spreadsheet” of information by calling self.getParameterSpace(), and calling hasNext() and next() to enumerate through each row of information. To learn more about putting information in a ParameterSpace object from a job and obtaining that information from the JobletParameterSpace object from a joblet, see ParameterSpace.

The resource that runs the joblet is determined from the resource constraint specified in lines 18-30 and 55-68 of demoIterator.policy, and from the values specified for the parameters os and cpu supplied on the command line. If these parameters are not specified on the command line, the default value for both is the regular expression .*, which means to include everything.

The constraints at lines 18-30 in demoIterator.policy define the work load for the resources. In this case, resources do not accept jobs if there are already four jobs queued up, and are not to run jobs if there are two or more jobs currently in progress.

To learn more about setting start, resource, or accept constraints in a policy file, see Job Arguments and Parameter Lists in Policies.

joblet_started_event

As the DemoIterator joblet is executed on a particular resource, it receives a joblet_started_event. When this happens, the DemoIterator joblet simply sleeps for a random amount of time to stagger the execution of the joblets and then sends a command to the operating system, if one was supplied as a job argument. The command is executed on the target operating system using the built-in function system(), which is an alternative to using the more feature-rich class Exec.

For more information on sending commands to the operating system using the Exec class, see Exec .

After the joblet is finished running, a joblet_completed_event is sent to demoIteratorJob, which increments the variable completed, and posts the updated value to the job fact jobinstance.memo (see lines 50-52 in demoIterator.jdl). You can see the text for the memo displayed on the Job Log tab in the list of running jobs in the PlateSpin Orchestrate Development Client.

For more information, see Starting and Stopping the PlateSpin Orchestrate Development Client in the PlateSpin Orchestrate 2.6 Installation and Configuration Guide.