The Unix Time Command : tips & tricks

If you have a program ./prog.e then in the bash/ksh shell you can type this command and the output on the screen details how long the code took to run:

$ time ./prog.e
real 24m10.951s user 6m2.390s sys 0m15.705s


Real time - Elapsed time from beginning to end of program (or wall clock time).The real time is the total time of execution.
CPU time - Divided into User time and System time.
User time - time used by the program itself and any library subroutines it calls.The user time is the time spent processing at the user/application process level.
System time - time used by the system calls invoked by the program (directly or indirectly).The sys time is the time spent by the system at the system/kernel level.

If the wall clock time is consistently much longer than the total of the system and user time, then the fetching of data to and from hard drives may be taking a good deal of time. In parallel codes, the code may be spending a good deal of time waiting on communication between processors.

By this command you can check your script performance.

Read more ...>>

Basics Of UNIX

The purpose of this post is to have a single page of frequently used basics commands for getting started with UNIX.

Basic UNIX Command Line (shell) navigation:

Directories:
Directories: 
Moving around the file system: 
Listing directory contents: 
Changing file permissions and attributes 
Moving, renaming, and copying files: 
Viewing and editing files:


Directories:

File and directory paths in UNIX use the forward slash "/" to separate directory names in a path.

examples:

/	root directory
/usr	directory usr (sub-directory of / "root" directory)
/usr/STRIM100	STRIM100 is a sub directory of /usr

Moving around the file system:

pwd	Show the "present working directory", or current directory.
cd	Change current directory to your HOME directory.
cd /usr/STRIM100	Change current directory to /usr/STRIM100.
cd INIT	Change current directory to INIT which is a sub-directory of the current directory.
cd ..	Change current directory to the parent directory of the current directory.
cd $STRMWORK	Change current directory to the directory defined by the environment variable 'STRMWORK'.
cd ~bob	Change the current directory to the user bob's home directory (if you have permission).

Listing directory contents:

ls	list a directory
ls -l	list a directory in long ( detailed ) format

for example:
$ ls -l
drwxr-xr-x    4 cliff    user        1024 Jun 18 09:40 WAITRON_EARNINGS
-rw-r--r--    1 cliff    user      767392 Jun  6 14:28 scanlib.tar.gz
^ ^  ^  ^     ^   ^       ^           ^      ^    ^      ^
| |  |  |     |   |       |           |      |    |      | 
| |  |  |     | owner   group       size   date  time    name
| |  |  |     number of links to file or directory contents
| |  |  permissions for world
| |  permissions for members of group
| permissions for owner of file: r = read, w = write, x = execute -=no permission
type of file: - = normal file, d=directory, l = symbolic link, and others...

ls -a        List the current directory including hidden files. Hidden files start
             with "."
ls -ld *     List all the file and directory names in the current directory using
             long format. Without the "d" option, ls would list the contents
             of any sub-directory of the current. With the "d" option, ls
             just lists them like regular files.

Changing file permissions and attributes:

chmod 755 filename	Changes the permissions of file to be rwx for the owner, and rx for the group and the world. (7 = rwx = 111 binary. 5 = r-x = 101 binary)
chgrp user filename	Makes file belong to the group user.
chown cliff filename	Makes cliff the owner of file.
chown -R cliff dir	Makes cliff the owner of dir and everything in its directory tree.

You must be the owner of the file/directory or be root before you can do any of these things.


Moving, renaming, and copying files:

cp file1 file2	copy a file
mv file1 newname	move or rename a file
mv file1 ~/AAA/	move file1 into sub-directory AAA in your home directory.
rm file1 [file2 ...]	remove or delete a file
rm -r dir1 [dir2...]	recursivly remove a directory and its contents BE CAREFUL!
mkdir dir1 [dir2...]	create directories
mkdir -p dirpath	create the directory dirpath, including all implied directories in the path.
rmdir dir1 [dir2...]	remove an empty directory

Viewing and editing files:

cat filename	Dump a file to the screen in ascii.
more filename	Progressively dump a file to the screen: ENTER = one line down SPACEBAR = page down  q=quit
less filename	Like more, but you can use Page-Up too. Not on all systems.
vi filename	Edit a file using the vi editor. All UNIX systems will have vi in some form.
emacs filename	Edit a file using the emacs editor. Not all systems will have emacs.
head filename	Show the first few lines of a file.
head -n  filename	Show the first n lines of a file.
tail filename	Show the last few lines of a file.
tail -n filename	Show the last n lines of a file.

Back to top

Read more ...>>

UNIX in Datastage

Almost every Data warehousing project requires that your job runs in batches at specified time slots. In many cases these batches may consist of jobs from different systems in addition to the ones in Datastage. In such cases the Datastage jobs are usually scheduled by using an external scheduling tool like ESP Scheduler, Control M, Autosys, TWS etc… 


This is made possible by writing scripts that will make your jobs run through the command line.  I would consider the command line a very powerful interface to Datastage which lets us do more than just run the normal job. The guides which are present in the Datastage documentation will be very helpful in exploring the various things that can be done through the command line. 

Basic of UNIX required to carry out your execution.

IN UNIX, the Datastage home directory location will always be specified in the “.dshome” file which will be present in the root 

directory.  Before you can run your Datastage commands you will have to run the following commands

  cd  `cat /.dshome`

This will change the location to the home directory. By default this will be /opt/IBM/InformationServer/Server/DSEngine

  . ./dsenv > /dev/null 2>&1

This will run the dsenv file which contains all the environment variables. Without doing this, your UNIX commands won’t run on the command prompt.

After you have done this then you can use any Datastage command for interacting with the server. The main command you can use is the ‘dsjob’ command which is not used only to run jobs but for a wide variety of reasons. Let’s look at the various ways you can use the dsjob command

To run a job:

Using the dsjob command you can start,stop,reset or run the job in  validation mode.

dsjob  –run –mode VALIDATE  project_name  job_name

This command will actually run the job in validation mode. Similarly you can use RESET or RESTART instead of VALIDATE depending on what type of run you want. If you want a normal run then you will not need to specify the –mode keyword as shown below

dsjob –run project_name  job_name | job_name.invocationid

Running with the invocationid would mean that the job would be run with that specific invocation id

Now if you have parameters to set or paratemeterset values to set then this can also be as set as shown below

dsjob –run –param variable_name=”VALUE” –param psParameterSet=”vsValueSet” project_name  job_name

To stop a job:

Stopping a job is fairly simple. You might not actually require it but still its worth to take a look. It acts the same way as you would stop a running job the Datastage director.

dsjob –stop  project_name  job_name|job_name.invocationid

To list projects, jobs, stages in jobs, links in jobs, parameters in jobs and invocations of jobs

dsjob can very easily give you all the above based on the different keywords. It will be useful for you if you want to get a report of what’s being used in what project and things like that

The various commands are shown below

‘dsjob –lprojects’  will give you a list of all the projects on the server

‘dsjob –ljobs  project_name’ will give you a list of jobs in a particular project

‘dsjobs –lstages  project_name job_name’  will give you a list of all the stages used in your job. Replacing –lstage with –links will give you a list of all the links in your job. 

Using –lparams will give you a list of all the parameters used in your job. 

Using –linvocations will give you a list of all the invocations of your multiple instance job.

To generate reports of a job:

You can get the basic information of a job buy using the  ‘jobinfo’ option as shown below

dsjob -jobinfo  project_name job_name

Running this command will give you a short report of your job which includes The current status of the job, the name of any controlling job for the job,

 the date and time when the job started , the wave number of the last or current run (internal InfoSphere Datastage reference number) and the user status

You can get  more detailed report using the below command:

dsjob -report  project  job_name BASIC|DETAIL|XML

BASIC means that your report will contain very basic information like start/end time of the job , time elapsed and the current status of the job. DETAIL as the name indicates will give you a very detailed report on the job down to the stages and link level. XML would give you an XML report which is also a detailed report in an XML format.

To access logs:

You can use the below command to get the list of latest 5 fatal errors  from the log of the job that was just run

dsjob -logsum –type FATAL –max 5 project_name job_name

You can get different types of information based on the keyword you specify for –type. Full list of allowable types are available in the help guide for reference.

There are a number of other options also available to get different log information. You can explore this in more detail in the developer guide.

 With the Datastage commands you can administer jobs, run jobs, maintain jobs, handle errors, prepare meaningful job logs and even prepare reports.
  

The possibilities are endless. If you like to code then you won’t mind spending your time exploring the command line options available.

Read more ...>>