Archive for August, 2010


How to get Todays Julian Date in Perl

There are 2 kinds of Julian dates. One that counts daily since time of Julius Caesar or the julian date based on current year. The latter seems to be more popular. Todays date is August 31st which is 2010243 or 10243 Julian date. This equals to 243 days since the first day of the year. There are some examples on the net but they get fancy with finding the leap year to taking actual julian date 2455440. The problem with many is they include the time of day as well and thus result in creating Perl code to clean it up. The method I use below comprises of just 3 subroutines, one to get todays date, another to calculate the days since the first of the year and finally puts together the last 2 digits of the year and the total days since the first.


#!/usr/bin/perl -w
use strict;
use Time::localtime; ### For get_date subroutine

## Declare variables for date handling
my $yyyy;  	
my $mo;  	
my $da;		


  &get_date(0); # Pass number of days in past to current or past date. 0 = today, 1 is yesterday etc.
   
   my $jyyy = $yyyy; #Since we need to make year unique for julian part.

   my $dayOfYear = &dayofyear($da,$mo,$jyyy); # Figure days of the year using subroutine.
   $jyyy -= 2000; # Removes the first 2 digits - likely a y2k gotcha if we used 1900 in 20th century.
   
   my $julianDate = $jyyy . $dayOfYear; # Put year and days together
   my $todaysDate = $yyyy . "/" . $mo . "/" . $da; # Put together todays date
   
   print "Todays date is $todaysDate \n";
   print "julianDate is $julianDate \n"; # Output 20100831 will = 10243


# -----------------------------------------------------------------------
# get number of days since first of the Year - Julian
# -----------------------------------------------------------------------

sub dayofyear {
    my ($day1,$month,$year)=@_;  # Passed todays date information
    my @cumul_d_in_m =
(0,31,59,90,120,151,181,212,243,273,304,334,365);
    my $doy=$cumul_d_in_m[--$month]+$day1;
    return $doy if $month < 2;
    return $doy unless $year % 4 == 0;
    return ++$doy unless $year % 100 == 0;
    return $doy unless $year % 400 == 0;
    return ++$doy;
}

# -----------------------------------------------------------------------
# get local time or past date.
# -----------------------------------------------------------------------

sub get_date {
   my $pastno = shift;
   my $pastdate = localtime(time -(86400 * $pastno));
   $yyyy = $pastdate->year() + 1900;  # localtime->year() return years since 1900
   $mo = $pastdate->mon() + 1;        # localtime->mon() return 0-11,  Jan. - Dec.
   $da = $pastdate->mday();

   $mo = "0"."$mo"  if ($mo <= 9);
   $da = "0"."$da"  if ($da <= 9);
   
}

My Best Practices with Perl

  • Add -w to shebang line to enable warnings. This helps point out flaws in your code during runtime.
  • use strict; and declare all variables with my. This forces you to keep your syntax clean and correct and avoid use of barewords.
  • Keep your code simple – KISS (keep is simple stupid). With Perl, there is more than 1 way to handle most objectives – try and keep it simple. I’ve experienced working with Perl scripts written by others and some people just have the knack of complicating the hell out of it. For example, one fellow wanted to find a file in a directory. Instead of just if (-e /path/filename), he or she read the entire directory into an ARRAY and then applied a grep to find the file. 10 lines of code to do what if (-e /path/filename.ext) can do!
  • Always comment your code and keep others in mind when doing so. Someone else may have to troubleshoot or take over your Perl script in the future. Also useful for when revisiting after 6 months. Sometimes I go back to my old Perl scripts and ask myself, “What the hell was I thinking!”.
  • Split up functions by subroutines. I never have one subroutine handle more than 1 function and they are usual designed to stand alone. Other subroutines will utilize other subroutines, but each remains 1 function. This practice allows me to keep my code organized and easier to update when needed. Its also great practice for code reuse.
  • Create modules of common functions. If you have several scripts utilizing a common function – create a module for them to share. I create modules for exception handling and logging, notifications, and many other things such as SecureTransfer or NDM functionality for file transfer in multiple scripts. This helps to reduce the time to update those features when needed. I have seen people duplicate common features in hundreds of scripts and find themselves having to modify each one when a change was needed. I’ve made this mistake myself and learned from it.
  • Learn and master regular expressions. This magic word, regex, is godsend! I use it all the time and its POWERFUL!
  • Practice code reuse. Build up a library of code that works. Saves a ton of time when scripting for new projects.

Migrating Oracle Warehouse builder and Data.

Oracle Warehouse Builder 10g currently does not support Data Guard that makes it easy to switch databases or migrate to another server and database. I ran into this similar limitation and had originally opted to redeploying all the mapping from the MDL export. This process what taking me between 4 to 6 hours because the Control Center GUI is extremely (not to mention painfully) slow. Finaly we worked with Oracle support to find a better solution to this problem. After some testing and several successful switched and migrations I have formulated the steps below.

Goal
Move SID and OWB installation and mappings to another server.

Pre-migration Steps

  1. Prepare 2 servers, app server and database server to have identical installation versions of database and OWB to the original systems.
  2. Have the DBA take a baseline of current setup using the owbcollect.sql that Oracle provides with OWB. This helps to compare after the installation of the new system is done. We do this step now because usually we end up putting it on standby mode before proceeding with the migration.
  3. Clone the SID and data and restore on new server. This process is beyond the scope here – usually the DBA takes care of this. I remember vaguely of the steps involved from my Oracle DBA training but I never had the opportunity to put it to practice.
  4. After the SID has been migrated and the database is running – start up OWB on the new server using the new SID information. OWB_HOME/owb/bin/unix/local_service_login.sh -startup /path/to/OWB  You can check to see if this is running by using ps -ef | grep OWB. You’ll receive a readout of the SID info at the end of the output.

Update the location credentials stored in the Control Center:

  1. Start the OWB Browser Listener. For Linux : OWB_home/owb/bin/unix/startOwbbInst.sh
  2. Start the Repository Browser. https://HOSTNAME:8999/owbb/RABLogin.uix?mode=design
  3. Connect to Warehouse Builder Browser as the Control Center Owner.
  4. Select “Locations Report” under Reports – Deployments
  5. Select the Location you would like to update
  6. Click on the “Unknown” link in the Validation column. (last column)
  7. In the “Connection Details” section of the Location Validation Report page, update the Host, Port and Service Name accordingly. Update to SID details to the new database server.
  8. Click on the “Update Details” button.
  9. Click on the “Test Location” button to check the location.
  10. Repeat steps above for each location that requires changes.
  11. Finally, Update the Control Center location “PlatformSchema” the same way as explained in steps above.

Update the Location Credentials stored in the Design Repository (DBA task)

  1. Using SQLPlus, connect as the warehouse builder repository owner.
  2. Execute the script UpdateLocation.sql (you can get script at Meta-link)
  3. SQL>@UpdateLocation.sql LOCATION_NAME HOST PORT SERVICE_NAME
  4. Do the above for each location that your OWB utilizes and provide the new server HOST, PORT and SERVICE_NAME(SID).

Note: LOCATION_NAME must be upper case or you will get a not found error.

Update the Control Centers Credentials stored in the Design Repository (DBA task)

  1. Using SQLPlus, connect as the warehouse builder repository owner.
  2. Execute the script UpdateControlCenter.sql attached to this document for each Control Center
  3. SQL>@UpdateControlCenter.sql CONTROL_CENTER_NAME HOST PORT SERVICE_NAME
  4. Do the above for each location that your OWB utilizes and provide the new server HOST, PORT and SERVICE_NAME(SID).
Note: LOCATION_NAME must be upper case or you will get a not found error.
Also: we were originally confused regarding the CONTROL_CENTER_NAME. This is almost always “DEFAULT_CONTROL_CENTER”.
Test the Migration
Log into the Design Center and then the control center to test a deploy or two to make sure all is working. Try one from staging and one from data warehouse. If successful deploy – then you should be good.
Finally run the baseline script on the new system using owbcollect.sql and compare with the original one you took prior to starting this process.Thats it – you’re done. You actually save about 5 hours with this process.

Troubleshooting
The only problem I’ve seen is when DBA was trying to update the locations credentials is that we we using the wrong location name. Or location that was being used in another environment but not with the system we were presently working with. Especially with confusion of lower environments having more locations for testing than production. Other than that I have not seen any other issues.

Calculating Time Lapse in Perl

I know there is a module that does this for you such as use Time::Elapse, but where I work, adding modules is a pain, and requires getting permission from upper management. I just find it easier to create the functionality myself.

My objective – to take 2 time stamps from a file and calculate the time lapse in minutes.


#!/usr/bin/perl -w
use strict;

my $start   = "02:01:16";
my $end     = "07:12:30";
my ($startHR, $startMIN) = split(/:/, $start); # split the time by colon to separate the hour and minutes.
my $startMinutes = ($startHR*60) + $startMIN; # Convert start time to minutes
my ($endHR, $endMIN) = split(/:/, $end); # start over with end time
my $endMinutes = ($endHR*60) + $endMIN;
my $range = $endMinutes - $startMinutes; # finally subtract the two.
print $range;

The result is 311 minutes.

My First Post

Technically this is not my first post. I have blogged in the past which you can find here: http://jklarsen.com/myblog.

I decided to start a new blog with entirely different focus on jklarsen.com. My goal with this blog is to share my experience with the technology field of IT. I am also deaf which provides some challenging obstacles at times and may be interesting how I get round those obstacles.

Currently I work for a large bank where I administer some of the Oracle applications as well as automate many of the nightly processing using bash, perl and autosys. I will dig back first to share some things I’ve learned, mistakes I’ve made and my thoughts on some of the technology that we use.