Home » Enterprise Guide

SAS Enterprise Guide: Check the Log Macro

Submitted by on April 2, 2012 – 6:20 am 20 Comments

I created a LogCheck macro [code provided] to review the log during a batch process. The SAS log files can be rather large and it is so much easier if you can just extract the issues in an easy to understand format. This macro reads in each line of the log file and keeps the ones with Error, Warning, Note, or specific text you want to trap.  Then it sends a colorful email to report what issues (if any) were found.

You can download the code and two sample programs to review or change for your situation.  Let me know what you think in the Comments section.  How can the code be improved?  Any ideas you have?  

Using the LogCheck Macro

Add the LogCheck macro to your batch job to report issues through email about the job.  When the job has no errors, you will receive an email similar to the following.  The subject line indicates no found issues – thus you have some assurance the job was successful.

sas bi logcheck 01

When the job has errors – the macros sends an email similar to the following.  Notice the subject line contains the number of Errors, Warnings, and Notes present in the log. The email lists each issue, where found in the log, and an excerpt from the log.  This makes it easier to determine if an error early on in the job actually caused the other errors and you can quickly fix the root cause.

sas bi logcheck 02

Adding the Macro to Your Code

CheckLog was designed to run with a batch job. To use this program, do the following:

1. Add the following code to the top of the SAS program you want to monitor.  Update the LogName macro variable to the location and name of your log file. Refer to the PROC PRINTTO documentation if you need more guidance.

       /* ==================================================================*/ 

       %let LogName=%QUOTE(path\file_log_name.log); 

       proc printto log="&LogName." new; run; 

       /* ==================================================================*/
            

2. Add this code to the bottom.  No changes required. 

      /* ==================================================================*/ 

       %let RUNDATE=%SYSFUNC( DATETIME(), DATETIME18. ); 

       %put PASS HEADER DATE=&RUNDATE.;  

        

       proc printto; run; 

       %logcheck(&logname.) 

       /* ==================================================================*/ 

3. In the LogCheck_macro.sas - update the EMAIL macro variable to send to the email address of your choice.  After making this change – run the two test programs (included in the download) to see how it works.  

sas bi logcheck 03

Change it Up for Your Environment 

I have the code check for Notes that actually can produce errors but do not stop production.  This can be annoying if it is something you do not care about – so you may want to prune this area of the code.

If there are some items specific to your environment that you want to trap – just add it here:

Some issues present as errors/warnings but may not be in reality.  For instance, Teradata generates a Warning that it cleared the table – which is what you want to happen.  It's not an issue.  So use this area of the code to change the type.  A TYPE=4 is a NOTE and will show as gray in the email.  Otherwise it will be a Type=2 that is bright yellow.  Since a Type=4 generates an email that indicates an issue- you may want to delete this line.

If you want to email the error-ridden log to yourself – uncomment the Attach= line.

Download the ZIP File Contents

 Download CheckLog Macro

There are three files in the ZIP file:

  • Checklog_macro.sas  /*Code – Offered AS IS*/
  • Program_wErrors.sas  /*Outputs the email with many errors*/
  • Prgoram_wNoErrors.sas  /*Outputs a clean email*/

The two Program files demonstrate how the Checklog_macro.sas program works. So modify the checklog_macro.sas program to add your email address and then run the program. If you do not have email setup at your site – obviously this won't be a great option for you. [Reference: SAS Global Forum Paper about Emailing from SAS]

Are there other Log Check Programs?

There are many other (probably superior) log check programs available from the SAS Global Forum papers and in other locations.  I offer this code to the SAS community to use as you will.

<

Never miss a post!

Get the latest BI Notes post in your Inbox when a new post is released! Click here for free subscription. You email address is kept private and you can unsubscribe anytime. Go ahead ... join us!

The following two tabs change content below.
Tricia Aanderud is a SAS Business Intelligence and Visual Analytics consultant based in Raleigh, NC. She has written several books about SAS, presented papers at many SAS conferences, and has over 10 years of SAS programming experience. Contact her for assistance with your next project.

Tags: , , ,

20 Comments »

  • Quentin says:

    Hi Tricia et al,
    A month later and I’m still thinking about log checking. But this time in the context of stored procedures.

    It seems to me as SAS programmers, we know how important it is to scan the log before trusting any results.

    But I feel like log scanning is frequently ignored in the context of developing stored processes that will be run by others. Arguably, log scanning is *more* important when developing web applications that will be used by non-programmers.

    So as SAS developers, how do we best incorporate log scanning into our stored procedures / applications? I was thinking about playing around with some something like the following, for a stored procedure that makes a pdf report:

    1. PROC PRINTTO to send log to a file on the server.
    2. Usual reporting code to create pdf. Do not display the pdf to the user, but write it to a temp location .
    3. PROC PRINTTO to close log file.
    4. %LogCheck the log file.
    5. If log is clean, open the pdf for the user to see/download.
    6. If log is not clean, pop up a message (javascript window? pdf report?) with a report of the errors found in log. (even if not interpretable by user, it’s enough that they will know something bad happened).

    Does that seem like reasonable approach? Have you seen papers/sites that talk about log checking in a server environment.

    I’m tried to avoid the potential problem of having a stored process happily return bad results, while the log reports unitialized variables, missing data created, merges with duplicate by values, etc etc.

    Thanks for any thoughts,
    –Q.

  • Chris S. says:

    I haven’t figured anything out, either. I just posted about this the other day:

    http://sas.cswenson.com/checklog/eg

    There are two ways to do this: Use PROC PRINTTO, as we’ve seen, and use the project flowchart with options to export the log as part of the project.

    These are the simple solutions. I bet there are more complicated ones…

  • I like this method Chris – especially during batch. You’re right – there are few key times – if the Teradata/Oracle extract failed – the rest of the job is wrong.
    I would prefer that execution stop and alert me over sending me an email that is 3 pages long.

  • Chris S. says:

    I’ve been thinking about this a lot, and it’s kind of a trend with SAS users: We seem to think that the log has all of the indication we would need about whether something went wrong. However, it’s not really the case – there are other indicators.

    A good one is the SYSERR macro variable, which could be checked after every step to see if anything failed. A macro that did this (e.g., see below) could halt the program and email you on the fly instead of running the whole program. In our log check reports, how many of those issues need to be fixed? Probably just the first 1 or 2. The rest were just reverberations from the initial issue(s) and didn’t even need to be mentioned or run.

    %macro InterimCheck;
    %if &SYSERR>3 %then %do;
    %put NOTE: Issues detected.;
    /* — Insert code to email user — */
    /* Abort the program */
    %abort cancel;
    %end;
    %mend InterimCheck;

    The above could be called after every step or just after certain risky steps (e.g., you’re waiting on someone else’s data set once a month). Of course, this could fail, too, if SAS has a critical failure and crashes. Then you wouldn’t hear anything about it. Both our macros can suffer this fate, except if you schedule it as a separate task. Then your server could crash, but then you’d have other problems to worry about.

    Anyway, just some thoughts about how we go about things. Perhaps a combination of both methods is best: check during AND after, and you can have very robust programs.

  • Chris S. says:

    That’s a good point: Within-network emails would be okay. If there’s a black hat in the office, you’ve already lost on the security front.

  • So in short – just know what is okay in your environment. In the example – I used my gmail account for simplicity. When used for work the emails were going to my work address so it was acceptable. Again – just know what is okay for your environment.

    Another idea – you could have the program send you a text message that simply said Errors/Warnings existed.

  • Chris S. says:

    Ha! Thanks for the compliment. They aren’t that good, are they? I keep finding problems here and there, at least.

  • Chris S. says:

    There are some odd error messages that come out of Oracle like that, too. I haven’t seen any other phrases, like warning, that have an odd format like that, so changing “error:” to “error” would probably be sufficient.

  • Excellent point Chris. I have not used this macro with any data that was that sensitive.

    You don’t get a report from me that doesn’t have traffic lighting in it! :-)

  • Chris S. says:

    Oh, just to be clear: Email is just plain text, and can be easily discovered by a black hat. You do not want to have an inadvertent disclosure of protected health information through email. The sector of education also has similar privacy laws. It’s best to be over-cautious in this area and simple avoid this kind of thing.

  • Chris S. says:

    Well, you know about mine. My macro assumes you’ve already done something, and you forgot to check along the way. So there’s no need to set up your code with PROC PRINTTO. Now of course in EG, that’s the only way to do it, as far as I can figure out. It also emails, but I like the style of your email by bringing more attention to what was identified. I have an update coming out soon that will work a bit better in EG. I’ve been reviewing it to get it ready for the SAS Global Forum later this month.

    One very serious note of caution: Sometimes these messages can contain information in the data set, which could include patient information like name, date of birth, address, and other protected information. This would be a violation (if not literally than in spirit) of HIPAA. I would remove the specific messages from your emails to avoid this very serious issue. That’s why in my macro only a summary is provided.

  • Lex:

    LOL … second vote for Chris!

    Tricia

  • Chris Swenson writes the kind of macros I dream of writing. :-)

    He’s also giving a talk about it at the SAS Global Forum. I’m hoping I will have time to attend it.

  • Julien Damon says:

    Maybe you can try the %checklog macro made by Chris Swenson.
    I used it for a year. It is rather complete.
    (but I’ve not tried your macro yet!)

    http://checklog.cswenson.com/

  • Lex Jansen says:

    Also check out this one (very extensive documentation):

    http://sas.cswenson.com/checklog

    – Lex

  • William Thong says:

    You can turn on Project Log in SAS EG, however just need to maintain it as it can grow in size. Here’s a PDF on how to automate logging in SAS EG http://www.sas.com/offices/NA/canada/downloads/presentations/Calgary/Auto_Error_Report.pdf

  • I don’t know about EG – I have not seen an option to do that. And I searched support.sas.com and did not find anything conclusive.

    I’m planning a post about working with the EG log – I’ll try to get to this week. Maybe it will at least show you some quick tips for using the log.

  • Oh my goodness all this praise! :-) Thanks so much.

    Wow … Quentin … a ton of great suggestions! Where have you been hiding?

    -I had not seen that SUGI paper before so I put it on my list. They make a great point about excluding Notes – that is a better way to think about it.
    -Hmm … Error without the colon … every tricky!
    -Oh love the idea about resetting the counter. So would you place that code before the %logcheck? Can it be part of the %logcheck?

  • Quentin says:

    A question- Do you think there is a way to do this in EG without using PROC PRINTTO at the beginning to direct the log to a file? Is there a default location for the log file in EG?

    In “regular” SAS, when you batch submit a program, the log file is written to the same directory as the program, with the same name, so it is possible to scan that file (actuallly you need to copy the file to a temp directory and scan the copied version). In interactive, you can use DM commands to copy the current log to a separate file, and scan it. I’m just about to start using EG, so don’t know where the log file is written by default. Just a thought.

    –Q.

  • Quentin says:

    Hi,

    Great post (and great blog, and great book!).

    My favorite paper on log-checking is by Li and Troxell:
    http://www.nesug.org/proceedings/nesug01/cc/cc4008.pdf

    They argue that it is safer to specify a list of notes to exclude, rather than than specify the notes to include. You end up with a longer list, but it’s a neat approach.

    A couple other small points.

    Searching for “ERROR:” will not catch a few error messages where SAS puts an error number before the colon, e.g.:

    ERROR 48-59: The format MNDDYY was not found or could not be loaded.

    I think we need to consider a situation in which the main code encounters an error and SAS enters syntax check mode (set obs to 0). Might be helpful to check for this and reset obs to MAX, to make sure the steps in %logcheck() run, e.g.:

    %if %sysfunc(getoption(obs))=0 %then %do;
    options obs=max replace nosyntaxcheck;
    %end;

    Thanks again!
    –Q.

2 Pingbacks »