Home » Stored Processes

SAS Stored Process: Benefits of Minimalist Programming

Submitted by on September 20, 2012 – 7:10 am 18 Comments

Minimalism

SAS stored processes can do a lot.  While I haven’t taken on the life of a true minimalist, recently I have come to see the benefits of limiting my stored process source code to just one statement.  And the best part of this practice is, I can still do everything I want.

When you create a stored process, SAS writes the source code somewhere (see Tricia’s post on where the code is stored).  Typically you enter the source code in the stored process wizard.  I am now in the habit of typing just one %INCLUDE statement in the source code, and I end up with a stored process window that looks like below.  I use the /source2 option to make sure the included code is written to the log. 

Minimalist Stored Process

Stored Process Code Tab

The full stored process source code ends up as below, because SAS adds a bit of wrapper code:

Minimalist Stored Process: Source Code

Stored Process Source Code

The main code is DoSomething.sas.  It has the usual DATA steps, PROC steps,  macro calls, maybe even %INCLUDE statements to call other other sub-modules.

Minimalist Stored Process: Main Code

Stored Process Main Code

So what are the benefits?

  1. You can store the main SAS code (DoSomething.sas) wherever you want.  I’ve been writing SAS code for MUCH longer than I’ve been writing stored processes.  I’m used to storing my SAS code in /Project directories.  With this approach, I can put the source code for the stored process in a central location managed by the SAS Administrator, but have DoSomething.sas in the project directory structure.
     
  2. Easier development of stored process.  DoSomething.sas can be written with anything.  Like Display Manager SAS and interactive tools like the data step debugger? Use DM SAS.  Like process flows and point-and-click generation of code? Use Enterprise Guide.  Like vi? Emacs? Ultraedit? Notepad?  Use what you like to write the program.  Then just have the stored process %INCLUDE it.
     
  3. Easier updates to stored processes.  Tricia has a great post on 3 ways to edit a stored process.  With this method, editing stored process code can be done with any program that can edit a .sas file (see #2 above).  You don’t need write access to the actual stored process source code, you only need access to your program, DoSomething.sas.
     
  4. Logical separation of stored process itself (stored in the metadata) from the main code.  Want to edit the stored process (modify prompts, change which server it runs on, etc.)?  Open the stored process.  Want to update the code?  Open the code.  I often have a simple Enterprise Guide project for each stored process, with a link to the stored process, a link to the code, and sometimes links to sample code used for reference materials, test scripts, etc. The visual workspace of EG makes it an excellent tool for organizing and accessing all of the files related to a stored process.
     

Want More Indirection?

Having the stored process source code be simply a %INCLUDE statement which calls another program is an example of indirection.  Don Henderson has a great post where he describes a setup having a single stored process capable of running ANY code module.  He uses the macro language, rather than %INCLUDE, to invoke code.  The equivalent %INCLUDE approach would be having a stored process with the source code: %INCLUDE &program /source2;  and have a prompt which allows the user (or more likely a developer) to specify the name of a SAS program to be invoked.  So it’s another layer of abstraction.  Neat stuff.

What Say You?

As I’ve mentioned before, I’m still new to stored processes.  So I look forward to hearing any thoughts as to the pros and cons of this approach.  I’m excited that Tricia invited me to start blogging here on occasion.  And mostly see it as an opportunity for me to share some ideas, and hopefully get some feedback for improvements!  

 


Learn More about SAS Stored Processes and Prompts

You can learn more tips and tricks for creating, debugging, and using SAS stored processes in the 50 Keys to Learning SAS Stored Processes book. It's a complete guide to SAS stored processes. Check Amazon for best pricing and quick shipping!

The following two tabs change content below.

Quentin McMullen

Quentin McMullen has been programming in SAS for 15 years, and for the past year has been working on SAS BI projects. He has presented at national and regional SAS user group conferences, and can often be found corresponding with colleagues on SAS-L.

Tags: , ,

18 Comments »

  • Paul Homes says:

    I’d echo Chris’ comments about using Windows explorer extensions for version control (TortoiseSVN and TortoiseGit are my favourites on Windows). That way you only have to get to grips with one interface rather learning how it’s done (slightly differently) in all the editors and IDE’s in use.

    You could also consider using the metadata export facility and checking in the .SPK package along with the source code. The .SPK includes the metadata and the source code but being a binary is not very diff-able (without unpacking or using special software). However having the binary .SPK alongside the normal text .SAS file means you have the metadata for the stored process versioned alongside the SAS code (which will make it easy to revert to an old version of the metadata if necessary).

    If you need to automate the process at all you might be interested in the batch metadata import/export facility documented here: http://support.sas.com/documentation/cdl/en/bisag/65422/HTML/default/viewer.htm#n0ix91vx4dmorbn156g7n6ll389q.htm

    Sometimes I also version control the SAS deployment directories (with lots of .ignores) to also track changes to config files, scripts, format catalogs, autocall macros etc.

  • Bill Dupre says:

    Quentin,

    Thanks for the reply. I will try to get back with you on our status once we get this working.

    Bill

  • Quentin says:

    Hi Bill,

    This sounds like a great plan to me. Full disclosure: I do not have experience with real version control systems (subversion, git, etc.) It’s something I’m hoping to start playing with as I continue to transition from SAS prgorammer to SAS developer.

    That said, your plan should work. As you understood, the benefit of this approach is that you separate the .sas code from the metadata. So yes, your .sas files can be managed by a version control system, and promoted/demoted between environments, etc.

    For the stored process itself, I have started having separate stored processes in /prod and /dev environments (in my case “environments” just means different directories, as we don’t have a /dev server). This is useful so that when you want to change the stored process itself (e.g. update prompts, etc.), you can work in /dev. It’s easy enough to just copy the /prod stored process to a /dev directory, and make the changes there, then copy it back to /prod after testing. If you do it that way, you have to remember to change the actual source code file as well. I didn’t say that well, but Angela Hall has a great post on copying stored processes:

    http://blogs.sas.com/content/bi/2012/06/06/backup-or-copy-all-components-of-your-stored-process-before-making-changes/

    Another option to consider would be having a prompt for your stored process, &ENVT which would resolve to PROD or DEV or TEST. Then your %include statement could be something like :
    %include “~/&Envt/BIsamples/Code/DoSomething.sas”;
    And you could switch from running prod code to dev code or test code just by changing the prompt value.

    Hope that helps. Drop a note (or a full blog post) to let us know how it goes, or if you hit any interesting points. I think there is a lot of interest in the SAS community on doing more with version control systems. So lots of folks could benefit from your experience.

    –Quentin

  • Quentin says:

    Thanks for the tip, Chris! I didn’t know about the ability to use relative paths. That’s very nice.

  • I currently have 3 version control systems in play in my various development projects: CVS, SVN, and Git. All of them integrate well with the file system (Windows, in my case), and that’s what I rely on to maintain the source management control. Even when using environments that offer “good” source control integration, such as Visual Studio, I use the file system (Windows Explorer extensions or command console) to do operations such as checkout and commits.

    I also work with SAS code projects and SAS Enterprise Guide in the same way. This is made possible by the “relative path” support that was added to the v4.3 release. “External” items such as Excel files and .SAS programs can be maintained in a path relative to the EGP file, and you can check the whole shebang in/out of source control without any assumption about absolute paths on the end user system. To find this little nugget in SAS EG, check File->Project Properties, the File References tab.

  • Quentin, some of the “auto” options that EG submits (as a favor to you, of course) you can override by supplying your own at certain points, such as via the Tools->Options->SAS Programs window.

    See this blog post for a few more details.

  • I don’t see why it would not work – but I agree. We need more input!

  • Bill Dupre says:

    Tricia,

    Thanks for the response.

    We would handle changes/errors through our testing process. The way our Java promotion process works is a developer writes and tests his code in his local environment then checks it in to Subversion. He could then use Jenkins to push that code to our dev-integration environment where it would be tested by others (e.g., QA team). Once we are satisfied with it in dev-integration we can promote it to a second test environment where it can go through UAT. At some point we can push it to production.

    My hope is to get SAS code to work the same way. It would be the testing phases that would prevent bad code from being in all environments at once.

    My thought is that we can have the Stored Process metadata defined the same in all three environments and only have to move code (which changes more often). If metadata has to change we would have an admin do that in each environment separately with testing at each stage.

    Does this sound reasonable to SASsers? I am still pretty new to SAS and want to hear from you guys.

    Thanks,
    Bill

  • Bill:
    That sounds like a reasonable approach to me.

    Just a question – how do you handle changes? For instance if start working in the code and make an error – then the stored process is broken in all three environments? Or do you just register a new stored process when you are working on version 2.0?

    I’d be curious for you to write a post about the process and the plus/minus of it. Seems like a lot of folks struggle with the version control issues for many of these tools. Other than DI Studio 4.3 – I’m not aware of any of the other tools supporting a version control system.

    Tricia

  • Bill Dupre says:

    Quentin,

    I am thinking about using this approach for a particular problem I am having. We are trying to implement an automated promotion process that takes Stored Processes and moves them from a dev-integration server up to a test server then to production (all separate VMs). This approach would mimic what we do with our Java stack.

    I think your solution would work because it would allow our SAS developers to just write the code in their EG instance (or whatever) and then put that code into our source code control system – Subversion. We could then create the Stored Process in each of the environments and automate the promotion of the SAS code to each environment (we would use something like Jenkins CI to move it from Subversion to dev first).

    Do you have any experience with this type of an approach?

    Thanks,
    Bill

  • Quentin McMullen says:

    Thanks again Don,
    I am considering writing a blog post that will begin with a group-therapy type introduction/confession: “Hi, my name is Quentin, I’m a SAS programmer, and I’m starting to like using Enterprise Guide”. Which might be going too far. But there are some things I like about it.

    That said, agree completely about frustrations with the wrapper code that the EG stored process wizard adds. Angela Hall has a nice post that summarizes the wrapper, http://blogs.sas.com/content/bi/2012/06/18/sas_eg_stored_process_wizard/.

    I usually turn off the automatic generation of %STPBEGIN/%STPEND. The fact that the EG wizard constantly prompts me to add %STPBEGIN/%STPEND is definitely an annoyance. If I don’t pay attention I can esily break a stored process by responding the wrong way to a warning dialog box popped up when the wizard can’t see %STPBEGIN/%STPEND.

    Even worse in my opinion is that I don’t think we have any control over other parts of the wrapper code. We can’t turn off generation of VALIDVARNAME=ANY, or the magic string at the end intended to close any unmatched quotes or unclosed comment blocks: ;*’;*”;*/;quit;

    I’d be happy if there were a simple option that allowed me to turn off all of the wrapper code.

    I haven’t tried using SMC to create a stored process. So if that is an easy way to avoid some of these wrapper code issues, I’ll be very happy! Thanks for the suggestion.

  • Quentin McMullen says:

    Thanks very much for commenting, Don! Your book “Building Web Applications with SAS/IntrNet” should be required reading for BI developers. I keep it out on my desk, right next to Tricia and Angela’s “50 Keys to Learning SAS Stored Processes”.

    Thanks for expanding on the runmacro approach (hope you noticed that I plugged it toward the end of my post). I think this %include approach is basically a “lite” version of the same idea. As I get deeper into this stuff, I wouldn’t be surprised if I switch over to %runMacro.

  • Quentin,

    First let me say that I make every effort to avoid using EG to create stored processes. So I admit that I am biased. The only time I use EG to create a stored process is when I have no other choice. I define my stored process using SAS Management Console and then typicall use Notepad to edit the code. If I have to, I will occasionally use EG to edit the code – but I edit it as a file, not as a stored process.

    The issue with EG is that it does rewrite your file. It includes code at the top and the bottom and in my case that quite often messes things up. It can also screw up the security settings (e.g., on Unix or Linux systems). There is an option in EG to tell is to not automatically add the stpbegin and stpend macros. It will prompt you if it sees those macros in your code and ask if you want them. But it still add extra stuff (garbage IMO) regardless.

    This is especially a problem if you don’t want the stpbegin and stpend macros included at all. I can’t count how many prior customers have called me in a panic because their stored processes never work. And 9 times out of 10 it is because they simply looked at the stored process in EG and it rewrote the code.

  • Like Quentin, I have lots of years of Jurassic SAS experience and I do something quite similar. Instead of %include I use a macro call – for a macro defined in an autocall library. Here is the final blog posting (in a series) that I did about this a couple of years ago:

    http://hcsbi.blogspot.com/2010/01/home-for-my-runmacro-stored-process.html

    It inludes a link to an article on sasCommunity.org with the complete code for the runMacro stored process as well as links to the series of blog posts.

    There in IMO several advantages of using a macro over %include. First, I need not worry about updating the path when, for example, code is promoted (e.g., Dev to Test, Test to Prod, etc.). And second, the macro to run can be a parameter. This allows for significant re-use. Quite often I am able to build complete applications with just one stored process and lots of macros instead of having to write a lot of single purpose stored processes.

  • Quentin says:

    Thanks Thomas,
    I’m still getting started with stored processes, glad to hear someone with more experience agree that it’s a useful approach.

    I don’t understand the ‘gotcha’ you are warning about, however. Let’s say the stored process source code is DoSomethingSP.sas. It has the one-line %include statement. And say DoSomething.sas has my ‘real’ code (DATA steps, PROC steps, etc.) Are you saying that when I edit the stored process, the stored process wizard will ask if I want to replace the code in DoSomething.sas with the code in DoSomethingSP.sas?

    I haven’t seen that happen before. And I don’t understand how it could, since I don’t think EG even ‘knows’ about DoSomething.sas. It isn’t defined in the metadata at all. The only link to it is the %include statement.

    Thanks!
    –Q.

  • Like Quentin, my SAS experience goes way way back. I do something similar. But I don’t using %include. Instead I package things as macros and save them in an autocall directory. Much more flexible that %include IMO (e.g., no need to change file paths when promoting to test or prod from dev). Here is a link to a blog post about this approach:

    http://hcsbi.blogspot.com/2010/01/useful-parametersextensions-for.html

    It includes a link to a page on sasCommunity.org for the complete code and links to the complete series of blog posts.

    I’ve had a number of projects where I have only one or two stored processes to support the entire application thanks to this approach.

  • This sounds like Enterprise Guide being helpful. ;-)

  • Thomas says:

    Hi Quentin,

    nice summary of what we’ve been doing for years for the same reasons you mentioned.
    Just be careful when using Enterprise Guide for creating/modifying the STP, as it will ask whether to replace the code in the project with server-code (i.e. overwriting your “real” source code with “%include …”) :o)

    br, Thomas