Are Your Backups Complete?

By Rich Loeber

No security plan is complete without addressing the issue of safe and complete backups.  When the AS/400 was first announced, the process was fairly simple and straight forward.  All you needed to do was save all the libraries on your system, save OS/400 in a way it could be reloaded and then make arrangements to backup your security and configuration settings.

As the AS/400 has morphed into the IBM i, things have gotten more complex.  Additional items needed to be saved and new command structures needed to be learned.  With today’s systems, it is not as simple as the original plan, but the tools are all there.  If you haven’t reviewed your backup plan recently, it would be a good idea to dust it off to make sure that everything you need is being saved on your system.

A backup plan generally has two objectives.  First, to be able to recover the entire machine in the event of a catastrophic loss of your processor.  This recovery can be either on your own machine, following repair, or at an off-site location.  The second objective is to be able to recover an individual object in the event of accidental or purposeful object loss.  Your backup plan needs to support both of these objectives.

To get a complete backup of your system today, your backup plan has to provide for the following components of your system:

1. The IBM i operating system
2. Your machine configuration
3. Your systems security settings
4. All native IBM i libraries
5. All shared folder files
6. All files in your Integrated File System (IFS)

The more recent additions to this list are these last two and this is where I want to concentrate efforts for this article.  If you’ve been around the IBM i for a while, you’re probably already quite familiar with items 1 through 4.  If not, feel free to shoot me an Email and I can help you out off-line.

It could be argued that numbers 5 and 6 on this list are both part of the IFS, but the IBM i provides some commands specifically designed for the shared folder system, so you can consider it separately if you want to.

You can save just the Shared Folder files (also known as the QDLS file system in the IFS) using the IBM i command SAVDLO.  To save all files in the Shared Folder system, you can use the following command:

SAVDLO DLO(*ALL) DEV(TAP01) OUTPUT(*PRINT) SAVACT(*YES)

Note, this command will produce an audit report of what was saved which you can print and keep with the backup or just hold in an output queue.  This report tends to be quite lengthy on most systems.  When you run this command, all document library objects on your system will be saved along with all folders and files in the entire Shared Folder system.  The SAVACT(*YES) parameter helps to make sure that the backup is complete by saving objects that may be in use.  If you run your backup in a restricted state, this parameter will not be necessary.

To save the files in your Integrated File System, you will need to use the IBM i SAV command.  This command can actually be used to save your entire system, but its use for native IBM i objects can be confusing so most shops limit its use to the IFS objects.  To save all files in your IFS, use the following command:

SAV DEV(‘/QSYS.LIB/TAP01.DEVD’)
OBJ((‘/*’) (‘/QSYS.LIB’ *OMIT) (‘/QDLS’ *OMIT))
SAVACT(*YES) OUTPUT(*PRINT)

Note that the output device naming method is quite different from standard IBM i command formats.  Also, if you are using the plan outlined above, you want to make sure that you don’t save objects again that have already been saved at points 1, 4 and 5.  The OBJ parameter controls this function.  In this SAV command, the OBJ parm contains three elements.  The first element specifies that everything is to be saved.  The next two sub parameters modify this by specifying that the QSYS file system and the QDLS file system be omitted from the backup.  By leaving these out, you will end up saving just the IFS objects you need.

If some of this appears strange to you, then it means you need to dust off your backup plan and bring it up to date.  Today would probably be a good day to do that.

If you have any questions about this topic, you can reach me at rich at kisco.com.  I’ll give it my best shot.  All email messages will be answered.

Recovering Your IBM i Security Configuration

By Rich Loeber

I once worked, many years ago, with a company that had a disaster recovery plan in place and actually tested it once or twice.  The plan called for us to arrive at the recovery center and then, almost immediately, turn security on the recovery box OFF to make things easier during the recovery.  There was no corresponding step later in the recovery to reactivate security, so for the duration of any disaster, the backup system was exposed to security violations.  Now, to give them some credit, this was in the days before networks and the Internet, so their exposure was probably pretty limited.  In today’s world, however, when you have to switch over to a disaster recovery site, you will need to have your security plan in place and built into the recovery process.

To understand how best to do this, you first have to understand what the different security objects and settings are on your system along with how and where they are saved.

System values make up a large component of your security policy implementation.  The system values are all stored in the QSYS library.  They are saved when you run a SAVSYS, the “Entire system” or “System data only” options on the SAVE menu or when you save your entire system using the “Run Backup” (RUNBCKUP) command.  When you get to your recovery site and restore your OS from one of these, your system values will get restored in the process.  Keep in mind that these settings will be restored the way they existed when your SAVSYS was created.

User profiles, including group profiles, are also stored in the QSYS library and are saved when you save the OS.  Since these can change more frequently than the OS, you can also save them using the “Save Security Data” (SAVSECDTA) command.  When you restore your OS, you will get the profiles as they existed when that backup was taken.  If you have saved the the profiles separately, make sure that you restore them again as soon as the OS has been restored so that you get the most current set of passwords available.

Job Descriptions can also have an important affect on how jobs run and how security is enforced on your system.  IBM generally recommends that you create your job descriptions in the QGPL library where they keep theirs, but a job description can be created and stored in any library on your system, especially if you are running any third party software.  When you backup a library using the “Save Library” (SAVLIB) or “Save Object” (SAVOBJ) command, the job descriptions will travel along and will be restored when those libraries are restored.

Finally we get to your resource security setup.  This, by its very nature, is a bit more complicated than the above items, but certainly manageable.  The good news is that a lot of the resource security information is stored directly with the object.  This includes its public authority setting, its object audit setting, the profile of the object’s owner, the primary group and the link, if any, to an authorization list.  So, when you save the objects in your library, you will also be saving these security items.  The two resource security pieces that are not included are the authorization list itself and the private authority configuration.

All authorization lists are stored in the QSYS library.  They are saved when you save your OS and they are also saved when you run the SAVSECDTA command.  As with the user profiles already mentioned, these can change more dynamically than the OS, so having a separate SAVSECDTA is a good idea.  After you restore your OS, make sure to include a restore of your security data to get the most recent versions of your authorization lists.

Private authorities are also stored in the QSYS library and are included as a part of the user profile data.  So, when you save the most recent profile information, you are also saving the most recent private authorities.  When you recover your profiles, you are also recovering the private authorities.  But …. there is a catch.  After your system has been recovered, including the most recent profiles and ALL application libraries, you will also have to run the “Restore Authority” (RSTAUT) command for *ALL users.  This will restore the private authorities to objects and to authorization lists.  This is a step that can be overlooked with dire consequences as far as your security implementation is concerned.

There is more information about all of this in IBM’s Security Manual for the OS.  I recommend you review the manual for more details.

If you have any questions about anything included in this tip, you can reach me at rich at kisco dot com,  All email messages will be answered as quickly as possible.

Restoring Your Security Configuration

By Rich Loeber

I recently wrote about saving your security configuration.  Once you’ve got your system backed up including all of the security information, what’s the best way to make sure that all of that security information is restored correctly when you have to do a full system restore.  Missing something or getting things in the wrong sequence could result in your objects being restored without the security configuration that you want.

First, you will need to plan the sequence of events in your restore operation.  For security to come out right, you should always restore your saved user profiles first.  The second task is then to restore the objects to your system.  Lastly, once the profiles and objects have all been restored, you should restore the private authorities to objects.

Let’s take a look at how to accomplish each of these steps in a way the makes certain that your security settings are all preserved.  As a safeguard, make sure you have access to the password for the QSECOFR profile on the system being restored.  You should have access to the current password and the password being restored.  If you have any serious security issues during the restore, you may have to logon as QSECOFR as a recovery option, so having access to these passwords may become critical.

To restore your saved user profiles, use the Restore User Profiles (RSTUSRPRF) command.  If you are restoring all user profiles, you should be aware that all settings for each profile will be based on the saved version of that profile.  If any changes have been made to a profile and you are restoring to the same system, those changes will be lost.  Also, make sure that the user profile being used to do the restore has both all object (*ALLOBJ) and security administrator (*SECADM) special authorities.  Otherwise, any profiles being restored with *ALLOBJ special authority could have that authority stripped during the profile restore operation.  This will not affect critical IBM Q profiles, in case you’re worried.

Once your user profiles are successfully restored, the next step is to get your objects restored.  You can use any of the following commands to restore objects on your system:

•    Restore Library (RSTLIB)
•    Restore Object (RSTOBJ)
•    Restore Configuration (RSTCFG)
•    Restore Object (RST) – for objects in the IFS
•    Restore Document Lib Object (RSTDLO) – for objects in shared folders (QDLS)

When restoring objects, be careful how you use the “Allow object differences” (ALWOBJDIF) parameter.  If you attempt to restore an object that already exists on the system and the object being restored is owned by a different profile than that being restored, the allow object differences command setting of *NONE will result in the object not being restored.  If you use a value of *ALL, then the object will be restored and the system owner will be preserved.

Also, you need to be aware that there are special considerations for public authority and authorization list values during object restores.  Generally, if an object is being restored that already exists on the system, the current object settings are preserved rather than applying those from the saved object.  For objects secured by authorization lists, the ALWOBJDIF parameter can result in objects not being restored when there is a difference between the current value and that being restored.  There is a thorough discussion of what is restored and not restored in the Security Reference Manual, Chapter 8.  Check on the issues of private authorities, object auditing, authority holders and more for these considerations.

To restore authorities, it is recommended that you run the Restore Authority (RSTAUT) command after all objects have been restored.  This will rebuild the object authorities in the user profiles.  Your restore will not be complete until this step is done.

Saving Your Security Configuration

By Rich Loeber

As your IBM i shop’s security officer, you’ve developed a security policy; analyzed the user base; classified the various points of information access and implemented your policy to protect the data assets on your system. You have a current user profile base that you’re maintaining on a regular basis. When new applications come along, you review the security requirements and make sure that they can fit within your established policies. You probably even have a plan in place for offsite backup storage for your shop with a regular schedule of backups and tape rotations.

But, have you given thought to how your security policies are stored on your system and how they figure into your backup process? If not, you might be in for a rude awakening when you need to restore your system following a catastrophic system loss. This tip will take a look at how the various pieces of your security implementation are stored on your IBM i processor. A future tip will then look at how your can make sure that your security setup can be restored successfully.

Your security configuration is stored in three different places on your IBM i server. You should be familiar with these storage locations and how they relate to your security implementation.

Some security information is stored with individual objects. These include things like public authority settings, who owns the object, what the owner’s authority to the object is, group authorities to the object, the name of any authorization list that applies to the object along with private authority information.

In addition to the security information stored with each individual object, there is also a wealth of security information stored with your user profiles. This information includes user profile attributes, the profile’s UID (User Identification Number) and GID (Group Identification Number), private authority information to objects, object ownership information, group profile information, profile auditing information and information about registered functions for the profile.

Lastly, there is security information stored with existing authorization lists on your system. This includes a list of objects secured by the list along with other normal authority information to be considered for objects secured by the list.

When you save the objects on your system, only part of the security information is getting backed up to tape. In order to get a complete backup of your system, including all of the current security information, you must not only save the objects, you must also save the security information. This requires using the Save Security Data (SAVSECDTA) command. This command will backup the user profiles, authorization lists and any authority holders that you have in your security configuration. Only when both the objects and the associated security data for your system are saved will you get a full backup of your security implementation.

There are some restrictions on the use of the SAVSECDTA command, so if you introduce it into your save/restore plan now, make sure that you understand those restrictions and accommodate them. Of special concern is the PRECHK parameter and the possibility that it could abnormally terminate your backup operation. See the HELP text associated with the SAVSECDTA command for more information.

IBM i Security New Year’s Resolutions

By Rich Loeber

Many people, self included, take this time of year for a little introspection.  We try to see where we have problems or weaknesses and then contemplate methods and strategies to make changes.  If we’re serious, we’ll sit down and make a list of things to do in the New Year.  As the security officer for your IBM i shop, this is a good opportunity to do just that for your installation and here’s my list of some items you should consider.

  • Finally take the plunge and move the security level of your system up at least one level.  If you’re running at level 20 (shame on you!), move to level 30.  If you’re at level 30, move to level 40.  Take the time to plan the move and use system security auditing to check results before you make the change.
  • Check your system for user profiles with permanent passwords; then change them all.  This will, at least, enforce an annual change in these passwords.  And, this means your personal password too!
  • Review the user profiles on your system and look for people who have left the company.  Make sure those profiles are disabled and their passwords have been changed to *NONE.  If you can do so easily, remove the profiles.
  • Review all user profiles with the *ALLOBJ special authority or *SECADM/*SECOFR user class.  Verify that each profile has a valid business reason for these high level access permissions.
  • Do a full audit of all of the security related system values on your box and make sure they are set up to enforce your company’s security policies correctly.
  • Audit your system backup plan and make sure that the tapes are being properly labeled and stored for quick and accurate recovery if needed.
  • Check on the way your backup tapes are transported to and from your off-site storage facility to make sure they are secure in transit.
  • Dust off your Disaster Recovery plan and make sure it still works.  Bring it up to date, then schedule an actual test.
  • Review physical security arrangements for your computer room and for all devices attached to your system.  Do a walk thru and actually look at the various work locations.  Check for things like passwords on post-it notes and lists of system resources.  Spank a few hands (not literally) for violators.  Your physical presence in the end-user’s environment will go a long way towards reinforcing the importance of security.
  • Resolve to review your system security audit journal on a regular basis.  If you don’t have it active, turn it on.  If you have it turned on but never look at it, develop a review process to check for problem issues.
  • If you don’t have network security implemented at the exit point level on your system, commit to getting this done in the new year.  Either write your own exit routines or take a look at one of the many packages available for this important area of system security.  If you’re new to this, take a good look at Kisco’s SafeNet/i Exit Point solution.

If you have other items to add to the list, let me know by email.  I’d love to hear about your new year’s resolutions.

For me, I’m going to just resolve to loose 20 pounds this year.  But then, that was my resolution last year and I’m weighing in at the same rate this year.  At least things didn’t get worse.  Let’s hope that your system security resolutions fare better.

More On Automating Your Disaster Recovery Restore

By Rich Loeber,

Some while ago, I wrote a security tip about automating your IBM i disaster recovery restore process. Of the many tips I’ve done over the years, none has drawn as much “fan mail” as this particular tip.

The essence of the tip is an implementation of the OS/400 command LODRUN to automatically restore a full system backup tape. The LODRUN searches the tape for a known program (named QINSTAPP), loads that program into the session QTEMP library and then calls it. Your QINSTAPP program can then load the contents of the tape. The only thing you have to do is create the QINSTAPP program and include it in the backup.

Well, I created that methodology quite a while ago, and a “fan” recently contacted me to let me know that, during testing, he found a problem with it. It seems that the current versions of i/OS don’t like it when the SAVLIB for *ALLUSR or *NONSYS libraries is not the first backup set on the tape. The tip clearly was calling for the QINSTAPP *PGM object to be the first object on the tape, so there was a conflict.

First, I have to commend my “fan” (Keith Scott at Hoover Materials Handling Group) for putting the process to the test. Over the years, I’ve given away hundreds of copies of the shell CL program for QINSTAPP and Keith was the only one to test it and find the problem. No matter where you get your code, testing should ALWAYS be a requirement, especially for something so important as a disaster recovery procedure. I tested this process long ago and it worked then, but it is problematic today.

The news is not all bad, however. With the current versions of i/OS, the LODRUN command supports a SEQNBR parameter where you can use a *SEARCH parameter. This will cause the process to scan the tape until it finds the QINSTAPP saved from QTEMP. At that point, it will load the program and run it.

There is a downside, however. Because the SAVLIB has to come first on the tape, there is a lot of tape searching going on before the actual save gets going. This could add 30 minutes or so to your restore process, depending on the kind of tape you’re using. But, the restoration process is still fully automated, so there’s a strong benefit.

The modified save process should now be changed to save your system in the following sequence:

  • SAVLIB for the libraries you want to save (either *ALLUSR or *NONSYS)
  • SAVOBJ for the QINSTAPP saved from QTEMP
  • AVSECDTA to save your security setup
  • SAVDLO for your shared folder objects
  • SAV for the IFS (other than shared folders)

The QINSTAPP has to be changed to first restore the security data which will sit on the tape right after the QINSTAPP itself. Then, the restore should force a rewind on the tape and restore the libraries. When that’s done, the remaining restores for the IFS will complete the process. When all is said and done, the last step must be a RSTAUT to restore object authorities.

To properly illustrate this, I’ve updated my sample QINSTAPP CL program and also created a companion QDRSAVE CL program to show the system save process in the right sequence. If you’d like to see these sample programs, send me an email (rich@kisco.com) and I’ll send you the shells that I recently created.

Automating Your Disaster Recovery Restore

By Rich Loeber,

Part of the job of a security officer is creating, maintaining and testing your disaster recovery plan. A major part of disaster recovery is recreating your computing environment on a completely different system and this always involves data and program restores. You might be one of the fortunate ones that has access to a comprehensive third party save/restore application that automates the disaster recovery restore process for you. If you’re in this group, this post is not for you. But, if you’re a small to medium sized shop that can’t afford (or won’t consider) one of these products, then you’re on your own to create a workable disaster recovery method.

If you fall into this group, then you should consider implementing a LODRUN program to automate your disaster recovery process. LODRUN is an i/OS command that is normally just used by developers of third party software as a method of controlling the installation process for their software packages. But, LODRUN can easily be used to automate your recovery restore process as well.

When you run the LODRUN program, the command calls a process which looks for a *PGM object at the beginning of the tape with the unique name of QINSTAPP. Once found, it will transfer this *PGM object into the special QTEMP library and then call it, passing along the parameters from the LODRUN command which includes the device name of the tape drive being used.

So, for you to have a controlled restore of your backup tape, all you need to do is create a CL program named QINSTAPP with a single parameter for the tape device name. Then, you can use that CL program to do the controlled restore from the tape onto the new system.

A typical off-site backup tape will contain the following items:

1. A backup of the user profiles
2. A backup of the configuration
3. Backups of the user libraries
4. Backups of the IFS objects

You are probably already familiar with exactly how you do backups for your shop. To use the LODRUN method for an automated restore, all you need to do is create a CL program named QINSTAPP that will restore the objects from the backup tape in the same sequence in which they are recorded on the tape.

Some hints:

1. Change your save program to transfer the QINSTAPP *PGM object to QTEMP and then do a SAVOBJ to save it as the very first object on your backup tape.

2. Be sure that you use the *LEAVE option so that the tape is properly positioned after each restore is completed.

3. Be sure to run the RSTAUT command after everything has been restored to get the authorities correct.

4. Use the CHKTAP command at the very end to rewind and/or unload the tape, this will give you better flexibility when it is time to make changes to your program.

5. Add a clear comment to the CL program you use for creating backups that advises any programmer that works on that program to make sure and mirror any changes in your restore program.

6. When you first create the QINSTAPP CL program, have your save program handy so that you get the sequence of events absolutely correct.

When you’re ready to run the restore, all you’ll need to do is mount the tape and issue the LODRUN command. At that point, you custom restore program will take over and give you a quick, complete and controlled restore of your system on your recovery system. If you’d like to see a sample QINSTAPP program, send me an email (rich@kisco.com) and I’ll send you a shell that I recently created.