Tracking Remote Access Users On Your IBM i

By Rich Loeber

In the IBM i world, it has always been possible to track access to the system by user profile.  It is just a simple matter of activating and configuring the system audit journal and sitting back to wait for some information to be accumulated.  Then, using the DSPJRN judiciously, you can get a list of who signed on to your system and when they did so.

This was all well and good when your normal method of system access was through a green screen terminal, either hard wired or using a terminal emulator like IBM i Access.  But, in this day of network connections through TCP/IP server functions in the OS, the model doesn’t work well.  Users can log into your system using FTP or IBM i Access with little or no trace left in your security journal that they were even there.

In certain cases, an audit journal record for “Process user profile swap” is left (a type PS record in the journal).  I have seen this happen on a logon from IBM i Access using the TCP Signon Server.  Apparently, after a successful logon, IBM i Access does a user profile swap to the QUSER profile and this gets logged in the journal.  So, you could scan the journal for PS records and get some information about which user profiles are establishing network connections using this method.

You can also see some indication about remote users connecting to your system using IBM i Access (and similar clients) from the System History Log.  If you run the DSPLOG command, you can scan for the CPIAD09 message.  You can do this using the following command form:

DSPLOG PERIOD((*AVAIL *BEGIN)) MSGID(CPIAD09)

This will show you some evidence of remote logon activity too.

But these are just a drop in the bucket compared to everything that can go on within your system “under the covers”.  For example, if someone attempts to sign on to your system via FTP using a user profile that doesn’t even exist on your system, you will never even know about it even though you are journaling all security events.  Since this is a popular method that hackers use to try and break into your system, it would be nice to know when such attempts are being made.

When IBM opened up the system to TCP/IP connectivity, their solution to this issue was, and still is, to provide exit points in the OS.  For each of these servers (such as FTP, Telnet, SQL, TCP Signon and many more), the OS lets you create your own program to monitor network activity and even control access to it.

The problem with this approach is that it is entirely passive.  The OS is shipped with no exit program in place and the fact that exit points even exist is still not widely known.  In the mean time, all sorts of nefarious system connection activity can be going on and you, as security officer would never know it.  My company sells an exit point solution for the IBM i called SafeNet/i and the most common comment that we get from new customers who are just starting to use our product is that they had no idea so much activity was going on via network server connections.

Another problem with this approach is that coding and maintaining your own exit point solution is a daunting task.  Over the life of these exit points (they’ve been around for more than a dozen years now), some of the exit point data streams have changed significantly.  To IBM’s credit, they have left the old data stream in place and created a new exit point for the enhanced version, but you have to re-code your application to take advantage of the improvements when they are made.

The best solution is for you to lay out some money and purchase a good exit point solution.  There are a lot to choose from in the System i marketplace today including our SafeNet/i.  The maintenance problem then becomes your software vendor’s headache, not yours.  And, most solutions cover all of the exit points available so that you’re fully protected.  These exit point solutions give you control over which user profiles can access your system, in many cases, what objects they can work with and what source IP addresses they can connect from.  And, all network activity can be logged so that you can finally see everything that is going on “under the covers”.

If you have any questions about this topic, you can reach me at rich at kisco.com, I’ll give it my best shot.  All email messages will be answered.

Controlling Remote Access to your IBM i

By Rich Loeber

These days, there is almost nowhere you can go on the planet where it is technically difficult to establish and internet connection and get beyond your current location.  We’ve all experienced this, especially with demands made on us during vacation trips and other travel requirements.  It seems like we’re always expected to be on call and available to react to issues back in the office.

While this sounds intrusive of our time, there are other situations where it is a huge benefit.  It allows people who are spread out all over the world to easily collaborate on projects just like they are in adjoining rooms.  It also lets you telecommute from home or even telecommute from a home office that is a long way from your physical office.  This gives you new options in choosing where to live, and it doesn’t always have to be near where you’re working.  It is just this kind of global connectivity that lets me live in a remote location in the Adirondack Mountains of northern New York and still be able to reach out to any potential customers on a worldwide basis.

The thing that makes all this possible is the Internet and remote access technology.  But, as you think about it, while you have access remotely to off site computing resources, so does anyone else who is connected to the Internet.  That is a huge security exposure to your IBM System i.  Even the much touted security on this exceptional system will have exposures.

Certainly, there are loads of infrastructure solutions that you can turn to.  There are no end of VPN (Virtual Private Network) solutions that you can implement that will create tunnels through the Internet to authenticate a connection, limit access and encrypt your data.  There are also firewalls you can implement that will limit traffic connections to your system.

But, these are all external to your IBM i.  What can you do, on your system, to protect yourself.  You need to be able to permit external access from users with legitimate needs but keep all others from doing the same.  There are so many different methods to establish remote access that you have to have a plan for each of them.  From simple FTP to Telnet, ODBC connections, IBM i Access connections, remote command submissions, file uploads and downloads, windows share drive connections and on and on.  The list is long and each one represents an exposure.

IBM has provided a good solution for this issue for more than 15 years now, system exit points.  The OS on your IBM i services each of these remote access requests through something called a Server Function.  FTP works through an FTP Server, Telnet through a Telnet Server, ODBC on most systems works through an SQL Server and so on.  Within the OS, IBM has implemented exit points in each of these server functions.

An exit point is a place in the OS where you can register a user written program to add processing that is not included in the OS.  The exit point passes a data stream that varies depending on the exit point, and your program sets a return code for the exit point in the OS to interpret.  The return code is typically a pass/fail code that lets the exit in the OS know whether or not to continue processing the remote access request.  Your program then examines the information in the data stream to determine if you want to allow or disallow the remote request.

A typical exit point program application might attach, for example, to the FTP server logon exit point and check the remote IP address of the FTP requester.  You can then easily compare this to a list of legitimate addresses that you have set up and then disallow remote access from any IP address that is not on your list.  Continuing with the FTP example, you can also create an exit program application for the FTP server point and then check the data stream to see which objects the FTP session is attempting to access on your system.  Again, you can compare these to a list of objects that the requesting user is trying access and the either accept for deny the request based on what you find.  You can even authenticate the user signed on to make sure that you have authorized them to even use FTP.

A word of caution is due at this point.  Programming for the exit point data stream is not for the faint of heart.  The documentation on certain exit points can be sketchy and testing can be problematic as it can easily interfere with normal operations on your system.  Fortunately, there are quite a few third party solutions available on the market today that are very good.  All that testing and figuring out of the data streams is done for you by programmers who have been wrestling with the issues for years.  I obviously recommend my company’s SafeNet/i as the best of the third party solutions currently available.

To give your IBM i the extra measure of control over remote access via network connections, the best solution is implementation of the exit point controls in the OS.

If you have any questions about this topic, you can reach me at rich at kisco.com, I’ll give it my best shot.  All email messages will be answered.

Hacking Report For Our IBM i – 2013 in Review

By Rich Loeber

In January of 2013 year, we issued our first Hacking Report for our IBM i system.  At that time, I promised to publish additional reports of what we are seeing on this test server.  This is our final report for the 2013 and I want to wrap of this one year experiment with our fourth quarter results and some observations about the entire year.

During the final three month period for 2013 we observed a hacking results that were remarkably consistent with the three previous quarters.  The bulk of hacking attempts mounted against our test server were once again in the area of FTP Signon and Telnet Signon.

Once again, someone knocked on the door of our test server 14 times a day trying to gain access.  This attack rate was remarkably consistent for the entire year.

Some interesting things to note ….

Thanks to our SafeNet/i exit point control software, we successfully thwarted 847 attempts to gain access via FTP and another 428 attempts to get a Telnet signon session during the final quarter of the year.   For the year, we was almost 4000 FTP hack attempts and just under 2000 Telnet tries.  The take home lesson is that you absolutely must take life on the Internet seriously.  Our server is small potatoes and does not have any high value assets on it, but hackers are there knocking at the door on a regular basis anyway.  I think it is just because it is there and they have been unsuccessful.

Brute force FTP attacks continued during the final quarter.  Once again, the profile named ADMINSTRA was the most popular one used.  In fact, this was true for each quarter during the year.  Other profile names used included ADMIN, REMOTE, SCANNER, SYS, BACKUPEXEC and WWW-DATA.  Once again, and this was consistent during the whole year, none of the typical Q profiles were attempted.

We also continue to see certain IP addresses with repeated access attempts.  The leading violator for the final quarter traced back to The RIPE network in The Netherlands.  The next two highest both traced back to the Asia Pacific Network Information Center in Australia.

For the full year, our server posted close to 1 million network transactions.  This is nothing in today’s computing environment, some of our customer’s servers can record that level of activity in just a few minutes.  But, taken as a whole for the year, 0.5% of those network access attempts were not authorized by us.  Hackers, you have to take them seriously.  Failure to do so will get you in the headlines as the next Target.

This will conclude our year of tracking hacker activity on our server.  If you have questions about details of the report, feel free to contact me directly by email (rich at kisco.com).

And More Tips for Securing FTP on your IBM i

By Rich Loeber

For the last few months, I’ve been writing about the security exposure from FTP server running on your IBM i box (Power System i/i5/iSeries/AS400).  I started by describing FTP as being the access weapon of choice by many hackers, especially those just starting out.  This was confirmed by our most recent Hacker Report.  Last time around, I talked about a couple of simple suggestions on how to get this situation under control.  Today, I’ll add a couple more tips into the mix to help keep a lid on safeguarding your system from an FTP intrusion.

First, there are lots of good reasons why you want to allow FTP access to your system.  It is an easy way to upload and download data to and from your system from remote locations.  You can also use it for program maintenance from one IBM i to another by moving save files between systems.  Many IBM i software vendors, including my company, distribute software updates using some form of an FTP connection to your system.  So, don’t be afraid of it, but use it wisely.  You can see my last two articles on FTP here and here.

One thing to keep in mind when thinking about FTP is that all the rules of OS security apply to someone connecting to your system.  In order to gain access, they must have a valid user profile and password.  Once they sign on, your current OS security plan will be in place.  So, having a good security implementation tied in to your established user profiles will go a long way towards keeping your data safe.  One additional fact to add into your mix is that in order for data to be accessible to FTP, it must have a minimum security setting of *USE.  If you have a user profile that is regularly using FTP and there are concerns about access, make sure that they do not have a minimum setting of *USE for any objects you do not want them working with.

A problem can easily come up, however, when a user profile is used in different contexts.  By this, I mean when a user has access to certain sensitive objects for their daily work flow that are accessed by program control.  But, that user is also an FTP user and logs in to do file transfers using FTP.  Having different contexts could create a security exposure.  When this user signs on using FTP, they will still have access to the sensitive data files that they are authorized for from their daily work flow.  If this situation exists, you need to address a way to deal with it.

One method, as discussed last time around, might be addressed by implementing controls through the FTP server exit point.  You might also think to issue a second user profile to the user for FTP use.  This solution is not great since the user can still, by choice, establish an FTP connection under their primary user profile and gain access to sensitive data that way.  Far and away, the best solution is through additional exit point controls.  This could be set up to disallow an FTP connection under certain known profiles, thereby forcing the user to make their FTP connection through a secondary profile that you provide.  If you don’t want to tackle implementing your own exit point programming, there are several products available on the market from IBM i developers, including SafeNet/i from my company.

The Sytem i OS also supports profile swapping, which could be another solution to this problem.  Using swapping, the user signs on with one profile, but then the OS swaps their profile to look and act like a different profile.  Information about this technique can be found at the IBM Information Center and has been a part of the OS since V4R5.

If you have any questions about this topic you can reach me at rich at kisco.com,  I’ll try to answer any questions you may have.  All email messages will be answered.

Hacking Report For Our IBM i – 3rd Qtr 2013

By Rich Loeber

In January of this year, Kisco Information Systems issued our first Hacking Report for our IBM i system.  At that time, I promised to publish additional reports of what we are seeing on this test server.  This is our report for the third quarter of 2013 and represents activity observed from July 1st through the end of September.

During this three month period, we observed a slight decrease in the volume of network transactions on our system, less than 1% in total.  At the same time, we saw a drop in the number of illegal access attempts that were rejected by our SafeNet/i software.  From 1,417 rejections last quarter, we saw an 8.4% decrease down to 1,209.  This represents an average of 14 unauthorized access attempts every day, down from 16 last quarter.

Life on the Internet continues to be unsafe.  Having someone knock on the door of my primary server 14 times a day trying to get in is not my idea of fun.  But this is what passes for “normal” in today’s internetworked world.

Some interesting things to note ….

On our server, the unauthorized access attempts continue to fall into two categories.  The miscreants attempt to access the system by FTP and Telnet.

SafeNet/i on our FTP Server on the IBM i OS rejected 792 access attempts which represents a decrease from last quarter.  During this time, the number of legitimate FTP access attempts was 746, so the unauthorized attempts exceeded the legitimate attempts.  We serve client requirements and software development needs using FTP, so we have to keep the FTP server active.  This is a clear warning message, however, that if you are keeping the FTP server active on your system, you really need to have access controls in place like those provided by SafeNet/i.

During this quarter, however, we saw that brute force FTP attacks using a large number of different common user profiles disappeared.  In its place, we are seeing repeated attempts to gain access using very common user profiles but cycling through multiple passwords on each attempt.  Using this method, the most popular user profiles used were ADMINISTRA, MYSQL, APACHE, TEST, TEST1, TEST12, TEST123 and WWW-DATA.  All of these profiles are common in the Unix world, so it appears that the IBM i platform is still not well recognized.

For unauthorized Telnet access attempts, we saw an increase in activity, nearly doubling, back to the level we observed during our first quarter report.  The access attempts via Telnet tend to come in single attempts or at the most, two or three successive attempts.  SafeNet/i captures these before an actual signon screen is presented, so they never get to the feature in IBM’s i OS that forces a profile to go inactive.  (The same is true for the way we are intercepting unauthorized FTP attempts.)

We also continue to see certain IP addresses with repeated access attempts.  The leading violator for this quarter traced back to Bright House Networks in Florida.  The next three highest all traced back to the Asia Pacific Network Information Center in Australia.

Another trend that we note that is disturbing is that for the 92 days covered by our study, there were only 5 days with no malicious activity.  That means that almost every day our server sits out there, someone is trying their hand at gaining access illegally.

We will continue to review our server’s status on a quarterly basis and report the results on our blog space.  If you have questions about details of the report, feel free to contact me directly by email (rich at kisco.com).

More Tips for Securing IBM i FTP

By Rich Loeber

A few weeks ago, I published a tip about the security exposure that FTP represents on your IBM i platform.  That tip has generated some interesting feedback along with some ideas from readers on how they address the issue.  This tip features some additional ideas for you to protect yourself from FTP abusers.

First and foremost is this.  If you don’t use FTP, or you only use it on rare occasions, then don’t leave the FTP server active on your system.  You can check to see if the FTP server function is active on your system by running the following command:

WRKACTJOB SBS(QSYSWRK) JOB(QTFT*)

Look for jobs listed named QTFTPnnnnn.  If FTP is active, you will find several of these jobs shown.  To turn the FTP server off, run the ENDTCPSVR command specifying the *FTP server option.  Most systems come from IBM with the FTP server set to start automatically whenever TCP/IP is started.  You can change this by running the Change FTP Attributes (CHGFTPA) command.  Prompt it with the F4 key and check the first parameter.  If it is set to *YES, then FTP is going to start automatically at every IPL.  Changing this to *NO will stop this from happening.

In our shop, we use FTP enough during the course of the day that we keep the FTP server up and active.  But, we have job scheduler entries in the system to turn it off at the end of the day and then restart it every morning.  That way, for 24 hours of possible exposure, 16 of those hours are completely protected.  On the rare occasion when we need FTP during off hours, it is a simple matter to log in and start it again manually.

If the FTP server is inactive, then it cannot be misused.

The other good way to protect yourself from FTP abuse is through the implementation of exit point programs.  The FTP server has an exit point that can be used to filter incoming requests.  This is also true of the Telnet server, another point of possible abuse.  One reader of my last tip suggested implementing the freeware SECTCP utility written by the former IBMer Giovanni B. Perotti.  This utility is available for free download, after a simple registration process, from the following website:

http://www.easy400.net/easy400p/downloads.html#d09

I have downloaded and reviewed this code, but have not implemented since I have my own exit point software already active.  But, the reader swears by the code and Mr. Perotti certainly has a terrific reputation in the IBM i family of users.  So, if you’ve been thinking about implementing exit point controls, this might be any easy entry point for getting started.  The source code is all included with the download and, in fact, everything needs to be compiled in order to install the software.  The user instructions on getting started all appear to be fairly simple.

Also, if you don’t want the bother of maintaining your own exit point code, there are quite a few very good products available from reputable IBM i software developers today.  FTP and Telnet controls are just the tip of the iceberg where exit programming for security is concerned.  I, of course, recommend my own product: SafeNet/i.

If you have any questions about this topic you can reach me at rich at kisco.com,  I’ll try to answer any questions you may have.  All email messages will be answered.

Watch Out For FTP On Your IBM i

By Rich Loeber

FTP (File Transfer Protocol) is a nice and easy way to communicate between systems.  With FTP on your IBM i, you can transfer files to other systems, including other IBM i’s, with ease.  You can also use it to move programs and files between systems, all with relative ease.  But, increasingly, FTP is also becoming the hackers weapon of choice when cruising the Internet.  And, with FTP’s QUOTE command (among others), a knowledgeable hacker could do some serious damage to your system.

I tell you this based on personal experience with my own IBM i.  In a recent 7 day period, I identified more than 1,500 attempts to sign-on to my system from people not authorized.  All of these were malicious hacking attempts.  When I did a Trace Route on many of these, they pointed back to source IP addresses in The Netherlands, China, Colombia, Russia and other parts unknown.  Some attempts would not even trace back successfully.

I initially thought that nobody would bother my system since it is just a numbered address with no DNS entry to make it easy to find.  But, this is clearly not the case.  Some hackers use automated attack programs to just cycle through entire IP address ranges, and these are the folks who regularly stop in at my system.

The method used, from my personal observation, is to break in using the same user profile, usually ADMINISTRATOR, trying a different password every few seconds.  They will often cycle through and retry the same password more than once.  I’ve observed one break-in artist try this 850 times in a row over a period of several hours.

I know all this about my system because I monitor all network traffic and track it using our exit point software.  We have our system configured to only permit FTP access from trusted IP addresses.  The list is, necessarily, very small.  This protects our system from malicious remote access via FTP.  Also, if a hacker were to get past this check (which they never have so far), our system has no default passwords, so trying to cycle through known IBM i profiles and default passwords will also end up in failure.

So, what can you do to protect your system from FTP attackers?

First, make sure that you don’t have any default passwords set up on your system.  Use the Analyze Default Passwords (ANZDFTPWD) command from the SECTOOLS menu for this.  Initially, run it with the *NONE option for the ACTION parameter just to get a listing.  Then, when you’ve reviewed the list, make sure that the profiles with default passwords have their passwords reset to either a different, unique password or are set to *NONE.

Next, implement some sort of IP packet testing to only accept FTP connections from trusted IP addresses.  You can do this like we do using an exit program attached to the FTP sign-on server.  Or, if you have a fairly recent version of the OS, you can use the IP packet filtering capabilities in IBM i Navigator.  This will let you allow known IP addresses, or address ranges, to access your system while keeping everyone else out.  When setting this up, make sure you keep an active connection to your system while you are testing so that you don’t accidentally shoot yourself in the foot and lock out all access to your system.  Remember, the IP packet filtering will apply to all users connecting to your system, not just FTP users, so this will be a bigger job than you may think starting out.

If you have any questions about this topic you can reach me at (rich at kisco dot com),  I’ll try to answer any questions you may have.  All email messages will be answered.

The Case For Better User Profile Names

By Rich Loeber

Ever since I started working in an environment where I needed a user profile and password (yes, there was a time when these were foreign concepts), I have always used a simple profile based on my first name.  As I moved from System/34 to System/36 to AS/400 to iSeries to IBM i on PowerServer, I just kept that same simple profile name.

But …. no longer.  I am now a firm believer that first name profiles are a very bad idea.  There are several reasons for this change.

First, recent studies show that people are really bad about picking secure passwords.  Current studies show that, unbelievably, the most commonly used password is “password”!  Check this for a recent list of the most commonly used passwords.

Since people are so bad about choosing passwords, having a user profile that is easy to guess makes a scripted attack very possible.  In a scripted attack, a hacker attempts to gain access to your system by trying a typical user profile in combination with the list of commonly used passwords.  If your system has simple first name profiles, then it could be at risk.  Even if the hacker does not gain access to your system, they could easily end up disabling a lot of your user profiles because of multiple logon failures.

There are two ways that a scripted attack is normally mounted.  The most obvious is via FTP.  In the scripted attack, a series of logon attempts is done using an automated FTP client.  Common profiles are coupled with common passwords and repeatedly tried to see which combination works.  When a match is found, the profile/password is reported back to the hacker so they can explore further at their leisure.

On our IBM i test box, FTP scripted attacks are all stopped before they get to the point where a logon is run.  This is because we use our SafeNet/i exit point security software that only lets FTP connections in from known and trusted IP addresses.  We collect the profiles used, however, so we can report to you on the most commonly used profiles that we are seeing.  From these FTP attempts, the most common user profiles used are:

ADMINISTRA
ADMIN
TEST
ALAN
ALBERTO
ALEX
AMANDA
ANGELA
ANNA
ANONYMOUS

Since we are protected by SafeNet/i, we were not too worried about the few common user profiles we had until a recent attack resulted in two profiles being disabled.  We quickly turned to our exit point log, but there was no record of the activity.  We also checked the system log (DSPLOG) but nothing reported there shed any light on why the profiles, including my trusted personal profile, got shut down.  As a last attempt to discover what happened, we turned to the system security audit journal.  What we found there was an eye opener.

We run the POP server on our IBM i.  We use it, in combination with the SMTP server, for sending outbound email from the system with our WebReport/i software.  This lets us send email without depending on any other office servers.  What we discovered from the system audit journal was that another form of scripted attack was taking place under the covers on the system with no knowledge on our part.  A little research revealed that we were being subjected to what is known as a “Brute Force POP3 Attack”.  It is like an FTP scripted attack, but broader in scope.

An analysis of the T-PW records recorded in the security audit journal showed that over a one month period, more than 20,000 attempts had been made to log into POP3 accounts.  Fortunately for us, we were protected by quite a few other best practices for IBM i security, not the least of which is that we do not use the POP server on our IBM i for inbound mail, so none of the mailboxes actually exist.  That, however, does not stop the hacker from trying.  And, since people tend to use the same password in all places, a user profile/password combination found at the POP server level could easily be tried to gain access to the system by other means.  From our analysis of these break-in attempts via the POP server, the most common user profiles are:

ADMIN
INFO
DATA
TEST
ROOT
MARK
SERVER
MAIL
MARY
SUPPORT
ORACLE
GUEST
DANIEL
GEORGE
ALEX

By looking at these, it is clear that the hacker or hackers are not aware of the type of system that they are trying to access.  None of these profiles are commonly used on the IBM i platform.  But, you can see some profiles that might easily be in existence on a normal system deployment which gives cause for concern.

What can you do?

Here are some simple ideas in brief:

  • Don’t keep the POP server active on your system if you don’t need it.
  • Don’t keep the FTP server active on your system if you don’t need it.  If you do need it, only have it active during hours when you expect it will be used and shut it down during other times.
  • Implement enforced password rotation if it is not already active.
  • Implement the user profile password rules to always require a numeric digit as a component of the password.
  • Review the active profiles on your system for simple first names and get them changed.
  • Check the common profiles used on most IBM i systems and make absolutely certain that their passwords are complex and hard or impossible to guess.
  • Implement exit point controls, check out our SafeNet/i product.
  • Consider disallowing vowels in your passwords.  IBM i system value password controls will let you do this.  At a minimum, rule our the letters E and A.
  • Check your system security audit journal regularly for T-PW records to see if you are getting unexpected password denials.

If you have any questions about this, or you need help with implementing any of these recommendations, feel free to contact me by email at rich @ kisco.com.  All email inquiries will be answered.

Watch Out When Restoring User Profiles

By Rich Loeber

I was recently confronted with an issue on one of our test IBM i boxes.  The box is implemented as a “warm” backup site for one of our customers.  Every night, a simple FTP of changed objects takes place from their server to our test box.  The theory is that the customer can afford to loose one day’s worth of processed transactions for the ease and relatively low expense of maintaining this sort of backup site.

The problem posed to me concerned their security configuration.  While their program base and data files were all being properly synced night after night, their passwords were growing woefully out of date on the backup site.  If they had an emergency, nobody would remember their older passwords.

I thought that the solution would be simple.  Just run a SAVSECDTA (Save Security Data) on their system, then restore it on our test box.  Before doing this, I compared the user profile base on both systems and found some areas for concern where the same user profile existed in both places.  Fortunately, the RSTUSRPRF (Restore User Profiles) command lets you either restore specific profiles or restore all profiles with an exclusion list.  I wanted to exclude all Q* profiles plus a handful of common profiles that exist on both systems.  My list ended up being fairly short.

I was all set to bring these profiles current, but when I tried the RSTUSRPRF, I was gently reminded that this command can only be run when the system is in restricted state.  Our test box hosts several websites including one that runs in secure HTTPS mode.  Thinking this would be a quick process, I shut down the system after notifying a few people that there would be a short interruption in service.  When the system came to restricted state, I ran the user profile restore which ran quickly and without any issues.

I then restarted the system, and here is where it got interesting.  At first, it looked like everything was fine, but I soon found that the web server instance that was using HTTPS was not restarting correctly.  After poking around for a few minutes, I found that it was objecting to the digital certificate that was specified for the site.  I fired up the Digital Certificate Manager (DCM) to see what was going on and the certificate looked just fine.  I decided to delete the certificate and re-add it, and here is where the train wreck was revealed.

When I tried to re-issue the certificate, I was advised that my password for the certificate store was invalid and that I would have to change the password before I could issue the new certificate.  I dutifully changed the password, but the same error kept coming back up.

After a long investigation period, I finally determined (with help from on-line friends) that the root of the problem was the RSTUSRPRF.  It turns out that the restore user profile process restores critical data and keys for the digital certificate store while restoring your profiles.  I now had this information from our client’s box, not ours.  None of the certificates on our system were valid any longer and additional applications on the test box were also failing because of this.

The solution was fairly simple.  You just have to restore the Digital Certificate Manager objects from a security backup from your system on top of the restore just done from the foreign system.  The RSTUSRPRF has the following format for this process:

RSTUSRPRF USRPRF(*NONE) SECDTA(*DCM)

It took me a while to find a current backup of our system’s security data (another story for another time).  As soon as I did this, then DCM would work correctly and the invalid digital certificates could all be re-created.  While our system was not technically down for this time period, several applications were knocked out for almost 6 hours.

Who would ever guess that restoring user profiles would end up hosing your digital certificate files?  A friend blames this all on IBM for such a kludgey implementation of DCM.  I have to say that I agree.  The save and restore of this certificate information should not be hidden along with the user profile save/restore.  The fix should be a new command from IBM for RSTSECDTA.  Let the user profile restore do just that and move other operations to a different command process.

Hacking Report For Our IBM i – 1st Qtr 2013

By Rich Loeber

For years, Kisco Information Systems has kept a lone test IBM i server hanging out directly on the Internet.  No firewall, no security appliances, just a direct connection with a dedicated IP address.

Not a very good idea you say?  Well, Kisco sells a network security software solution called SafeNet/i and what better environment to test and prove that the software works.  Using a combination of the best IBM i OS security practices along with a full implementation of SafeNet/i, Kisco is happy to report that their server has never been hacked successfully ever since this test server was first placed on the web more than 15 years ago.

That’s a good record!

But, that is not the purpose of this report.

To help IBM i shops understand the reality of network threats, we are now reporting some results of what we see on this test box.  We hope that it will help IBM i users to better prepare for the very real threats that exist.

This report shows what we’ve seen on our server during the first three months of 2013.

During this time, our test box reported 211,346 network transactions that passed through the various exit points registered to SafeNet/i.  Out of this total, 1,603 (0.75%) were identified as illegal access attempts and were denied.  That represents about 18 times each day when someone tried to gain access to our system, but was not authorized for that network activity.

Of these 1,603 access denials, all of them fell into just two categories during this test period.  FTP access connections accounted for 1,025 and the other 593 are Telnet connection attempts.  All of these connection attempts were refused by SafeNet/i before the requests even reached IBM’s OS.

A further look at these access denials shows that 1,015 of them came from user profiles that do not exist on the server.  The most popular profile, by far, was “ADMINSTRA” which accounted for 700 failed attempts.  The next most popular was “ALAN” with 37 followed by “TEST”, “ADMIN” and “ALBERTO”.  All of these were FTP connection attempts.  These all appear to be FTP script connection attempts, probably cycling through a series of popular password combinations.  It argues strongly for user profiles on the IBM i that are not based on people’s first names or job functions.

Looking at the access denials from a different perspective, we see that all 1,603 during this test period were denied access because they originated from IP addresses not recognized by SafeNet/i.  For FTP and Telnet connection, SafeNet/i only allows a connection to be established when the IP address is recognized.  By carefully maintaining the table of legitimate connectors to the system, illegal connection attempts are controlled.

Of these illegal connection attempts, 49 source addresses tried to connect multiple times.  The most persistent tried to connect 367 times in succession, all of which were denied.  There were others from different source IP addresses who attempted to connect 271 times, 195 times and 118 times.  Some only tried twice.

Are these illegal connection attempts really something to be concerned about?  To check this, we did a reverse lookup on the most common IP addresses that were denied.  Two of the addresses checked back to an ISP in Brisbane, Australia.  Two others were tied to ISPs in Scranton, PA and Galloway, NJ neither of which are associated with any known developers that we normally work with on this server.  The obvious conclusion was that these access attempts were malevolent which is all the more troubling since the IP address of our server is not generally known to the public.

During this study period, 18 valid source IP addresses connected over and over again to get their legitimate network work completed.

For those attempting Telnet connections, the pattern is a little different.  Within the IBM i OS, all of these failed attempts are logged under the common user profile of QSYS.  Telnet attempts, however, do not follow the brute force attempts that FTP users try.  They tend to be solo connection attempts or just two in a row.

Kisco Information Systems will keep an eye on these connections attempts and will periodically issue updates on the results by quarter.  Feel free to check back to our IBM i Security Blog for future reports.  If you have questions about details of the report, feel free to contact me directly by email (rich at kisco.com).