Resolve AIX error “Cannot open or remove a file containing a running program”

Situation: When trying to replace an IHS (IBM HTTP Server) module we were getting the following error, even though the web server had been stopped:

./ Cannot open or remove a file containing a running program.
Environment: IHS 6.1 / WebSphere 6.1 on AIX 6.1.
Solution: The issue was resolved by using the slibclean command. According to the manual,
The slibclean command unloads all object files with load and use counts of 0. It can also be used to remove object files that are no longer used from both the shared library region and in the shared library and kernel text regions by removing object files that are no longer required.
This file could be located in the /usr/sbin/ folder.

Apache Access Configuration and Errors

To set up access restriction using Apache’s built-in access configuration feature you would need to update three files:

  1. Main Apache configuration file (httpd.conf): Make sure the Directory section of the file that refers to your website folder has “AllowOverride All” or at least “AllowOverride AuthConfig”. For example,

    <Directory “/home/websites/securedsite”>
    AllowOverride AuthConfig

  2. Password file (.htpasswd): You can use the Apache’s built-in htpasswd tool to encrypt passwords. This utility should be located under the /bin/ folder. Type in “htpasswd” for all the options and more help on this tool. For each user a line such as “user1:RsQvdHxiKcH5w” should be added to your password file. The password file can have any arbitrary name and would be normally located in the top-most folder of your website that is supposed to be password protected. Give read permission to this file.
  3. Access Configuration File (.htaccess): This file sets up the authentication for our purpose. It can in fact contain any allowed customization at the folder level. Here is an example:
    AuthUserFile /home/websites/securedsite/.htpasswd
    AuthType Basic
    AuthName “Restricted Website”
    require valid-user

Some errors (in your Apache error log) that you may come across while setting up this access control include:

  • configuration error:  couldn’t perform authentication. AuthType not set!
    This tells you that “AuthType” parameter is not set in your .htaccess file.
  • Permission denied: /home/websites/securedsite/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable
    The file .htaccess does not have the proper read permission.
  • Permission denied: Could not open password file: /home/websites/securedsite/.htpasswd
    The file .htpasswd does not have the proper read permission.
  • No such file or directory: Could not open password file: /home/websites/securedsite/.htpaswd
    The path to the .htpasswd file is not correct. A typo maybe?
  • configuration error:  couldn’t check user.  No user file?
    This probably tells you that the AuthType is not properly set (in my case, I had to change it from “Digest” to “Basic”).

ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414)

Situation: We were receiving “500 Internal Server Errors” for HTTPS hits to all our WebSphere applications. The following errors appeared in the plug-in log file of our IBM HTTP Server:

ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414) ….
ERROR: ws_common: websphereGetStream: Could not open stream
ERROR: ws_common: websphereExecute: Failed to create the stream
ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to ‘xxx’on host ‘yyy’; will try another one
ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
ERROR: ESI: getResponse: failed to get response: rc = 2
ERROR: ws_common: websphereHandleRequest: Failed to handle request
Environment: IHS 6.1 / WebSphere 6.1 on AIX 5.3.
Solution: This issue has been addressed in this IBM article, however what really helped us resolve the issue was the steps explained in this post.
As explained in this post, the errors were because the WAS SSL certificate was not trusted by the WAS plugin configured for IHS.
These are the steps we followed:
Extract the default Personal Certificate:
1. Login to the WebSphere Application Server Administrative Console.
2. Select Security > SSL certificate and key management > Key Stores and certificates.
3. Select NodeDefaultKeyStore for a stand-alone deployment or CellDefaultKeyStore for a network deployment.
4. Click Personal Certificates, select the default check box, and then click Extract.
5. Give the extracted file a path and name, such as: /root/defaultCert.arm. Note: The convention is to give the file a .arm extension.
6. Leave encoding set to Base64.
7. Click OK.
Locate your *.kdb file:
1. In the httpd.conf file, find the directory in which the plugin-cfg.xml file is stored by searching for the WebSpherePluginConfig line. It should look something like this:
WebSpherePluginConfig “/opt/IBM/HTTPServer/Plugins1/config/webserver1/plugin-cfg.xml”
2. Find the directory in which the key database file (*.kdb) is stored by searching for the term “keyring” in the plugin-cfg.xml file.For example:
<Property Name=”keyring” Value=”/opt/IBM/HTTPServer/Plugins1/config/webserver1/plugin-key.kdb”/>
Note this location as you will need to use it later.
Add the extracted certificate to your key database file:
1. Go to the directory for ikeyman and start it:
cd /opt/IBM/HTTPServer/bin
You would need an X-Window server to perform this step. We used Xming.
2. Click Key Database File > Open, and then select a key database type of CMS.
3. Specify the filename and location you found above. For example: plugin-key.kdb and /opt/IBM/HTTPServer/Plugins1/config/webserver1/plugin-key.kdb.
4. Click OK, and then enter the password. Note: If you have not given this file another password, the default password from WebSphere Application Server is “WebAS” (case sensitive).
5. Click Personal Certificates drop down and then select Signer Certificates.
6. Click Add.
7. Browse to the file you exported with the extension *.arm, Select it, then Open and click OK. Supply a name if prompted.
8. Select Key Database File > Save As and save to the original location.
9. Select Key Database File > Exit.
10. Restart the IBM HTTP Server.