wiki
Wiki: SFEE Server Administration FAQ

SFEE Server Administration FAQ

SFEE Community Home -- FAQ Index -- Find Page

SourceForge Installation and Administration

What is the recommended installation for SourceForge Enterprise Edition?

?See wiki page on installation

How do I backup SourceForge Enterprise Edition?

SFEE has essentially 4 data components to it that require the Sys Admin to proactively backup in case of system failure. These components are:

• the database
• the filestorage
• the search indexes
• the scm data

As with most large and complex applications, the recommended method of backing up SFEE involves shutting down the application. While it is possible to backup the application while it is up, the backup itself will not provide 100% data consistency. You should undertake one of these so-called 'live' backups only as a last resort and only after ensuring all the potential consequences are fully understood.

If, for whatever reason, a full, offline backup is not possible, you should do the following to ensure that as little data is being changed as possible. While these steps won't completely replicate the offline backup, they can mitigate most of the issues w/ doing a live backup. Again, we do NOT recommend this. Offline backups are your friend. Just schedule the downtime and take the hit.

QUIETING THE SYSTEM

First, you'll want to turn off the web server so users can't get to the UI (assuming you are routing through Apache and not direct to JBOSS):

service httpd stop

Second, on the SCM server, remove the execute bits on the SCM:

chmod -x /usr/bin/cvs 
chmod -x /usr/bin/svn

This will prevent anyone from executing 'cvs' or 'svn' for new checkins/checkouts/tags. Please note however, that it won't affect existing running processes. Either wait for them to finish, or kill them.

Finally, pause the search indexing engine:

touch /usr/local/sourceforge/var/searchIndex/LOCK_INDEXES

THE DATABASE

The majority of SFEE data is stored in the database. To backup the database, follow the recommended procedure from your db vendor. If you're using PostgreSQL, you can use the directions outlined in the FAQ entry called "How do I backup my ?SourceForge PostgreSQL database?"

THE FILESTORAGE AND SEARCH INDEXES

Any documents uploaded to SFEE, or attachments to an artifact or forum as well as the search indexes are stored on the filesystem as normal system files. As such, please use whatever normal backup method you use to ensure filesystem restorability (tar, dd, cpio, Ghost, etc). You should backup the following directory and ALL its subdirectories:

/usr/local/sourceforge/var

THE SCM DATA

On the SCM box, you will also need to backup your SCM data. This data is contained in your companies various repositories which are located under:

/cvsroot

and

/svnroot

For the CVS repositories, you should use whatever normal filesystem backup method your company prefers, much like the FILESTORAGE section above. For SVN repos, it is highly advised that you use the 'svn dump' action to export them to normal files and then backup those dump files. To do this, simply:

svnadmin dump /svnroot/my_repo > /some/backup/path/my_repo.dmp

Please note that due the nature of SFEE, there are several interdependencies between these various data storage points. As such, you must take great care to ensure that all these components are backed up AT THE SAME TIME. You will NOT be able to use a database backed up yesterday morning with a ~filestorage from the night before.

Once the backup is complete, remove the file:

/usr/local/sourceforge/var/searchIndex/LOCK_INDEXES

Turn on the execute permissions for the SCM binaries:

chmod +x /usr/bin/cvs 
chmod +x /usr/bin/svn

and restart Apache:

service httpd restart

How do I determine which version of SourceForge is currently running?

You can determine the version of ?SourceForge you are running by looking in one of the following areas:

Click on the question mark icon in the upper right corner of the UI. You should see a drop-down menu with an "About ?SourceForge" option. Selecting that option will present you with the version number.

The version number is also stored in the following files:

• $SOURCEFORGE_HOME/etc/version
• $SOURCEFORGE_HOME/etc/integration-version

Apache Administration and Tuning

Can I improve performance with mod_deflate

Apache 2.x ships with the mod_deflate module which allows you to compress the content before sending it over the wire to the user. This results in less data to transfer and less bandwidth consumption, which improves the user's performance experience. To enable this feature, add the following to either httpd.conf or ssl.conf (depending on if you're running your site with HTTPS or not):

# Turn on mod_deflate
DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio
LogFormat '"%r" %[outstream]n/%[instream]n  [%[ratio]n%%]' deflate
CustomLog logs/deflate_log deflate
<Location />
   SetOutputFilter DEFLATE
   BrowserMatch ^Mozilla/4 gzip-only-text/html
   BrowserMatch ^Mozilla/4.0[678] no-gzip
   BrowserMatch bMSI[E] !no-gzip !gzip-only-text/html
   SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
   Header append Vary User-Agent env=!dont-vary
</Location>

Once Apache has been restarted, your sites content will be compressed for those browsers that support it (except for images).

How do I generate SSL Certificates for my SFEE Install

In order to use HTTPS with your SFEE install, you need to have a valid SSL certificate installed for Apache to use. There are two (2) kinds of certificates: self-signed, and CA-signed. Self-signed certs are ones that you yourself generate and 'authorize' and cost you nothing (though some browsers/applications will not like them much). A CA-signed certifcate is a certifacte authorized by a Certificate Authority (like Verisign). They can cost several hundreds of dollars, but every browser/application recognizes the CA cert.

In either case, you first have to generate what is known as a CSR, or certificate signing request. To generate the csr, do the following:

• login as root on the SFEE server
• generate your server's key: openssl genrsa -out servername.key 1024
• generate the csr itself: openssl req -new -key servername.key -out servername.csr

You now have the private server key, and the csr. Whatever you do, do NOT lose the key! The csr and the resulting certificate will be tied to it and will not function if you lose it or try to replace it! If you are getting an 'official' CA-signed cert, you will provide them with the servername.csr file and they will reply (eventually) with the cert. If you are doing a self-signed cert, create it like:

openssl req -new -key servername.key -x509 -out servername.crt

You should now have all three (3) pieces needed to make Apache use SSL. Simply edit /etc/httpd/conf.d/ssl.conf and look at the SSL setting to determine where to place the appropriate files and under what name. Once this is done, simply restart Apache:

service httpd restart

NOTES: You should replace 'servername' above with the real domain name of the server in question, e.g. sourceforge.mycompany.com. You also need to be very precise in answering the questions when generating the CSR. Most CA-signing companies will refuse it if the information is not correct. And finally, when building the CSR, it will ask for the 'common name'. You should reply with the real domain name of the server, e.g. sourceforge.mycompany.com. This value will be built into the cert and, if incorrect, will cause browsers to complain that the cert doesn't match the hostname and your users will have to acknowledge it every time.

Can I improve user performance with mod_expires

Apache 2.x ships with a module called mod_expires that allows you to specify the Content-Expires header by file type. You can improve the perceived performance of ?SourceForge for your users by utilizing this module to allow caching of the various static content pieces in ?SourceForge. Simply add the following lines to httpd.conf and ssl.conf :

# mod_expires configuration
ExpiresActive On
ExpiresDefault "access plus 0 seconds"
ExpiresByType image/gif "access plus 1 days"
ExpiresByType image/jpeg "access plus 1 days"
ExpiresByType image/png "access plus 1 days"
ExpiresByType text/css "access plus 1 days"
ExpiresByType text/javascript "access plus 1 days"
ExpiresByType application/x-javascript "access plus 1 days"
ExpiresByType image/x-icon "access plus 1 days"

Header append Cache-Control public

Once Apache is restarted, these changes will take affect. If it's not obvious what we're doing, we are specifying a 'default' of immediate expiration. This is needed so that your changes to pages in ?SourceForge aren't cached by the browser and you get apparent 'weird' behavior. The next set of lines tell the browser to cache and GIF/JPG/PNG images as well as any Javascript and CSS files for 1 day. This value can be adjusted up as needed in your environment. Finally, we tell the browser to cache the favicon.ico file.

Database Administration and Tuning

Maintaining PostgreSQL's optimal performance by vacuuming the database

PostgreSQL maintains its performance over time when properly tuned. However, do to the nature of SFEE and its 'never delete anything in the database' policy, it is necessary to help PostgreSQL maintain optimal performance by periodically running vacuumdb.

We recommend you create ~postgres/vacuum.sh as:

#!/bin/bash
# SFEE Vacuum and Analyze Script.  This script, as described in the knowledgebase, 
# vacuums and analyzes your postgresql database to ensure proper database health.  
# This is not a supported script, but provided by CollabNet for use and enjoyment in the 
# community.  
# NOTE:  Please change the DBNAME variable below if your database name isn't sfee.
DBNAME="sfee"
do_vacuum()
{
    for table in $*
    do
        echo "analyze $table" | /usr/bin/psql $DBNAME
        echo "vacuum $table" | /usr/bin/psql $DBNAME
    done
}
HIGH_PRI_TABLES="audit_change mntr_subscription"
MED_PRI_TABLES="artifact item audit_entry document document_folder folder"
LOW_PRI_TABLES=`echo "\d"|/usr/bin/psql $DBNAME|grep table|awk '{print $3}'|egrep -v 'artifact|item|audit_change|audit_entry|mntr_subscription|document|document_folder|folder|objecttype'`
WHICH_PRI=$1
case $WHICH_PRI in
    HI)
        do_vacuum $HIGH_PRI_TABLES
        ;;
    MED)
        do_vacuum $MED_PRI_TABLES
        ;;
    LOW)
        do_vacuum $LOW_PRI_TABLES
        ;;
esac
exit 0

and then create the following crontab entries for the postgres user:

03,18,33,48 * * * * $HOME/vacuum.sh HI 
08 01,07,13,19 * * * $HOME/vacuum.sh MED
11 04,16 * * * $HOME/vacuum.sh LOW

Once this is in place, your ~PostgreSQL db should remain at its optimal performance level.

NOTE: We are assuming you've already tweaked the ~PostgreSQL settings as per our article on the same. We are also assuming that you've disabled the statement_timeout setting for the postgres user, like so:

ALTER USER postgres SET statement_timeout = 0;

How do I backup my SourceForge PostgreSQL database

Databases cannot be backed up as part of your normal OS backup as the database always has file handles 'open' which prevent the backup software from properly dealing with them and the database may also have data in memory that it has not yet flushed to disk. As such, you need to actually use a database specific tool for backing up the database to the filesystem (which can then be backed up as part of the OS backup). Fortunately, PostgreSQL includes such a utility as part of the included software. Simply create yourself a cronjob that runs the following as the postgres user:

pg_dump -Fc -b sfee | gzip -9 > sfee_backup.tar.gz

Obviously, you can adjust the filename to include a date/time stamp or other information. The backup is mostly transparent, but it does include some overhead on the database, so you should schedule them as frequently as possible without negatively impacting the performance of the database.

How do I restore my SourceForge PostgreSQL database

If you ever have the need to recover your database, you can use the pg_restore utility that ship with PostgreSQL to restore your database (assuming you are using pg_dump to back it up). Follow these simple steps:

• recreate the db user

 createuser -P sfee

• recreate the db

 createdb -E UNICODE -O sfee sfdb

• restore your data

 pg_restore -d sfdb /restore/sfdb.tar 

Once you have restored the database, it is a good idea to manually vacuum it before turning ?SourceForge back on.

How do I upgrade my PostgresSQL Software

Upgrading your PostgreSQL database is fairly straightforward. In fact, if you are using the rpms provided by your OS vendor (and you should be) then upgrading will be handled automatically (in most cases) by their system upgrades. If, for whatever reason, you are not using the rpms provided by the OS vendor, or you're just super-paranoid about it then the following are the steps to ensure a smooth upgrade.

First, we're gonna take a backup of the database:

su - postgres
pg_dump -Ft -b -o your_db > $HOME/pre_upgrade_your_db.tar
exit

Now, we stop PostgreSQL:

su -
service postgresql stop

Now, we copy our existing config files (if you're upgrading in the same series, i.e. 7.4.10 to 7.4.17 say. This will NOT work for upgrading from like 7.4.10 to 8.1):

mv ~postgres/data/{pg_hba,postgresql}.conf /tmp

Next, we remove the existing PostgreSQL rpms:

rpm -q -a|grep postgr|while read line; do rpm -e $line; done

Next, we remove the database from the disk

/bin/rm -r /var/lib/pgsql/data/*

Now, re install the new rpms:

rpm -Uvh path_to_rpms>/*.rpm

Next, we initialize the PostgreSQL 'internal' databases:

/etc/init.d/postgresql initdb
exit

Now, we put the config files back:

mv /tmp/{pg_hba,postgresql}.conf ~postgres/data

Next, we start the PostgreSQL server:

su -
service postgresql start
exit

Finally, we put our old db back:

su - postgres
createuser -W sfee_db_owner
createdb -E UNICODE -O sfee_db_owner your_db
pg_restore -d your_db $HOME/pre_upgrade_your_db.tar
exit

Oracle can't connect to my SFEE Installation

?SourceForge Enterprise Edition uses the thick Oracle JDBC driver, which is comprised of two parts. One of these is provided by SFEE, the other is in $ORACLE_HOME. If these two different components are incompatible, SFEE will be unable to make a connection to the database. The simplest way to correct this is to overwrite the .jar included with SFEE with the one from $ORACLE_HOME. This can be accomplished as follows:

cp $ORACLE_HOME/jdbc/lib/ojdbc14.jar /usr/local/sourceforge/jboss/jboss-3.2.6/server/default/lib/

A restart of the application will be required to use the new .jar.

James

I have made changes to the James config, how can I enable them without restarting SFEE?

You can restart James without restarting all of SFEE with the following command:

/usr/local/sourceforge/james/james-<version>/bin/phoenix.sh restart

Please keep in mind, that any changes made to the James config that are not stored in the SFEE configuration, may be overwritten upon future upgrades.

james-<date and time>.log

If you find that ?SourceForge starts up normally, but email and search don't come up, or immediately die, you should check the log at:

/usr/local/sourceforge/james/james-<version>/apps/james/logs/james-<date and time>.log

This log will show the bootstrap output for James and any issues encountered therein.

smtpserver-<date and time>.log

James records every incoming SMTP connection attempt to:

/usr/local/sourceforge/james/james-<version>/apps/james/logs/smtpserver-<date and time>.log

If you are fielding reports from users claiming that they are sending mail to ?SourceForge and the email is not showing up in the application, you should start your troubleshooting here as this log will show if the user's mail server is even attempting to connect to James to deliver the email.

mailet-<date and time>.log

James records its general email processing info for received and sent email in:

/usr/local/sourceforge/james/james-<version>/apps/james/logs/mailet-<date and time>.log

For a general overview of the email processing, this is the log to check.

dnsserver-<date and time>.log

James records all errors related to resolving DNS for outbound mail to:

/usr/local/sourceforge/james/james-<version>/apps/james/logs/dnsserver-<date and time>.log

If you find that some of your ?SourceForge users are receiving email, but significant groups of others are not, you should consult this log to determine if James is experiencing difficulties in resolving their domain or MX records.

remotemanager-<date and time>.log

James can be administratively controlled by access its management port and authenticating with the proper credentials. If you've firewalled your ?SourceForge installation as instructed, this port should not be available to access, however, it would be good security practices to review the remote manager log at:

/usr/local/sourceforge/james/james-/apps/james/log/remotemanager-<date and time>.log

Any remote logins to this port will be logged here and should be acted upon immediately.

JBOSS

boot.log

If ?SourceForge is failing to start up, or is starting but is throwing errors on every page, then (typically) something went wrong during the JBOSS bootstrap process. Fortunately, JBOSS logs this process to:

/usr/local/sourceforge/jboss/jboss-<version>/server/default/log/boot.log

Search (Phoenix)

phoenix.log

?SourceForge uses a product called Phoenix to manage its search services. Phoenix is, itself, managed by James. If you find that ?SourceForge starts up without issue, but then search throws error, or dies, you should examine the Phoenix log for issues:

/usr/local/sourceforge/james/james-<version>/logs/phoenix.log

It may be necessary to 'bounce' Phoenix to get a 'clean' log. You can do this without restarting all of ?SourceForge by executing:

/usr/local/sourceforge/james/james-<version>/bin/phoenix.sh restart

default-<date and time>.log

Every item that the Phoenix search engine indexes, is logged to:

/usr/local/sourceforge/james/james-<version>/apps/search/default-<date and time>.log

If you are concerned that new items are not showing up in your search results, or that updated existing items are not showing up in search results based on the changes to the item, you should 'tail' this log while updating the item. If there are issues indexing the item, they will be logged to this file. This file is also the place to start your debugging when search is up, but not functioning as expected.

SearchReindex.sh fails on SFEE 4.4

Running them ?SearchReindex.sh script on a clean SFEE 4.4 install fails with the following message:  Soap43 SDK must exist: /usr/local/sourceforge/sourceforge_home/sfsoap/sf_43_sdk/soap/lib 

This occurs as the search indexer included with SFEE 4.4, requires the 4.3 java SDK. You can obtain the java 4.3 SDK here: https://sfee.open.collab.net/sf/go/rel2014 This file should be extracted to /usr/local/sourceforge/sourceforge_home/sfsoap assuming you installed SFEE into /usr/local/sourceforge.

After installing the 4.3 SDK, you may receive the following error:

Could not login (; nested exception is:
        java.lang.NullPointerException)

This is due to an older version of axis.jar being included in the 43 SDK. You can correct this by copying a newer version of axis.jar into the sf_43_sdk directory as such:  cp /usr/local/sourceforge/sourceforge_home/lib/axis.jar /usr/local/sourceforge/sourceforge_home/sfsoap/sf_43_sdk/soap/lib/ 

You may need to substitute '/usr/local/sourceforge' for the location of your SFEE install.

SourceForge Application Server

session-info.log

?SourceForge records information about every user session and ever 'page request' it serves to users in the file:

/usr/local/sourceforge/log/session-info.log

Generally speaking, this not is of little value in debugging any issues except, occasionally, performance issues.

vamessages.log

?SourceForge records just about everything it's doing to the following file:

/usr/local/sourceforge/log/vamessages.log

You can use this log to debug startup issues, performance issues, system errors, exid backtraces, JVM issues, SQL issues, etc. Since this log contains so many different types of log messages, it grows extremely rapidly. As such, the file is automatically rotated by ?SourceForge when it reaches 100M and ?SourceForge will keep the previous 10 copies of the log. If you are having any kind of issue, this is probably the log you want to look at.

server.log

?SourceForge writes its startup and shutdown info, as well as any system errors to the following file:

/usr/local/soureforge/log/server.log

If ?SourceForge is not starting up properly, but JBOSS is starting up without issue, then you should review this file. If you or a user encounters a 'system error' while using ?SourceForge, it is logged here. Additionally, if you or a user are seeing an 'exid' string in the application, the Java stack trace for that exid will be logged here.

Tomcat

catalina.out

The ?SourceForge SCM Integration server runs an instance of Tomcat and then launches ?SourceForge inside the Tomcat container. If you are experiencing issues creating/altering repositories or adding/removing users from repository access and the other ?SourceForge integration logs are not providing any clues, you may wish to review the Tomcat log at:

/usr/local/sourceforge/tomcat/jakarta-tomcat-<version>/logs/catalina.out

Sometimes, OS-level errors will be flagged into this log and not others. In our experience, it is pretty rare to find something in this log that is not logged elsewhere.

?CategoryFaq