Multiple project Trac set-up

I’ve already installed Trac and I now want to be able to set-up multiple projects with the minimum of fuss.

My requirements are:

  • Not having to mess with the Apache configuration every time I add a new project because I don’t want to have to restart apache
  • Make some simple modifications to the stanadard trac.ini so that, for example, the logo at the top links to the home of that trac project
  • Improve upon the default authentication where logging out involves closing the browser (which is a drag when accessing multiple projects)
  • Change the default wiki page text

The most important job is to get Apache set-up properly. I’m using mod_python so:
apt-get install libapache2-mod-python

I then set-up a VirtualHost for http://my.trac.url, thus:

<VirtualHost *>
DocumentRoot /var/www/my.trac.docroot
ServerName my.trac.url
ServerAdmin webmaster@my.trac.url
LogFormat "%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i""
TransferLog /var/log/apache2/trac-access.log
ErrorLog /var/log/apache2/trac-error.log

<LocationMatch "/.+/">
SetHandler mod_python
PythonInterpreter main_interpreter
PythonHandler trac.web.modpython_frontend
PythonPath "sys.path + ['/export/trac']"
PythonOption TracEnvParentDir /export/trac
PythonOption TracUriRoot /
</LocationMatch>

</VirtualHost>

Note the LocationMatch. From the docs:

This will instruct Apache to use mod_python for all locations different from root while having the possibility of placing a custom home page for root in your DocumentRoot folder.

Therefore in /var/www/my.trac.docroot I’ve placed a one line PHP script that redirects users to our main website url.

After restarting Apache I then set-up a new project ‘project1′ as per my mini Trac install how-to and visiting http://my.trac.url/project1/ gives me the vanilla Trac interface so we know it’s all working.

The rest of the configuration is with Trac itself and is down to personal requirements and mine are already listed above. I’ve scripted everything I need to do to get a new project up and running and don’t intend to go though it all here. You can however download it and use it for your own purposes. You use this at your own risk and you should bear in mind the following cavets:

  • Assumes that the Account Manager plugin is already installed. I did easy_install http://trac-hacks.org/svn/accountmanagerplugin/trunk
  • The first user input is used as the Trac project name AND should match the name of the already set-up svn project
  • I’m using MySQL on the backend
  • A few of the trac.ini settings at the end are hard-coded though it is easy to change them

I’m a lot more impressed with this version of Trac than the much older version we were using before and with the above set-up I can now have a project up and running within a few minutes.
Related
http://trac.edgewall.org/wiki/TracModPython
http://trac-hacks.org/wiki/AccountManagerPlugin

Subversion over ssh

My background task over the Christmas holidays was to ensure that I had the ability to give read/write access to a subversion repository situated on a machine within our corporate network so that staff or external contractors can access it via the interweb but without me needing to open up additional ports on our firewall.

The server running Debian ‘etch’ is sitting within our DMZ and is only open to the outside world via port 80 for apache and also for ssh listening on a non-standard port. Internally we can access the repositories using the svn:// method between the internal network and the DMZ using the standard svn password-db authentication (and the handy password caching that this provides).

To give external access it seemed a no-brainer that I’d tunnel over ssh especially when I found out that I could set it up so that authentication would be dealt with by maintaining system (i.e. ssh) users for remote access (which all relevant staff have anyway) and could then administrate separately my local-only svn users that access via the plain-text svn://method. On balance I figured this would work for me as I’d rarely need to give access to an ‘outsider’ and when I did then them having a non-privileged user account on the machine wouldn’t be a big deal.

In the end it turned out to not be too difficult to get working though there was a bit of hoop jumping; I was expecting as much after reading this in the svn book:

Once you’ve jumped through these hoops, your repository should be accessible by all the necessary processes. It may seem a bit messy and complicated, but the problems of having multiple users sharing write-access to common files are classic ones that are not often elegantly solved.

Permissions were the biggest issue and the first thing that I did was to add all external users to the svn group. I already have everything chmoded 2770 with owner/group being ‘svn’ but when accessing over ssh the lock files (amongst others) are read-only for everyone except the svn user. To fix this I had to create two wrapper scripts for the svn and svnserve binaries respectively that set the correct umask:

#!/bin/sh
umask 002
/usr/bin/svn-real "$@"

…where svn-real is the renamed svn binary and this script is /usr/bin/svn, repeat for /usr/bin/svnserve.

That was pretty much it except that for external access the way to connect was slightly different. Firstly the connection ‘URL’ was of the form svn+ssh:// for example:
svn co svn+ssh://hostname/path/to/remote/project/trunk

Also, because we’re connecting over ssh via a non-standard port I needed to set an environment variable for the user on the client machine. To do so I simply added the following to their ~/.profile:
export SVN_SSH="ssh -p PORT_NUMBER".

That was it.

So, in summary to give external access all I now need do is:

  • Create a system account on the svn server
  • Add that account to the svn group
  • Tell that user to create a local SVN_SSH environment variable so they connect via ssh over the right non-standard port

Complicated to set-up? Not really. Easy to administer? Yes. Secure? As good as I could make it.

MySQL replication

More brain dumping, this time after setting up MySQL on my test server to replicate offsite as the data on there is becoming important.

To start with both machines are running fully updated and upgraded Debian Etch and MySQL 5 as a Debian package with identical copies of the MySQL databases in /var/lib/mysql/.

On the master I:

I ensured that bind-address was set to the machine’s IP address as by default it is bound to localhost (therefore not allowing external connections) in /etc/mysql/my.conf

Checked that the server-id and log_bin were set in /etc/mysql/my.conf

Issued the MySQL command GRANT REPLICATION SLAVE ON *.* TO 'USERNAME'@'SLAVE_IP' IDENTIFIED BY 'PASSWORD';

Issued the MySQL command FLUSH TABLES WITH READ LOCK; and then SHOW MASTER STATUS;, noting down the values of File and Position.

On the slave I:

Checked that the server-id and log_bin were set in /etc/mysql/my.conf (server-id being different to that of the master)

Issued the mysql command:

CHANGE MASTER TO MASTER_HOST='MASTER_IP',
MASTER_USER='USERNAME',
MASTER_PASSWORD='PASSWORD',
MASTER_LOG_FILE='FILE_NOTED_EARLIER',
MASTER_LOG_POS=POSITION_NOTED_EARLIER;

To check it was all working I used a combination of the MySQL command SHOW SLAVE STATUS; ensuring that all looked OK and simply made changes on the master and saw them appear on the slave.

If the slave is constantly showing “trying to connect” to the master when you show status then to debug make sure that it can telnet to 3306 on the master. If not then you have to ensure that there is network access between the machines and re-check the bind-address on the master.

Search and replace multiple files with sed

Ok, if you ‘do’ linux then you’ll probably already know this one. I did, kinda, but had to Google to remind myself of the exact chain of commands. So here I am writing it down so it’s easier to find next time.

In my case I had a load of apache conf files where I needed to replace each occurence of an IP address with an asterisk. A combo of find and sed worked well:

find /etc/apache2/sites-available -type f -exec sed -i 's/192.168.255.2/*/g' {} ;

Move multiple subversion repositories

Need to move multiple subversion repositories?

Well I just finished putting a simple how-to together which details the steps I took when moving our 30+ svn projects from one server to another.

It went really smoothly and took about a quarter of the time I was expecting it to.

Anyway, here it is.

Related:

Removing DOS linebreaks from your files using Vim

This morning I’m debugging an issue on a script that gets cronned every minute. I’ve a shell open on the server and the file in question open using Vim and I notice each line has a trailing ^M…. Aargh the dreaded DOS linebreak.

Dusting off my rusty vim-foo I simply do a search and replace using Vim without having to exit the file.

The command is:

:%s/^M//g

Job done, well actually no, there is still an issue with the script but now I at least know that ^M is not the culprit.

Note: To get the ^M I did NOT type a caret and M, I typed Ctrl-V then Ctrl-M

Flush the qmail queue

My secondary mail server (that runs qmail) had a load of email queued up today that I needed to flush. Googling comes back with people saying run qmail with daemontools then you can use on of the nice qmailctl scripts to do a ‘doqueue’. Needless to say I’m not running with daemontools (my primary is), it’s just a secondary server that does *nothing* else and I wanted to keep it nice and simple.

Anyway, I finally found out that sending an ALRM signal to qmail-send works.

Specifically, this worked:

[code] bash:# kill -ALRM (qmail-send PID)[/code]

Nightmare MySQL migration

I’ve been migrating a large number of websites over from one server to another, always an enjoyable task.

So I get to the last one, copy the database over and do the normal mysql import along the lines of:

[code]mysql -ufoo -pbar database_name < database_name.sql[/code]

Errors++

Turns out the Donke^H^H^H^HDeveloper who built the application didn't care about things like reserved words or spaces in the table and field names. Check out this example of a table, made up but containing valid examples:

[mysql]
CREATE TABLE Site-Password (
# ^^^^^^^^^^^^^ HYPHEN and RESERVED WORD
Name of AdviceClient varchar(255) default NULL,
# ^^^^^^^^^^^^^^^^^^^^ SPACES
Date - TEXT varchar(50) default NULL,
# ^^^^^^^^^^^ TWO RESERVED WORDS!, SPACES and HYPHEN
MeetingDate datetime default NULL,
NextMeetingDate datetime default NULL,
Advice Type - 1 varchar(255) default 'Initial Interview',
# ^^^^^^^^^^^^^^^ SPACES, HYPHEN
KEY Key Carer (Key Carer),
# ^^^^^^^^^ RESERVED WORD and SPACES (Genius)
#
);[/mysql]

Anyway, I got it working in the end by hosting it on it's own virtual server, with *exactly* the same version of MySQL (4.0.17) and using the mysqldump flag --quote-names.

Reckon I lost 4 hours on that.

Excluding folders when creating a tarball

When creating a tarball, to exclude folders one simply uses the –exclude= flag:

For example to tar up the contents of /var/log but to exclude /var/log/apache & /var/log/mail/

tar -jcvf logfiles.tar.bz2 /var/log/* --exclude=/var/log/apache/* --exclude=/var/log/mail/*

Just in case you didn’t already know then the ‘j’ flag in the ‘jcvf’ tells tar to use bzip2 compression.