Creating an Industrial Strength Plone Server

Anyone can setup a generic Plone server for development, but how do you set up an industrial strength Plone server that’s rock solid and easy to maintain?

We’ve been using Plone for a couple of years for quick websites that have a bent towards discrete web based applications – sort of a “web 2.0” flare kind of thing. For example our Center for Grid Computing site uses software build management, wikis, and issue tracking to help manage the creation and support of grid applications running on the ARDA platform.

Plone is really a nice tool – its easy to install, runs on just about any operating system, has an elegant and powerful object oriented architecture that lends itself to application development with very little coding, and has a very large international community developing it and plugin products that run on top of it. However with all of that said, if you intend to run Plone for a large site or a site that has a large community there are a whole bunch of things we have learned that make Plone really perform and also lower the maintenance significantly. This post will focus on how to install and configure Plone when you want a big and robust site.

This post will have several sections:

1. Installation the basic Python/Zope/Plone system
2. Configuration of Apache for reverse-proxy web caching
3. Configuration of Zope/Plone caching and other tweaks

Most people when they start out with Plone do the sensible thing and just grab the Win32 installer and bingo they are up and running right away. However once you get a bit of experience this approach starts to show some issues:

• Python/Zope/Plone become inextricably tied so that patching and upgrading become very difficult
• Moving a plone instance from one server to another becomes nearly impossible
• Creating backups of your Plone sites and databases is also difficult
• Hosting multiple Zope servers on a single OS becomes a nightmare.

The way to get around all these issues is to install the major pieces separately and control where they are installed so that they are not in a merged directory tree. This post will describe how to do this so please read on if this sounds interesting.

 Industrial Strength Plone Server Part 1: Installation the basic Python/Zope/Plone system

Installation of Python

Python is the basic language, interperter, and component library that the Zope server and Plone software are written in. You need to get and install this first. So go grab Python and install this to a directory on your server. I’m grabbing the Win32 installer for Python 2.4 and installing this to C:Python24.

Once you have that you done need a few more things so you might as well get them now. You will need these components:

• Python Imaging Library – I grabbed 1.16 as I was writing this.
• Python Extensions for Windows – I grabbed build210 for Python 2.4
• libxml2 processor for Windows – I grabbed version 2.6.23
• elementree for Windows – I grabbed version 1.2.6-20

These are all supplied as windows installers so grab them and install them to the default locations – which will be the Python24 directory created above in my case. Yes, people will differ on this on the internet probably saying that this is not a good thing because then you can’t easily have multiple versions of python and zope running on the same box. Well I go for simplicity and having multiple Zope server versions running on different python versions is not simple – thus for me the components go under the base python system.

Test all this by opening a command prompt and typing: python

You should get a python interpreter prompt, if you don’t something bad has happened – see the “oh no it didn’t work” page (TBD: use google for now I suppose…). Exit the interpreter using CTRL-Z.

Installation of Zope

Zope is the application server and component framework that Plone is based on. Zope is quite simple to install so just grab the installer from the Zope site and execute it. I like to install Zope to a specific directory based on its version. In this case I grabbed Zope 2.9.6 and installed it to C:Zope296

When running the installer do not have them start the thing up for you or create an instance for you. We’ll do that ourselves so uncheck the boxes and finish the installer.

Once the installer is completed open a command prompt and go into your c:Zope296Zopebin directory and execute the make instance script using this command:

python mkzopeinstance.py

Follow the prompts and use a directory location outside the tree where all the other stuff went. I like to put this stuff on a separate drive. For the example I used d:zope-instance-t1.

After that completes open a command prompt and cd on over to your Zope directory and go into the bin. Zope will by default run on port 8080. If you don’t like this then go into the zope.conf file in etc and edit the port-base parameter to use a port that you like. I changed this to 1000 so that my zope server will run on port 9080.

You can start your zope server by typing “runzope” at the command line. There’s a handy script right there to make a service so you should run that as well so that you have a service in the Services control panel. Carefully check the console output to make there there are no errors. A message about missing libxml2 is ok – don’t worry about that.

If all looks ok then go and login to the Zope admin console. Its on the base url and port you used. In my example: http:\emynBeriad.arnor.com:9080

Before going to the next step kill the server with a CTRL-C.

Installation of Plone

To install Plone is easy. What you want to do is go to the Plone site and grab a recent version – I’m grabbing 2.5.1 for this example.  Download and extract that somewhere. What you want to grab is the “Plone for all Platforms” NOT the installer binary for Windows.

Take that, extract it, and put everything into the Products directory under the instance you created – in my example d:zope-instance-t1. Then restart your Zope server and watch the trace in the command window for errors.

Now you can go to your Zope admin server and login and add a Plone site. Log back in and go to the root. Make sure you’re in the root. Using the Add drop down select box, pick off Plone Site, give it an ID, Name, and Description and create a site. This will create your site. When the site is created click on it in the admin console tree and then click on the View tab. Your Plone site should come up in the panel now with the default screen.

Industrial Strength Plone Server Part 2: Configuration of Apache for URL Rewrites and Web Caching

Zope’s internal web server is ok, but you really shouldn’t bog down your app server dealing with stupid web traffic. IIS is a loser web server since all it can do is serve pages unless you get filters and things from various sources. What you need is a web server that can do things like mask urls against hostnames and aliases, do advanced content caching, have strong support for security, broad use on the internet, and a modular architecture that can be extended. Apache’s httpd does all these things and is essentially the de facto web server out there.

Apache is also supported on multiple OSs so if you decide to move your Plone stuff around – taking it to Linux, BSD, etc. is much easier.

I’m off the soapbox now lets get down to the job at hand – configuration of Apache on Windows as a front to multiple zope server instances on a single box where each server has its own hostname alias. We will be using Apache’s url rewrite features. Its a great idea to go to the Plone site and get a background on this topic. Just head over there and search on “apache configuration with plone”. Alex Limi has written some good things on this topic – but they don’t really discuss the specifics too much or anything specific to Windows. So what I’ll do here is provide a specific example.

The first thing to do is to go and grab apache. I grabbed 2.0.55 and installed it with the win32 installer they have. I installed it to my base C drive in case you are following along. Its installed as a service and that’s how I normally run it. There’s also a command line sometimes that’s a good way to debug, but we shouldn’t need it for this.

So what we want to do is enable url rewriting so I can use nice hostnames to refer to the multiple plone sites I have running on my machine. In my case I have a plone server running at this url:

http://emynBeriad.arnor.com:8082/TestSite1

I would like to have this site respond when I enter this url:

http://www.testsite1.com

To enable the url rewriting we need to muck about in the main apache configuration file: httpd.conf. This file is in the c:ApacheApache2conf directory. the first thing we need to do is turn on a few modules. Scroll down in the file to the section on Dynamic Shared Objects. Make sure the following modules are unrem’d:

• LoadModule proxy_module modules/mod_proxy.so
• LoadModule proxy_connect_module modules/mod_proxy_connect.so
• LoadModule proxy_http_module modules/mod_proxy_http.so
• LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
• LoadModule rewrite_module modules/mod_rewrite.so

These modules allow all the rewriting stuff to work. After you do this restart apache. If it does not startup then go back thru the lines above and rem them out until you figure out which one is causing a problem – then google it.

Next up is the rewrite rules. A rewrite rule is a very freaky wierd thing that I have not quite yet figured out completely. Basically it is a way to get apache to take one url and transform it into another one working with another component in your zope server called a “virtual host monster”. The rules and syntax for how to do this are pretty complex. The Plone site is ok and there’s a lot of information also on zope.org and apache.org on this topic.

Here’s the easy part: the Virtual Host Monster is already in your zope server. Don’t mess with it.

Open up your apache conf file and scroll to the bottom. Section 3 is on virtual hosts. The first thing you need is to identify your system’s base identity with a line like this:

NameVirtualHost 127.0.0.1

You will then need to create a virtual host entry for each plone site/instance that you want to host. On my server I have a bunch of them. I have a section for each just like this one:

<VirtualHost www.testsite1.com> 
 ServerName www.testsite1.com
 RewriteEngine On
 RewriteLog "c:temprewrite_log1"
 RewriteLogLevel 1
 RewriteRule ^/(.*)
 http://emynBeriad.arnor.com:9080/VirtualHostBase/http/www.testsite1.com:80/TestSite1/VirtualHostRoot/$1 [L,P]
 ProxyVia on
</VirtualHost>

This entry is pretty self explanatory except for the important line – the rewrite rule. What the heck is that crap? Basically this line tells Apache to take a url coming in on www.testsite1.com and rewrite this to emynBeriad.arnor.com:9080/TestSite1 sending the request to the Plone server. Plone will handle the request and stream the traffic back to apache. Apache will then send the request back to the client. In this example I also have added extra logging for tracing these errors. You probably wouldn’t want that on a prod box so get rid of that. The point is that the “user” at the browser is entering “http://www.testsite1.com” and apache is routing this behind the scenes to “emynBeriad.arnor.com:9080” and back. The user never sees this…they just see “www.testsite1.com” in their web browser. With this you can host a bunch of zope instances on one machine. Pretty neat!

If you’re having trouble: read, read, and re-read and keep working at it. Its very helpful to run apache from the command line and keep an eye on the rewrite logs too.

Industrial Strength Plone Server Part 3: Configuration of Zope/Plone caching and Other Tweaks

Setting up the caching services is pretty easy. There are lots of different technologies to deploy, but first you should get a baseline of your system’s performance. Apache ships with a crude benchmarking tool called ab.exe that’s in its bin directory. Use this to get a baseline of your system.

There’s a lot of caching possibilities, but so far I’ve found Plone’s basic CacheFu to be remarkably good – improving performance by several hundred percent. The install docs for it are good so just follow them. This product is located in the plone products directory. Once you get CacheFu installed don’t forget to go into the Site Setup in plone and add the base URLs for caching. It really improves performance quite a bit. For more advice on caching definitely check out Joel Burton’s materials from the 2006 and 2007 Plone Conferences

Apache’s own web caching should be active by default, but its definitely a good idea to check it in the httpd.conf file. Squid is another option that you can add to Plone. I have not yet tried Squid, but many people say this is a definite must. Let me know if there’s a good doc on how to get it working with Plone.

I’ve spent some time messing around with the Zope server’s configuration and tuning that. The Zope server will caching things in memory and you can control this in the zope configuration file. You can control the number of objects cached using parameters in section for the zope db. Add a line like this to set a specific number of objects:

cache-size 5000

I’ve found this helpful, but also dangerous. Too big a cache and the Zope server can use up all available RAM on the machine. So be careful.

Threads is another thing to play around with. I have found that changing this to 6 for heavy traffic sites is helpful and to 2 for light traffic sites is good too.

The final thing you can play around with is ZEO, but I’ve not taken that on. Again I would refer you to Joel Burton on the Plone site. He gave a great presentation on this at the 2006 Plone conference and I think he did an update in 2007. ZEO is a Zope technology that allows you to cluster your database for scalability.