Hyperactive

{{toc}} 

h1. DistributedHyperActive

For a while now, we've been quietly talking about ways to make Hyperactive run as a distributed application - that is, to make it run across more than one computer in more than one data center.  There are a lot of possible approaches to making this happen, and each of them has its own strengths and weaknesses.

h2. Why?

Basically, we need resilience and reliability. What happens if a server gets taken? Can people still read content? Add and edit content? How much work is it to get it up and running again? Can we keep it working while under heavy load by spreading the load across multiple computers?

h2. How?

This is where things get complex.  There are quite a number of different ways to distribute a web application across multiple servers.  For the purposes of news production in an increasingly hostile legal environment, we cannot put all our boxes in the same datacenter, which is how many "distributed" setups run.  We assume that a warrant can be served on a datacenter rather than on a single machine, and in fact that it will be generally desirable to have boxes in multiple legal jurisdictions. This means that we need to be able to run Hyperactive, or at least some important parts of it, on machines which are physically distant from each other, separated by the Internet (unless we can find datacenters which straddle borders).

h3. Static HTML producer and distributor

This is the way Mir works - the production server makes static HTML of all files which can then be copied to other "mirror" servers. These mirror servers can do most of the work of serving the content, but can't update it. If the publish server goes, nothing new can be added, and a new server needs to be added. Mirror servers also get copies of all uploaded media files (photos, videos, audio) and serve those too.

It's instructive to take a look at how things work in Mir so that we can see some of the benefits and drawbacks of how it deals with these questions.  First, let's define some terms:

*The Producer* is a mechanism which takes the generated output of a full HTML page and outputs it to disk.  Crucially, it needs to do this whenever something gets changed - it can't wait until a user asks for the content in order to generate the static HTML file.  In Mir, this is basically a batch job which will loop through whatever has changed recently and regenerate it on-disk.  These batch jobs can take a while to run.

*The Distributor* is a mechanism for getting the static HTML files outputted by the Producer, and also any uploaded media files, onto multiple mirror servers, in order to distribute the load among multiple boxes, and to provide a bit of redundancy in case a server is lost.  In a regular Mir setup, the job of distribution is performed by rsync.  Although it certainly does work and is reasonably reliable, there are some problems with rsync when it's used like this:

a) on larger sites, with potentially hundreds of thousands or millions of files, rsync takes *a lot* of memory, and is slow to figure out what has changed.  

b) rsync is a polling rather than an event-based technology.  This means it needs to be run periodically rather than being notified itself of when it needs to grab new stuff. When we combine consideration of the first problem of slowness and high memory use, with the fact that installations needing multi-server setups are likely to be large and high-traffic, we end up with an annoying problem. We can't rsync pull to the mirrors every few seconds, because rsync may need 5 minutes to check what files have changed on the publishing server (for a large site).  So we only run rsync ever ten minutes (to be safe).  This means that it may take as long as 15 minutes for a change which a user makes to Article X to be reflected on a mirror site.  

*Current status of a producer in Hyperactive:* Hyperactive is already designed with easy cacheability of HTML pages in mind, to the extent that during the G20 summit demonstrations in London, 97.7% of all requests were served as static HTML (via Apache) rather than as dynamic requests requiring the Ruby executable and Rails framework to be loaded.  This approach has the disadvantage of constraining us slightly in our user interface design, and the advantage that we can actually set the site up with a static HTML producer very easily. The site has extremely good performance on crappy hardware.

*What it would take to make a producer in Hyperactive:* We'd probably need to override some methods in the normal Sweeper classes, which are part of the Rails framework. Currently Hyperactive uses the normal Rails full-page caching mechanism, which works more or less as follows.  Let's take the example of a published article as an example.

# A user publishes an article.  The title, body, and other necessary stuff gets saved to the database. The user is happy.

# A (potentially different) user views the article.  Since there is no HTML page existing on disk to show this user, Rails fires up, grabs the data, formats the page, and sends it to the user's browser.  *As a byproduct of this*, Rails also saves the generated HTML output as a file on disk.

# Another user views the article. Because there is a cached HTML file on disk, the web server will give that back to the user without firing up Rails at all.  It should be noted that serving a page as a static file like this is roughly 100 times faster than hitting Rails for the same thing.  Put another way, the same server is likely to be able to handle 100x as many viewers using static HTML as using a Rails application without this sort of caching.

# If the article is edited by someone (let's say a site administrator turns it into a feature), the act of saving the page destroys the cached HTML file on disk.  

# The next time it gets viewed, the article will again be cached as an HTML file on disk.

The main problem with all of this from the standpoint of using Hyperactive as a static HTML producer is that the HTML caching comes at the wrong point for our purposes.  Ideally, we'd like the HTML file to be generated *whenever the article is saved*, rather than *when somebody views the article*.  Personally I (yossarian) have not investigated the relevant classes, but my guess is that we could override the way caching works so that caching happens in an after_save controller filter, or in some kind of Observer object.

Static HTML generation one should be fairly easy one to accomplish quickly.   

*Current status of a Distributor in Hyperactive:* It's nonexistent.

*What it would take to make a Distributor in Hyperactive:*

# Some sort of event-based distribution system so that we can push new HTML and media to the mirrors as quickly as possible.

# A way of having the mirrors poll the publishing servers and pull things across the network at intervals of, let's say, 5 seconds or less.  This is still less desirable than a real event-based system.

It needs to be kept in mind no matter what we do, we're going to need to solve the problem of file transport for media files if we want to make Hyperactive run across multiple servers.

h3. Master-Slave MySQL replication

This would have multiple servers able to act as the publish server, though only one (the master) is running at any one time. The master sends all changes to the database to the others (the slaves). Then if the master goes offline, one of the slaves can be started and off we go again.

Anything not stored in the database would need to distributed in another way - eg rsync of media files.

(though I wonder if the 'no slaves, no masters' crowd would object ;)

Links:

* "MySQL manual on replication":http://dev.mysql.com/doc/refman/5.0/en/replication.html

* "Short tutorial to set up MySQL replication":http://www.howtoforge.com/mysql_database_replication

* A "MySQL replication adapter for rails":http://blog.rapleaf.com/dev/?p=5

* "Master-master replication with rails and MySQL":http://fivepoundsflax.blogspot.com/2007/03/mysql-5-replication-and-rails-schweet.html

h3. Master-Master MySQL replication

h3. reverse proxies

Other servers can be set up to serve the content, spreading the load in times of high usage. The first time the proxy is asked for a page, it asks the publish server, and after that it just returns its copy of the page, until the expire time has passed, at which point it asks the publish server again.

It spreads the load, but does not provide a full copy of the original.

Details:

* apache does "caching":http://httpd.apache.org/docs/2.2/caching.html 

* It helps to have "expires headers generated by the publish server":http://www.stephensykes.com/blog_perm.html?157 

* Thus we could have static files (images, videos, javascript, css ...) cached for a relatively long time. 

* Meanwhile for the articles we could use the "CacheLastModifiedFactor Directive":http://httpd.apache.org/docs/2.2/mod/mod_cache.html#cachelastmodifiedfactor - this allows for the cache time being proportional to the age of the page. So a page modified in the last hour could have a short cache time, while a page not touched for hours could have a long cache time.

* and we could mandate a very short time (say 60 seconds) for the top level index pages - the home page, /articles /events /videos ...

h3. couchdb - distributed database

Having a distributed database such as "couchdb":http://couchdb.apache.org/ means that the rails code can run on multiple servers. couchdb can be installed from the ubuntu repositories from intrepid onwards.

It can be used from rails by using "activecouch":http://github.com/arunthampi/activecouch/tree/master - see "these tutorials":http://barkingiguana.com/tag/couchdb/ Also see "this series of articles":http://aimee.mychores.co.uk/2008/09/07/post/320/ about rails and couchdb. Aswell as activecouch, it also covers "relaxdb":http://github.com/paulcarey/relaxdb/tree/master and "basic_model":http://github.com/topfunky/basic_model/tree/master which have various pros and cons.

h3. distributed filesystem

Such as "mogilefs":http://danga.com/mogilefs/ - This stores files across multiple computers and could be used in combination with a distributed database or mysql replication to provide a full copy.  Some points about mogilefs:

* From the website: "It's meant for archiving write-once files and doing only sequential reads. (though you can modify a file by way of overwriting it with a new version)"

* It does have a "ruby library":http://seattlerb.rubyforge.org/mogilefs-client/

* Found a "tutorial for storing uploaded images":http://barkingiguana.com/2008/10/31/scaling-using-mogilefs-for-storing-uploaded-images

Should be fairly easy to modify it to include audio and video uploads.  Adding the cached html files would be more difficult though as this is done within the depths of rails, and the way the file system works is quite different. But then again the cached files are less of an issue - they are pretty cheap to regenerate.

We should look into if it works well across the internet, or if it uses a lot of bandwidth and should only be used within a datacentre.

Setup guides

* http://www.imvu.com/blogs/index.php?blog=12&title=how_to_setup_mogilefs&more=1&c=1&tb=1&pb=1

* http://mogilefs.pbwiki.com/Another+How+to+Install+MogileFS+-+Debian+Sarg

h3. Run our own Amazon S3 equivalent

Although it doesn't do the replication part, there's no reason why something like "Park Place":http://github.com/why/parkplace/tree/master couldn't be used as the basis of an S3-style storage infrastructure for Indymedia.

h3. Push replication (and syndication) via xmpp message streams

This is a complex one and will take a little explanation.  The basics of it for those with a knowledge of Ruby and a little imagination are attached below.

h2. Views

h3. mish

I'm not that keen on the producer/distributor model. 

Firstly, it help spreads the load, but doesn't truly allow us to keep going when the publish server goes down. We can't publish, hide ... 

If all we want to do is spread the load, then why not just use a reverse proxy cache. Much simpler to set up. A little bit of lag, but we should be able to manage that fairly well using the mechanisms mentioned above.

If we want to keep going when a server disappears then we need to have multiple publish servers, whether in a master/slave arrangement (which has the minor downside of needing a little manual involvement to set up) or in some master/master or truly distributed model.

Whatever our aim, the producer/distributor model is not the best choice.

Also, given we have lots of tag pages, event calendars going way into the future ... I don't think we would want to always generate every single page we want to.

h3. yossarian

I agree with Mish about the relative worth of a producer/distributor model vs a reverse proxy cache.  Mir works very well but it is quite slow to distribute pages to its mirrors, because of the limitations of rsync (its distributor mechanism).  Speeding up the distribution mechanism by using some kind of event-based messaging system (maybe a combination of inotify and xmpp, say) would be great, but by the time the "simple mirror" has the tools to do the job, we're talking about installing a language executable, enough libraries that it can accept an xmpp message (or a REST request, or whatever) and update its stored HTML pages accordingly.  By the time you've installed all this crap on the allegedly "simple" mirror, it seems like you might as well just install Hyperactive and be done with it, because it's not that much more complex to just set up the entire Hyperactive web application.  So, I agree that a proxy (like squid, say) makes more sense than a "mirror" which is fast but relatively complex to set up.