Tuesday, August 03, 2010

Website Performance: Memory as the primary storage

We normally use databases/filesystems as the primary source of storage and add caching (Memory/RAM) to improve performance of the application.

Consider the opposite approach. Use Memory as the primary storage and file-system as a recovery source. So, perform all read and write operations directly in memory but log inserts/updates in file-system. The writes to file-system can be asynchronous (delayed inserts) and thus never become a bottleneck.

This is a risky proposition and should be considered only if:
1) The database operations are becoming a bottleneck and you have tried all possible optimizations. Only the problematic data sets should be considered for this approach.
2) The data is non-critical i.e. it is acceptable even if the data is not available for certain time period. The time duration that this data will be unavailable will at least equal the recovery time from file-system.
3) Typical database constraints (unique, foreign key etc.) do not apply to the data.

No comments: