fbm - index

fbm - file and backup management system

Origins of fbm:

The current version of fbm was developed for our own SOHO environment. Full backup on CDs yielded stacks of coasters with multitude copies of the same archive files, while need to limit this process resulted in insufficient coverage of most recent or changed files. With HDD capacity of our system going into hundreds of gigabytes we realised that something must be done to provide fast, simple and well-controlled system. Instead of installation of clumsy heavy-weight (and potentially unreliable) tape backup systems we decided to create smart database system which will automatically assure that there is always sufficient number of instances of each file available regardless of storage media. Each instance is allocated reliability evaluation and user can request different total storage reliability for various data.

At the same time we realise that many small and even medium sized companies are struggling with the same problem. Stacks of tapes of unknown reliability are shuffled, directories on servers are changed again and again to match them to capacity of most recent backup device and boxes of CD coasters grow in the cabinet. Resulting backup system is costly but seldom as reliable as management and sysadmins hope.

We feel that we are not only ones who need something better. fbm hopes to provide modular software system for backup of any type of computer stored digital data. Comparison of fbm and traditional backup shows main differences.

Basic design:

Basis of this system is database of all instances of every file. fbm evaluates if every file is stored sufficient number of times on various media and creates copying requests on as needed basis if insufficient coverage is detected. fbm will also recreate damaged sections of working storage media if required using information in its database. fbm can be used to locate spare copies of individual files, or recreate older versions of file clusters. Main differences between between fbm and generic backup are listed in

fbm is based on PostgreSQL and the current version of scripts is written in python. The current early version is tested (and used) on Debian Woody Linux but working principle is platform independent.

System as planned will allow user to request specific relability and will achieve it by combining old and new copies of files, deposited on various media. Database of file instances becomes control center which can use old copies on CDs, new copies on DVDs, tape backups, spare space on various networked HDDs , USB based storage etc. There is no practical limit to total capacity of data and almost no restrictions as far as type of storage media is concerned. Old backups created by previous systems can be used if file format can be read into database, including ad-hoc created stacks of CDs or even floppy disks.

There are no real technical problems with implementation of fbm. Basic scripts already exist and can be used for trials (creating very useful backup in the process). Many more scripts will be needed to care for other types of hardware and for other more advanced tasks. GUI control center may complement command line interface at a later stage.

Current situation:

A few scripts exist which are currently used to keep our backup system under control. There is little or no sophistication but even at this level advantages are obvious and safety of our data significantly improved. Full system will require many more tasks - see task list.

$Date: 2003/12/20 06:43:52 $