Local Data Availability Service

It will be important for GAIA to keep its data availability tables up to date. In the case that no information is available GAIA will have to assume the data is available. The following procedure is proposed.

  • All nodes have a data availability web service which reports in realtime the number of images they have for a given data channel and date (or short period, say 30 days).
  • The database stores the master node for each data channel. (It may be NULL for datasets where the master copy is not on a GAIA node, in which case some ad-hoc method will be needed.)
  • A database table indicates which node holds the master copy of the database.
  • All nodes run a nightly cron job, which terminates early if it does not hold the master copy of the database. The node holding the master copy proceeds to query all other nodes about the recent status of data channels for which they are the master node.

bubble By pulling data no-one needs remote write access to a database. Any node can take over the role of holding the master copy of the database, or the master copy of a data channel and everything should continue regardless. There may be other uses where it is helpful to get a realtime view of data on a given node, though for most purposes GAIA will just refer to the local database.

todo.gif Have some means to store when the last time each channel was synchronised, and report the value back via a REST webservice. The status webpage should somehow report such information.

Topic revision: r4 - 2007-03-23 - 22:14:04 - SteveMarple
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback