SteveMarple's Work Log

See also To do list.

2008-01-19 Friday

  • Discussed OMTI dataset with EmmaSpanswick. Outcome was to create images and not to worry about missing ones, since it is a true reflection of the dataset. Emma remarked that the THEMIS dataset will use 3s resolution images, not 1 minute integrated data. This is different to the riometer datasets. We should perhaps add a remarks column for each data channel where such information can be kept.
  • Updated database to indicate that OMTI data is kept at calgary.canada.gaia-vxo.org node.
  • Added pre- and post-install scripts to the update_database.php script. This solves the database replication problem reported below.

2008-01-13 Sunday

  • Discovered a database replication problem. If the user running the update_database.php script does not have superuser access in PostgreSQL then the earthdistance functions are not created. Giving the gaia user superuser access is not desirable, so two possible workarounds come to mind:
    • Install the database functions into the template0 (or template1?) database so that they are inherited when the new database is created. (Does this work? Not tested)
    • Define a post-install script which the user runs. Obviously this will need to run under a different unix account (probably the postgres user). That could be done by suxec'ing the post-install script, or by using sudo.

2008-01-12 Saturday

  • Created and installed make_crontab.php script to generate a crontab file customised for each installation. The script randomises the minute at which the scripts are run to avoid all nodes querying the master node for database updates at the same time.
  • Created basic version of data_availability page for monitoring the data availability of each data channel. It is intended as an administration tool as each node can be selected individually, and the script compares the files on disk with the data_availability table in the database. In order for different nodes to be examined the script must redirect the web browser to a different node if necessary.

This behaviour, combined with the node icons (flags), gives some unintended results. Normally clicking on the node icons would direct the browser to fetch the same script (with the same query parameters) from a different node. However, that node receives a request for data availability information which can only be processed by the node originally accessed. Thus the browser is redirected back to the original node. Hence it appears that the node icons do not work in this instance.


2007-05-06 Sunday

  • Early flight back to Calgary.

2007-05-04 Friday

  • Discussed GAIA installation details etc.

2007-05-03 Thursday

  • Gave seminar on data sharing, dicussing GAIA and Lancaster's web request system.
  • Discussed INTERMAGNET details with Jennifer.

2007-05-02 Wednesday

  • Met staff at NRCAN.
  • Discussed GAIA progress, riometer installations, SAMNET and toured magnetometer and riometer installations.

2007-05-01 Tuesday

  • Travelled to Ottawa.

2007-04-30 Monday

  • Updated NRCAN seminar to include comments by Emma and Eric.
  • Fixed missing images for GAIA viewer.

2007-04-29 Sunday

  • Some minor additions to NRCAN seminar.

2007-04-27 Friday

  • Wrote seminar for NRCAN visit.

2007-04-26 Thursday

  • Started writing seminar for visit to NRCAN.

2007-04-25 Wednesday

  • Updated the CDF output function in MIA to be closer to the GAIA requirements. More work remaining.
  • Updated the NRCAN CDF files.
  • Updated cdf2txt so that by default times are printed as CDF epoch values (ie milliseconds since year 0). By removing the requirement for time manipulations in cdf2txt and gnuplot the time to plot has been reduced from 5.8s to 4.3s for 1s resolution CDF file. Still much slower than expected from the initial testing. Most of the time is spent by gnuplot.

2007-04-24 Tuesday

  • Started work to add lineplots to the viewer.
  • Added NRCAN riometer details to the riometers database at Lancaster.
  • Added NRCAN riometers details to GAIA.
  • Created CDF files for NRCAN riometer data (2006-12-06 and MEA, OTT and SAS only).

2007-04-22 Sunday

  • Updated the tool which generates the offline data files. Fixed incorrect paths for thumbnails.
  • Reworked the viewer programs so that web requests were only made for data availability. 'Normal' information (projects, stations, channels, nodes etc) now loaded using <script> tags.

2007-04-21 Saturday

  • Submitted abstract for 34 Annual Meeting on optical methods.
  • More development on offline browser.

2007-04-20 Friday

  • Created tool to add processed images (images oriented and chosen palettes applied) to an archive for offline viewing. Had problems with PHP's support for tar archives - it closes the archive after each operation which prevents streaming the tar file, unless all files can be added simulatnaeously. Similar problems with ZIP. Really need an archive format which is valid when two or more archives are concatenated together. Tar has an end-of-file marker and ZIP has a catalogue. Unix's ar format has the feature but does stores only filenames, not paths (at least in POSIX mode). Created own archive format as an interim solution.
  • Created unpacking tool for the GAIA archive format.

2007-04-19 Thursday

  • Received notification that summary images for 2005 from the South Pole riometer were ready. Implemented FTP synchronisation and transferred files.
  • Removed all document.write() commands from the main viewer page, DOM manipulation used instead.

2007-04-18 Wednesday

  • Movie tools development.
  • Bug fixes for Safari.

2007-04-17 Tuesday

  • Starting development on the movie tool.
  • Created a date range control which will accept start/end times. To be used by the movie tool.

2007-04-16 Monday

  • Detailed view mode functional.
  • Created proof-of-concept for movies. Individual frames are stored as DOM nodes within the document and replace the preceding frame when an interval timeout is reached. Rendering is very fast, but when loading images can exceed the number of simultaneous HTTP requests.
  • Extended gaiaMirror class so that calling function can request callbacks for onLoad, onError and onAbort.
  • Improved gaiaMovie class to limit the maximum number of images which are loaded simultaneously. Uses the gaiaMirror callbacks to load more images. gaiaMovie supports onReady and onProgress callbacks so that playback can be disabled until all image loading is completed.

2007-04-15 Sunday

  • Fixed bug which stopped Internet Explorer from displaying date controls. In XHTML the name attribute is not permitted, and unusually IE is the strictest browser! Reverted to using id attribbutes.
  • Experimented with CSS positioning for placing the timeline marker on the detailed view keogram. Easy if the image and maker have positioning absolute and are placed inside a div container with positioning set to static. Centre alignment more difficult.

2007-04-14 Saturday

  • More development on the the detailed view section of the GAIA viewer.

2007-04-13 Friday

2007-04-12 Thursday

  • Fixed some bugs which prevented Internet Explorer from running GAIA. One was a function passed to sort which failed to return a value (my fault). The other concerned deleting a property applied to an image by the gaiaMap functions (to indicate that gaiaMap is already performing mirroring operations). IE doesn't like delete being used, even though it was happy for a value to be added! Set value to false instead of deleting. IE's useless error message (it reported line number and character - but not file!) didn't help. Firebug's console.log() functions worked.
  • Discovered IE doesn't like the layout for the dynamic maps so some tweaking of absolute/realtive positioning values is required. Alternatively could use a frames-based version to keep relative and absolute positioned elements separate.

2007-04-11 Wednesday

  • Investigated the odd behaviour noticed with mirroring code. Found a bug in updateState which meant that selecting a channel added a duplicate channel to the list of channels. updateState fixed and checks user-supplied data to ensure it doesn't happen. The duplicate channel replicated a problem seen when quickly browsing summary images, which arose when two or more mirroing operations are attempted on the same image. Hard to guard against due to possible race conditions so implemented a different strategy - the viewer does not reuse the same image element but creates a new one each time, whihc replces the previous element. This ensure that mirroring operations cannot be attempted multiple times on the same image.
  • Further development on the geographical data availability tool. Now interacts with main viewer page, adding or removing channels and changing to detailed mode (even though detailed mode not implemented yet).

2007-04-10 Tuesday

  • Created first draft of geographical data availability tool. Concerned that we are trying to fit too much information into it. I suspect users will be confused by the fact that the map shows stations, but GAIA display data from channels. This poses some difficulties about how you can state whether a station has data or not when it contains multiple channels. Then there is also the case that apparently identical stations are considered different if they are in different projects.
  • Found some pecularities in the gaia_map code and CSS. It appears that JavaScript cannot (simply) enquire what style property exists when an external stylesheet is used. Until the corect method is found the style properties for the map image and div elements will have to be placed directly into those elements in the HTML file; not a major problem since it only affects a few elements.

2007-04-09 Monday

  • Fixed various bugs with mirroring code, data availability, etc.
  • Modified how the keograms are displayed. A section (div element) is created for every data channel. Their relative order is reset when ever the user alters the display order. They are shown or hidden depending on the choice of channels the user wishes to see, and whether the channel has data for the selected day. Gives definite improvements when re-rendering the page.
  • Prefetch data availability details for adjacent days using asynchrous XMLHTTPRequest.
  • Generated data availability details in the database and timed how long it takes. With 2 years of availability data the dump file took 0.364s to create. rsyncing and recreating the database took 13.5s (slave system had only 1 month of availability data). It's quick enough that we can afford to update the database quite frequently but slow enough that we are right to create new databases without destroying the current one.

2007-04-08 Sunday

  • Included new date control into the viewer.
  • Rewrote data availability code, now have a class which hides the JSON structure.
  • Modified the data availability service and updated the DTD so that the XML validates.

2007-04-07 Saturday

  • Completed the new date control but not incorported into the viewer. Provides all the functionality we should need:
    • user-definable earliest/latest date limits
    • configurable inital date
    • increment button to advance/retard date/time by user-definable quantity
    • legal to have multiple controls for the same task (eg many previous/next buttons)
    • does not require ID attributes to be set
    • does not require elements to be inside a form
    • user-definable as to whether UTC or local timezone is used
    • earliest/latest date limits can be altered (and the year menu automatically regenerated)
    • year input can be from a select menu or text input box
  • Rewrote the mirroring code (again), this time object-oriented and using references to the object and image elemts. Cyclic references should not be an issue (see code for explanation). Tested and works ok on Firefox and Internet Explorer. Has option to show make images visible after loading, thus hiding the annoying flashes where the broken image icon appears.
  • Now using new mirroring code in the viewer. Seems to work fine.
  • Fixed issue with Firefox unexpectedly calling onload. It was due to setting the src attribute to an existing image before initialising mirroring operations; the initial image then called the mirroring callback.

2007-04-06 Friday

  • Reworked the JavaScript end of the web services. Now object-oriented and hides the underlying XML/JSON structure.

  • Created new date control (object-oriented). Some care needed to avoid cyclic references which interfere with JavaScript's garbage collection.

2007-04-05 Thursday

  • Meeting with Emma and Brian to work out how various parts of the user interface should work, including the map tool.
  • Added function for reloading the JavaScript files to speed up the development and debugging process. Reloading can be initiated from the firebug command line. Works quite successfully provided some certain 'rules' are followed:
    • Do not create new objects with { }, use new Object.
    • Only create the window._nameSpace_ object it it does not already exist. If it were always created then anything which held a reference to objects or functions etc in the file would not be updated since the link between the copies would be broken. To get total reliability that everything is updated then this rule should be applied to all arrays, objects and functions which might be referenced outside of the file.

2007-04-04 Wednesday

  • Modifed MIA software to produce better images and keograms. Valid data now occupies full height of keograms (top and bottom pixels were do not plot values previously). Keograms and images default to using 0dB for the lower limit.
  • Investigated a log mapping of the absorption values but didn't find anything which worked well. Found that a fixed lower limit of zero and autoscaling upper limit gave clearest results.
  • Investigated issues with the image mirroring process. One problem is that when refreshing the page the images are destroyed but the timeout function set runs. Should really cancel any running timeouts when the images are destroyed. A work around is to create unique IDs for the images. Another possibility is to include a reference to the image in the callback function (closure), however this approach is fraught with difficulties as it will cause circular references, preventing the garbage colelctor from destroying unwanted objects. More work required.

2007-04-03 Tuesday

  • Added option to select by project from main viewer page. When updating it checks to see if all channels from a single project are displayed, of so that project is indicated, otherwise the Default selection option is displayed.
  • Discovered problem with IE using apply(). Requires an object (where?). All oterh browsers work ok with it.
  • Included keograms in the viewer. FOund some issues with the mirroring process.

2007-04-02 Monday

  • Added options for selecting channels by project and image orientation to the GAIA viewer.
  • Created channels_orders database table to store details about options for ordering the channel data when displayed in the viewer. Added channel order option to viewer.
  • Viewer now shows which channels are selected, does not yet show the images for those channels.

2007-04-01 Sunday

  • Created a debug palette to display valid data values in black/grey/white, bad data in red and do not plot values in green. Useful to check that thumbnails images do use the two special values correctly.
  • Sent latest round of comments to Allan Weatherwax on the South Pole keograms and thumbnails.
  • Created test images and web page to investigate how different browsers handle image resizing. Created results page using screenshots from the most common browsers. Summary: no browser interpolates when increasing image sizes by an exact multiple. When decreasing image sizes opera interpolates and the others miss out pixels. Conclusion: it appears safe for data providers to generate images smaller than GAIA requires. Images larger than GAIA specifies should not be permitted as bad data and do not plot values will either be rendered incorrectly (Opera), or not rendered at all (FireFox, Sea Monkey etc).
  • Rewrote data_availability web service as original version was too slow. Speed is important because it will be called every time the date is altered. New version now takes a date and channel ID instead of project,station and channel.
  • Viewer updated to include data availability statistics. show_status displays which of the selected channels have data.

2007-03-31 Saturday

  • Viewer development. Renamed debugtool to show_status as now intend to use it in the final version (maybe adding save state and load state options, using cookies).
  • Simplified the callback handling. Now have just one list of callbacks for all state changes. Callback functions can now register for multiple state changes, and when call may be told of multiple state changes.

2007-03-30 Friday

  • Viewer development. Added a page about CreatingNewTools to the wiki to document how to add new tools. Tested the process with a debugging tool to show status.
  • Changed how state information is updated in the viewer. Now have a updateState function which is responsible for updating the state (possibly changing multiple values) and call the callback functions.

2007-03-29 Thursday

  • Included the channel orientation information on the projects page so that data providers can check their details.
  • Received keograms from Allan Weatherwax for IRIS South Pole riometer data, checked and gave feedback about changes to make.
  • Changed the standard installation directory to /var/www/gaia. Moved all configuration and bin directory otside the subversion directory tree.

2007-03-28 Wednesday

  • Discussion with Emma and Brian regarding the GaiaCdfFileFormat.
  • Experimented with how palette control buttons could be included on the viewer page. Converted the palette PPM files to PNG and rotated so that the images could be included on the palette control buttons.

2007-03-27 Tuesday

  • Completed the ISO date control script and wrapped everything up inside a JavaScript object to act as a namespace.
  • Tidied up the strftime function by putting it inside a gaiaSprintf object to act as a namespace.
  • Returned to working on viewer page but found nasty JavaScript bug originating from JavaScript's lack of a block namespace. Solution: put any initialisation code inside a function (so that variables are in function scope, not global scope) and then call the function.
  • After consultation with Mikko decided to make default installation directory /var/www/gaia. Directory struture to look something like:
    /var/www/gaia
    base directory for installation
    /var/www/gaia/conf
    user's configuration files (server_config.php, Apache config files etc).
    /var/www/gaia/db_dumps
    database dumps
    /var/www/gaia/svn
    checked-out subversion repository

2007-03-26 Monday

  • Created JavaScript function for writing a standard ISO date control into a web page. Virtually complete.
  • Adapted strftime.php to put the useful functions into the gaiaDate namespace. Also add some functions (eg strftime(), isLeapYear(), isUTCLeapYear()) into JavaScript's Date class, if they are not already present.
  • Got fed up with JavaScript not having a sprintf function so wrote my own version (derives from strftime, uses regular expressions to match the flags, width, precision, conversion etc of the format specifier. Supports all standard conversions, also width and precision (explicit and dynamic versions using * and *m$), and most flags (left-justified, zero padding, + signs). Not all alternative output forms are supported (#).

2007-03-25 Sunday

  • Updated palette.php to allow each channel to have a default palette. Added default_palette_id to channels table to store the default palette. Updated the channels webservice to include the default palette for a channel.
  • Updated sync.php so that a channel is not synchronised if the current host is the definitive host for that channel. Fixed bug where target directory was set incorrectly.
  • Updated the MIA tools to make GAIA summary plots so that temporary files are created in a temporary directory, so that they aren't transferred to another host running rsync.

2007-03-24 Saturday

  • Modified database:
    • Removed url column from projects table and replaced it with a project_urls table so that mutliple URLs can be associated each project, working in a similar way to how project_contacts works.
    • Added channel_institutes, channel_pis, and channel_contacts tables so that each data channel (ie instrument at a given location) can have its own institute, PI and contact details. This feature will be most useful for larger projects (eg GLORIA or satellite missions).
    • data_fstr column in channels table now called summary_data_fstr to indicate it is used for summary plots, and that higher temporal resolution data may be available elsewhere.
  • Updated various web services (particularly channels) to work with the modified database schema.

2007-03-23 Friday

  • Further discussions about SummaryPlotRequirements, in particular with regard to scaling issues (can use different scaling for each day, thumbnails and keograms should use same scaling).
  • Removed concept of fallback servers.
  • Removed format string columns for keograms (keogram_fstr), thumbnails (thumbnail_fstr) and fallback servers (fallback_keogram_fstr, fallback_thumbnail_fstr) from the database.
  • Created an instrument_types table in the database. There is no need to distinguish between imaging and widebeam riometers, that functionality can be tested for from the has_keograms column.

2007-03-22 Thursday

  • Found bug in MIA functions for making GAIA summary plots: when making multiple days the previous day's keogram was included in the ZIP file, and then removed from disk. Fixed.
  • Updated local_data_availability web service to handle zipped thumbnail archives.

2007-03-19 Monday

  • Discussion with EricDonovan and EmmaSpanswick about including satellite data in GAIA. Decided to incorporate two frames of reference, earth for most ground-based instruments and sun-earth for satellite and ground-based instruments near the poles. Updated the SummaryPlotRequirements and added details of the ImageOrientations to the wiki.
  • Tested current version of GAIA viewer v2 in Safari. All works, even the window focus() commands which Firefox ignores.

2007-03-18 Sunday

  • Database replication code now removes old databases. Persistent connections can prevent an old database from being deleted, but once those connections have timed out the database will be deleted the next time an update is scheduled. Pre/post-install scripts are probably not required if trusted procedural languages are installed into the template1 database.
    bubble Is is reasonable to require all node operators to install the necessary trusted procedural languages into template1?
  • Added further instructions to the NodeInstallationGuide, and tested them when updating existing nodes to follow the current practive of running GAIA from a subversion repository.
  • Created a script to download and install the GAIA database.

2007-03-17 Saturday

  • Further development on database replication. Now have a working system which follows the ideas and requirements discussed in DatabaseReplication.
    todo.gif Pre/post-install scripts are not yet implemented, but will be required by nodes where the plpgsql trusted procedural language has not been installed into the template1 database and thus is not inherited by new databases.
    todo.gif Dropping of old databases not yet implemented.
  • Rewrote NodeInstallationGuide. Current practice is that all nodes will run from a local copy of the subversion tree, enabling software updates to be performed as easily as svn update.

2007-03-16 Friday

  • Investigated DatabaseReplicationOptions, and rejected all available methods except "lazy replication", for which no code seems to exist.
  • Devised own version of lazy replication for GAIA, described in DatabaseReplication.
  • Began work on implementing database replication.

2007-03-15 Thursday

  • Modified GAIA database to store details whether thumbnail images are stored in a ZIP archive.
  • Modified GAIA database to store image file format.
  • Added PHP functions to generate the keogram and thumbnail filenames from basic information such as project abbreviation, station, channel etc.
  • Modified colour palettes to follow new SummaryPlotRequirements.
Meaning Value Old value
data 0..253 1..254
bad data 254 0
do not plot 255 255
Colour palettes generally use 127,127,127 for bad data and 255,255,255 for do not plot.
bubble When greyscale colour palettes are used should the resulting images be PNG with transparency to indicate bad/missing data? The underlying colour from the browser will still show through so perhaps it is not necessary.
  • Added a reversed greyscale palette which may be helpful when printing images.
  • Modified palette.php to include support for thumbnail images which are stored in a ZIP archive.

2007-03-14 Wednesday

  • Continued development of GAIA v2 browser. The "select data channels" tool updates the list of channels to display in the summary browser (not displayed yet but the variable does change). The projects menu option in the maijn browser is set to custom after thw "select data channels" tool modifies channels. Updates work the other way too, so that changing the projects option in the main browser window updates the "select data channels" window, demonstrating that the callbacks do work.
  • Discussed with Mikko the fine details for SummaryPlotRequirements, and updated wiki page to show new requirements.
  • Discussed with Allan Weatherwax summary plot details for South Pole and other imaging riometer data.
  • Updated the MIA functions to generate GAIA v2 summary images.
  • Investigated how zipped images archives could be included. Probably not difficult, and should be able to use unzip and pipe to stdin.

2007-03-13 Tuesday

  • More development work on GAIA v2 browser. Created a "select data channels" tool which will allow the user to change which datasets are displayed. User can add/remove datasets individually or as groups (all, by project, channel type, continent).

2007-03-12 Monday

  • Investigated reason why GAIA test maps do not work with Safari. Discovered that new Image() does not function as required in Safari and should be replaced by document.createElement('img'). Location of markers plotted by Safari appears correct, but the identification of clicked locations is incorrect.
  • Fixed the onClick bug in dynamic maps when using Safari. The problem was that in looking up the DOM tree to find the absolute location Safari also took into account the location of relative positioned elements. The fix is easy, stop looking up the DOM tree once the first non-absolute positioned element is found. The fix is compatible with Firefox and Opera (other browsers not tested).
  • Discussed layout of GAIA with Emma; proposed layout is to have a simple summary browser with a toolbox button which opens a window with advanced options, such as select stations from a form, from a map and getching full-resolution data. Other possibilities include modules for machine vision analysis and events database; also deep links to useful sites such as the LANL SOPA web pages. Created some initial toolbox pages to test the concept.

2007-02-23 Friday

  • Minor fixes to palette, so that abbreviations use the same case as stored in the database, not converted to lower case.
  • Changes to file structure, so that each project has its own subdirectory.

2007-02-22 Thursday

  • Created cdfrename to rename variables in CDF files. May be useful for the CANOPUS dataset downloaded from CDAweb, so that the data is stored in a variable called Absorption instead of Rio. Requires CDF v3.1 or higher.
  • Modified cdf2txt so that a lookup variable can be used to decide which data index should be outputted, so that CDF files which contain data from many sites can still be used.
  • Found that the downloaded CDFs do not use site abbreviation in the site name but the actual name, which is probably not what we will want. Solved the problem by including regular expression matching for the lookup value.

2007-02-21 Wednesday

  • Added GaiaCdfFileFormat details and comments to the wiki.
  • Updated cdf2txt so that valid minimum and maximum values can be extracted from the CDF file (if present) or obtained from command-line arguments. Values outside of the valid range are replace with NaN. A file header can be prefixed to the output, either from a command line argument or from a file (very useful for plotting as the output can be piped into gnuplot with the plot script preceeding the data; avoids the need to generate temporary files).
  • Created a lineplot PHP script to demonstrate/test the concept.

2007-02-20 Tuesday

  • Used cdfexport to convert CDF file to text. Had problems using CDF v. 2.7, but works with 3.1. Cumbersome, since it requires an export settings file to be created first by using cdfexport in interactive mode.
  • Created simple program to convert CDF file to plain text. User specifies the names of the epoch column and one data column. Program outputs two columns. For multi-dimensional data the user can indicate which row,column etc should be used from each record. CDFexport took 1043ms, my cdf2txt took just 32ms.
todo.gif Need option to use the global attributes which specify the maximum and minimum valid values and convert any points outside of that range to NaN (CDF doesn't support NaNs).

2007-02-19 Monday (Family day)

  • Modified web configuration on GAIA server so that DocumentRoot now uses checked out subversion files, for easier updating of the webserver files.
  • Searched for good, free equidistant cylindrical maps, didn't find anything larger or better.
  • Wrote some Matlab code to generate map images using the m_map toolbox.
  • Created cropped polar maps (azimuthal equidistant projection) of Europe and Fenno-Scandinavia.
  • Found and fixed minor bug which had allowed markers outside of the (cropped) map area to appear in the HTML page. Now also check if (x,y) coordinates lie within the image.
  • Downloaded some Blue Marble equidistant cylindrical images from http://eobglossary.gsfc.nasa.gov//Newsroom/BlueMarble/BlueMarble_monthlies.html. Created own versions at 5400x2700 pixels by setting the land in the bathymetry images to transparent, colorizing in GIMP (use values 220, 100 and -33 for hue saturation and lightness(?)) and overlaying onto a land image (August 2004) modified by GIMP's HSV colour tool (set master to +30).
  • Used own version of blue marble images to create high resolution polar map of Fenno-Scandinavia. Makes Matlab's m_map version look very plain.

2007-02-18 Sunday

  • Fixed positioning issues with the dynamic maps.
  • Added extra projections to the gaia_maps.js library. Now handles azimuthal (equidistant polar plots), orthographic and equidistant cylindrical.
  • Added ability to plot the locations of the GAIA stations and nodes on the maps. Uses the GAIA web services to get the locations.
  • Added options to display in various projections to the test page.
  • Added code to refresh marker positions after the projection has been changed.
  • Found interesting 'issues' with Opera and IE. With opera the markers are sometimes placed incorrectly because the image has not loaded yet so the correct size of the marker is not available. Solved by having the marker refresh itself using onload callback. With Internet Explorer cannot set style attributes using setAttribute (at least not for IE 7), need to set using the JavaScript object notation.
  • Tested and working in all major browsers except Safari (test system not available).

2007-02-17 Saturday

  • Put GAIA v2 code into subversion.
  • Watched THEMIS launch!
  • Started work on a polar map for GAIA, using the JavaScript to dynamically add markers to the map.
  • Example code worked, but positioning changs if the browser is resized.

2007-02-16 Friday

  • Meeting with Emma, Mikko and Brian about GAIA progress and what still remains. Suggested JavaScript method for dynamic maps accepted.
  • Set up equipment to watch THEMIS launch. CBC arrived to film. Launch scrubbed 6 minutes before lift off!

2007-02-15 Thursday

  • Comparison of possible methods to combine many small images in one file. Without compression one day of IRIS data occupies 5.8Mb. Using tar (no compression) takes 2.7Mb. Extracting the imagr for 23:54 (ie near end of file) takes approximately 20ms. Using zip the values are reduced to 1.6Mb and ~10ms, ie 27% of the original disk space. Problems with using zip and tar are that the filename of the tar or ZIP archive must be known, and the names of the files within it. Currently there is no way for the database to store the extra metadata.

  • Investigated methods for DynamicMapGeneration, and experimented with producing different projections. The Mapping Hacks book is very helpful. Combining that with MMPS, images from NASA and flatplanet it should be possible to have some fairly simple JavaScript code which can map between pixel location and latitude/longitude.

2007-02-14 Wednesday

2007-02-13 Tuesday

  • Updated MIA (Multi-Instrument Analysis) toolkit for Matlab. Magnetograms and SAMNET summary plots can now be created with the new version (where all data samples are timestamped). About ready to merge new branch to main trunk in subversion, ready for use at University of Calgary and elsewhere.

2007-02-12 Monday

  • Created add_palette program (C, using Imlib2) to add colour maps to keograms and thumbnails. Does the same job as
pngtopnm image.png | ppmtopgm  | pgmtoppm -map map_file.ppm | pnmtopng
or
convert image.png map_file.ppm -virtual-pixel edge -fx 'v.p{0,u*v.h}' -
but quicker and uses less processes.

Method Approximate time Total number of processes
NetPBM tools 82ms 4
ImageMagick 1100 ms 1
add_palette
(uses Imlib2)
55ms 1

2007-02-11 Sunday

  • Updated palette to flip thumbnails and keograms. Can select North, South, poleward or equatorward at top of thumbnails and keograms. Can also select look-up and look-down modes for thumbnails.
  • Modified day_viewer script to include keograms and thumbnails with colour maps. Also options to choose colour map and image orientation.

2007-02-10 Saturday

  • Created script to add colourmaps to thumbnails and keograms.

2007-02-09 Friday

  • Added mirroring functions to day_viewer script, works under Firefox, Internet Explorer, Galeon, Opera and Konqueror. Not tested yet with Safari.

2007-02-08 Thursday

  • Began implementation of mirrored images using real GAIA data and web service output.
  • Modified day_viewer test script to monitor node status. JavaScript modifies the innerHTML property of a paragraph to write updated status.

2007-02-07 Wednesday

2007-02-06 Tuesday

  • Completed local_data_availability service which reports on what data is available locally (searches disk).
  • Wrote update_data_availability script which uses our own web services (XML, loaded using PHP's SimpleXML classes) to find out what data is available, and updates the data availability table.

2007-02-05 Monday

  • Debugged day_viewer page. Now loads data from selected instrument and date. No attempt at switching to alternative nodes if the data is missing.
  • Completed entering channel details to the GAIA database. Some uncertainties on how to deal with file names which are not predictable (they include a varying second count, see GaiaBugs).
  • Started writing local_data_availability web service script, which will be used to update the data availability tables in the master database.
  • Wrote basic Timestamp and Timespan classes for PHP as it seems they do not exist! Timespan class includes the option to store months and years (for addition of timespan and timestamp objects the timestamps are converted to a calendar representation first, before being converted back to the native unix timestamp representation used internally.

2007-02-04 Sunday

  • Wrote simple PHP script to validate XML output from all registered web services using xmlstarlet.
  • Updated day_viewer test page to use revised web services.
  • Wrote data_availability web service script.

2007-02-03 Saturday

  • Completed web services for all major database tables (data_availability service still to do).
  • All web services produce valid XML, though the DTD should be revised with regard to how country and continent definitions.
  • Navigation through the various services is possible using the href attributes, as can be seen by using the HTML output.

2007-02-02 Friday

  • Added HTML output support for the REST web services. Not intended as a 'real' service but mainly as a debugging aid. Elements with href attributes are converted to links.
  • Added option of including comments in the REST web service output (except for JSON).
  • Added function to decide which output format to use for the web services. Now additional output formats do not require all service scripts to be updated.
  • Included channel details for all projects. Channel details are missing the keogram and thumbnail strftime format strings. Start/end dates also missing, but they can be generated once the data availability table has been computed.
  • Added extra web services (continents, countries, index (lists actual services), institutes) and revised some existing ones.

2007-02-01 Thursday

  • Began development of the data availability web service for GAIA.
  • Attended talk about the HierarchicalTriangularMesh (HTM), given by BrianJackel. The HTM can be used used as a spherical mesh at different resolutions. Advantages include a simple referencing scheme which allows software to easily compute if trixels overlap. Very similar to the geodesic mesh MIA uses for antenna calculations, but much improved.
  • Added Matlab output support for the GAIA web services.

2007-01-31 Wednesday

  • Tested nodes test page. Firefox, Internet Explorer, Safari, Konqueror, Galeon and Opera all work correctly.
  • Implemented web service to output all projects, sites and data channels in XML or JSON.
  • Wrote demonstration page which uses JSON to fetch node and projects/sites/channels data for a dynamic web form. Form works under Firefox, Konqueror, Galeon and Opera. Safari and IE not tested.

2007-01-30 Tuesday

  • Rewrote the PHP functions which convert PHP data structures to XML and JSON. Now implemented as a class. Maps PHP data types (numeric, string, boolean) to corresponding JavaScript values in the JSON output; XML output is pure strings. XML output neatly formatted, indent and newlines configurable; similar for JSON output, but spaces also configurable.
  • Modified the GAIA DTD, id attributes are now of type ID, which means they must be unique within the document.
  • Implemented a test page which fetches the GAIA nodes information by making a web request and fetches JSON data. JavaScript then writes out details about the GAIA nodes.

2007-01-29 Monday

  • Discussed options for including South Pole riometer data in GAIA with Al Weatherwax. (Can either important images, or export data and have MIA create the plots).
  • Reinstalled linux on GAIA server. This time using Fedora Core 6. RedHat Enterprise linux (AS) didn't have PHP5; what's the point in having a properly supported OS if you then install critical packages like PHP from source, without an easy upgrade path!
  • Modified database: changed mirror references to node, as GAIA nodes need not be exact copies (only the database will be an exact copy).
  • Updated update_gaia_db script. Now backs up the database; optionally can drop the database before updating. If the database was dropped the commands to include the Earth distance stored procedures are re-run.

2007-01-28 Sunday

  • Rewrote xml_encode function mutliple time in an attempt to get indented output working correctly. Gave up, for now output is all on one line.
  • Wrote /rest/nodes.php to output details of GAIA nodes in both JSON and XML. Resulting XML verifies.
  • Modified GAIA database; all references to programs now renamed to projects.
  • Modified GAIA database to create dedicated tables to details of project PIs, project contacts and project institutes. Now we can have multiple PIs, and institutes associated with a project. Note that order is not preserved, so output in alphabetical order. The previous version of the projects table could store up to 4 different contacts, by having 4 separate columns, the new approach is more flexible in storing the data, though possibly more difficult to fetch it.
  • Wrote /rest/projects.php to output details of GAIA projects. Includes xlink attributes to fetch site details for each project.

2007-01-27 Saturday

  • Investigated options for parsing XML in clients (tedious) or converting to JavaScript (possible licensing prolems, possible lack of support for XLST in Konqueror and other browsers).
  • Investigated JSON (JavaScript Object Notation) as a means of transferring data to the client, looks useful, but validating it more difficult than XML. Less supported and undersood than XML, though far easier to handle than XML in a JavaScript-enabled web browser.
  • Investigated options to convert XML to JSON on the server. Some, but unclear if licence from IBM compatible with GNU GPL. Requires making XML then reparsing to JSON (has adavantages, would mean we would be "ating our own dog food"), unclear if attributes will be lost.
  • Finally settled on making PHP object and using json_encode to convert the PHP nested arrays to JSON.
  • Could not find similar approach to convert to XML. Wrote own xml_encode function so that the same data structure can be ouput as either JSON or XML. GAIA will speak two languages (very Canadian!)

2007-01-26 Friday

  • Installed Sarissa in order to check XMLHttpRequest compatibility with Safari. When using the Sarissa library code works correctly with FireFox, Internet Explorer, Safari, Opera and Konqueror.
  • Continued investigation of solutions to the DynamicMapGeneration problem.

2007-01-25 Thursday

  • Finished configuring Emma's laptop.
  • Obtained old Mac Powerbook G3 for testing Gaia web pages, discovered it would not power up. With Miko fixed PSU lead, which was shorted near the ferrite. Operating system not new enough to support Safari 1.2, so too old for Gaia testing; requires OS upgrade.

2007-01-24 Wednesday

  • Installed Fedora core 6 on laptop for Emma.
  • Discussion with Gaia team regarding direction for implementing Gaia. Results were:
    • To use JavaScript and client-side scripting, OK to limit to a subset of browsers: Internet Explorer 6+, Firefox, opera and Safari should all be supported.
    • Public interfaces to information must output XML
    • Our client must use the public services.
    • Web services should be RESTful.
    • Started documenting the (proposed) GaiaXmlElements we should use.
    • Disovered 'interesting' future in Firefox. It seems (but to be confirmed) that the search engine box does not work when viewing a tab which shows XML, which is something we may encounter if some content is displayed directly with XML and XSL/XLST.

2007-01-23 Tuesday

  • Investigated reasons why JavaScript code for mirroring images did not work in Internet Explorer. Problem was due to how the event handlers were set; use
    obj.onload = myHandler
    instead of
    obj.setAttribute('onload', 'myHandler(this)')
    The change required the handlers to not take any parameters, but for both the this variable can be used to acess the image object which generated the event. For some unexplained reason the onerror handler was called not matter which way it was set, but not for onload and onabort!

  • Investigated options for DynamicMapGeneration. Installed both mapserver (but no mapping data) and GMT to investigate their usefulness. BrianJackel's suggestion of coastline plus layers combined with EmmaSpanswick's suggestion of a precomputed table lookup for converting server-side imagemap pixel locations to latitude/longitude looks to be the most promising answer.

2007-01-22 Monday

  • Installed 4th 500Gb hard drive in new Gaia computer.
  • Installed RedHat Enterprise Linux (Advanced Server edition) on new Gaia computer.
  • Ported some data channels information from v1.0 Gaia database to new database. Discovered different names for the same filters, even for the same data channel! (e.g., 660 nm and 660.0 nm). Therefore data channel names should be stored in a separate table, to prevent unnecessary confusion.
  • Tested image-mirroring JavaScript code:
    Firefox 1.5.0.1 (Fedora linux)
    worked perfectly
    Opera 9.10 (latest current version)
    worked perfectly
    Safari
    worked, but refreshing list of mirrors via AJAX/REST did not work
    Konqueror 3.5.5 (Fedora linux)
    worked, but refreshing list of mirrors via AJAX/REST overwrote the document with the XML URL
    Internet Explorer
    worked, but timeouts were not cancelled, all mirrors marked offline afterwards. Seems to work correctly if setting onload as an attribute (but no valid XHTML!). From http://www.washington.edu/webinfo/snippets/javascript/events.html it looks like the onload attribute should be set to a function (not string containing function and parameters), so then need to use something to detect the event.

2007-01-21 Sunday

  • Investigated and experimented with XSL to format XML into a human-readable version. Requires a modern browser (how modern?). If necessary can get perform XML->HTML conversion on the server using PHP (requires libxsl, available for Gentoo and Fedora).
  • Updated channels table of the database to store URL where each data channel can be synchronised from. Location is defined as a URL, with the protocol rsync denoting an rsync connection to a daemon, and rsync+ssh and rsync+rsh denoting rsync running over an SSH or RSH connection. (rsync+ssh notation borrowed from subversion).
  • Modified the server_config.php requirements (and server_config_template.php) file so that the location of the summary plot images can be customised per-mirror. Also needed for sync script to compute target directory for downloads.
  • Created inc/sync.php command-line PHP script which enables mirror operators to conveniently synchronise the summary plot images. Source locations are fetched from the database, as are which datasets the mirror stores. The server_config.php file is used to identify the mirror on which the command is running, and the target directory for downloads. Tested to rsync and rsync+ssh protocols.

2007-01-20 Saturday

  • Updated mirroring functions to include timeout. Can now set timeout to a smaller value than the browser's normal timeout, thereby allowing an alternative mirror to be automatically selected in a reasonable timescale, say 10 seconds.
  • Database modified:
    • Store separate timeouts for each mirror.
    • Added separate fallback keogram and thumbnail format specifiers for accessing data from a server other than a GAIA mirror.
  • Wrote /restws/data_channels.php and /dtd/data_channels.dtd to describe the data channels in XML format.

2007-01-19 Friday

  • Wrote /js/mirrors.js.php and /testing/mirrors.php to test out ideas for mirroring process. Functional but more work required, need to only choose mirrors which have the data. Also need to (quickly?) detect and remember situtation where a mirror is offline.
  • Added mirrored_img() and get_list_of_mirrors() to /inc/functions.php.
  • Wrote /restws/mirrors.php to generate list of mirrors in XML format.
  • Wrote /dtd/mirrors.dtd to describe the mirrors XML file. Validated using http://www.validome.org/xml/validate.
  • Added REST webservices functionality to /testing/mirrors.php; the list of mirrors can be updated by pressing a button, which fetches are parses the XML file.

2007-01-18 Thursday

  • Created and updated documentation:
    NodeRequirements
    documents what software is needed.
    NodeInstallationGuide
    documents how to create a new mirror installation.
    ProgrammingGuidelines
    coding guidelines, naming conventions etc.
    SiteLayout
    documents where different file types should b stored.
  • Updated server_config.php and config.php scripts to work with global variables, rather than calling a function.
  • Updated generate_apache_config.php, now defines MultiViews so that file extensions can be omitted from URLs, which will help to separate the public interface (ie URL) from the the implementation. /js/config.php now called /config.js.php, but access as /js/config.js.
  • Installed earthdistance contributed code into PostgreSQL databases, can calculate distance between mirrors, and hence arrange mirrors in order of distance.
  • Experimented with JavaScript onerror() handler for loading images. Under HTML 4.01 and XHTML there is no onerror attribute for the <img> tag, so code such as
<img src='/some/image.png' alt='missing image' onerror='missingImage(this')/>
is not valid HTML or XHTML, and the W3C validator indicates as such. It appears the only valid method is to use a <script> tag immediately after the <img> tag to set the onerror() handler.
tip Initially the src attribute should be set to some fake URL, and then set to the correct URL after installing the onerror handler, otherwise the error may be generated before the handler is installed.

2007-01-17 Wednesday

  • Revised database, created countries and continents tables. Country data is combined from ISO 3166 and the CIA factbook. Continent data, and lists of which countries are in which continent taken from WikiPedia.
  • Created GaiaCredits page where credits/references can be stored.
  • Created GaiaNodeSelectionProcess to describe thoughts about how mirroring should work.
  • Created /js/config.php script to generate configuration details for use in JavaScript.

2007-01-16 Tuesday

  • Investigated REST and AJAX.
  • Generated first REST service from ORA book; see JavaScript: The Definitive Guide (4th edition), by David Flanagan, section 17.2.7. Working with XML Documents.
  • Wrote /restws/mirrors.php to generate mirror information in XML.
  • Wrote DTD for mirrors (/dtd/mirrors.dtd).

2007-01-15 Monday

  • New server for GAIA arrived (Fedora 5 installed by Brian).
  • Discussion with Brian, Mikko, Emma and Greg about computing strategy (servers, backups, wikis etc).
  • Created PHP script to generate Apache configuration for GAIA.

2007-01-14 Sunday

  • Added /mirrors.php and /programs.php to display details generated from database. Discovered that PHP's PDO class does not honour PDO::ATTR_FETCH_TABLE_NAMES for PostgreSQL databases.

2007-01-13 Saturday

2007-01-12 Friday

  • GAIA database development. Created channels table to handle different channels from instruments, e.g., all-sky-imagers will have a separate channel for each filter, X-ray satellites images can have separate channels for different energy ranges. Imaging riometer will normally have just one channel.
  • Updated casper.
  • Began PHP experimentation. Started porting SQLmaker.pm from Perl to PHP.

2007-01-11 Thursday

2007-01-10 Wednesday

  • Setting up Wiki etc

2007-01-09 Tuesday

See also To do list.

Topic revision: r100 - 2009-03-11 - 12:05:02 - SteveMarple
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback