Following the experiment, How to:
backup the data
extract runfile info for FileMaker
insert new database records
(28 November 2006)
Rule: Megabytes are cheap, cats are expensive. One can never be too thin, too rich or have too many backups.
Rule: In the McCrea lab data is collected on the Linux machine “dave” and archived to another Linux platform called “dave2"
Rule: All experiments live in separate directories on the Linux platforms. Since 1997 each experiment has been assigned a unique 2 letter code (aa - zz) indicating the sequence of experiments and not the project type. The naming conventions ensure that data and directories are sorted from oldest to newest.
The directory naming convention is: experiment_code - month - day - year
formatted as : XXmmmddy
eg. the directory named “ experiment gjnov226" is experiment “gj” done on November 22 in 2006.
Rule: Every experiment is backed up to a second Linux machine immediately after the experiment (dave2).
Rule: Camera data is collected on the “Camera PC” (sc41) as two files with a .prn and .dat extension. Their naming convention is dictated by the camera data collection program as “yymmdd”. We prefer to put the prn and dat files into a zip archive. The naming of this archive has been somewhat arbitrary. A good convention would be to call it by the experiment code as gjpics.zip.
Rule: Each run from an experiment becomes a record in a Filemaker Pro database. Each experiment is a record in the same database. This record is a “master” record containing overall information about the experiment (purpose, results, traces, waveforms etc).
Rule: As of May 2006, the name of the database file is fixed as “McCreaLab_Data.fp7." This file is a shared database run on a Windows XP server. The computer (22.214.171.124) is (Nov 2006) is in the equipment rack right next to the Linux capture system ("dave").
Rule: The procedures outlined here create 5 copies of the data:
1) the original on dave (and Camera)
2) the corrected runfiles on dave/home/exp/2006/gjnov226
3) an archive of the corrected data on dave2/data/gjnov226
4) a CDROM of the archived and corrected data. This CD may contain more than one experiment.
5) a second copy of the same CDROM or DVD to be stored off site.
Rule: Analysis is preferably done on the runs in dave/home/exp/2006/gjnov226 and not the archived data. Some users will make further copies of subsets of the data for particular analyses or combine runs from different experiments into a single directory. User discretion dictates whether these runs will be backed up onto CDROM containing analysed data.
1. At the end of the experiment back up data from dave to dave2 using the same directory name on both machines. back to contents
Create this directory (eg. gjnov226) on dave2 and transfer all Linux data files (from dave to dave2) using “FTP” or “rmtfetch”
Zip and transfer the camera files (from Camera to dave2) using FTP into the same directory.
Go to bed.
2. The day after the experiment. On dave2, update the descriptors for each run in the experiment. Make the descriptors as detailed as necessary. Fix up any calibration information (gains, channel names, etc.) using the “fixcal “ program or the calibration submenu of the “analysis” program.
The camera prn file should also be updated to “mark the film” with cells numbers and stimulation intensities. The procedure to do this is covered in Tutorial #12. http://scrc.umanitoba.ca/doc/tutorial/tutorial_12.html
3. The data with corrected and completed run descriptors and calibration information is then archived onto a separate hard drive (mounted as /data) on dave2 before any analysis. Copy the corrected camera data into the same directory.
Make a directory with the experiment’s name in the data partition on dave2. The path to this archive is “/data” (i.e from the root directory on dave2).
eg. mkdir /data/gjnov226
Copy all the files in the experiment into the archive data directory
eg. cp /home/exp/2006/gjnov226/* /data/gjnov226).
4. Preparing the runfile records for Import into FileMaker. back to contents
Extract the run header, descriptors, and channel information from all runs in the experiment into an ascii file that is formatted for our database.
Use “getrundata” program (script) to extract information in the run files into a text file that will be imported into Filemaker Pro.
Pipe the output to a file and give that file a “.csv” extension
eg. getrundata > gjruninfo.csv
Getrundata can be invoked without a path name if you invoke it from the directory containing the rundata files of interest. For example if you are in the directory called “/home/exp/2006/gjnov226", running getrundata will extract data from all run files in that directory. Getrundata can also be used recursively and with pathnames to extract information from multiple directories.
5. FTP the csv file to a Windows computer where the Filemaker Pro program can find it.
6. In Filemaker, open the database (open remote, fms (126.96.36.199), choose McCreaLab_Data). back to contents
1. First create a new “master” record for the experiment. The “master” record is used to easily find experiments. The easiest way to do create a new master is to start by clicking the “ShowMasters” button on one of the layouts. This script searches the database and displays all experiments (i.e. all “master” records). Go to the last master record. Duplicate this record (Records - Duplicate or ctrl D). You now have two “master” records with identical fields. The new record will have a serial number one larger than the last “run” record stored inthe database. (Note: you can also create a new record with (Records - New or ctrl N) and do everything manually).
2. The important fields in your new master record are the FilePath and RunName. Enter the correct name of the new experiment into these fields. You should now have something like FilePath = “/data/gjnov226" and RunName = “gj0master.” Note the blue colour - it is simply a visual aid when looking for where different experiments start on a list.
3. The master record can contain other information concerning the overall experiment. Good things to enter would be a general summary about the experiment’s purpose, who was present, results, traces, waveforms etc. The master then becomes as easy way to find types of experiments as well as to count the number of experiments used for a study.
7. Having made the master, import the information from each run in the experiment into the database.
1. Select Import Records from the File Menu, then File and locate the csv file created with “getrundata”
2. Select open to bring up the window titled Import Field Mapping. This shows how the current (last used) template for importing will match data in the csv file with fields in the database.
3. Check if all the fields match up correctly (i.e. the filename is really the filename and the descriptors are in the descriptor field). Select the >> on the scan data button to see the next record and a few more to check for the field match up.
4. If everything is OK then you may select Import. If you need to change the field match ups, you can just move the bold fields to the lines where they are matched up (read Filemaker Pro help if you have problems here.)
8. Using the Find option in Filemaker Pro ensure that the the imported data is in the database as new records with the proper names (eg. gjc0101). At this point you should have an updated Filmaker Pro database.
9. If you did not create the Master record before the import, you must create it now.
Any record that you create following the import of runfile records will be numbered as the last record in the database. This creates a problem since sorting by serial number will put the master after the runs. Because we are a suspicious lot, we anticipated your inability to follow instructions and set the default serial number increment to be “2". Thus it should be possible to manually change the number of the master record so that it is one less than the first runfile record. The Layout “Table 1" has a conveniently located serial number field on it.
You can also automatically renumber the database. First Find and display sort the imported experiment. Take note of the serial number of the first runfile record. Create a new Master record and enter the properly formatted data in the FilePath and RunName fields. Re-sort so that is field goes to the top of the display. Click in the Serial number field of the Master record. Enter the serial number of the first runfile record into the Master While the cursor is still in the Serial number field, Select “Records - Replace Field Contents” and the “Replace with serial numbers” option - “Increment by 2" “Replace”. Now you should have the displayed records renumbered with the Master preceeding the runfile records.
11. Burn a backup CD (or DVD) with the data from the experiment. That is done on dave2 machine using the “K3B_DVD_Creator” program. This program is very “windows-like”. From the Data files menu select Add directory and enter the directories you want to copy onto the CD. Note that after selecting each directory the size will be shown and you have to make sure that the total selection will fit on the disk. In the Burn menu, select verify. Label the CD with the experiment name. Then make a second version of this CD to be stored off site.