Habitat Model

Batch CHaMP Analysis

This document describes the process of setting up and running CHaMP visits through the habitat model software. The whole process is geared towards batch processing multiple CHaMP visits in a single habitat model run. As such, much of the workflow is focused on obtaining and organizing all the necessary data for the desired visits in a very specific folder structure. At the highest level the workflow is:

  1. Creating folders to store data.
  2. Downloading CHaMP topographic and hydraulic model data.
  3. Unzipping CHaMP data.
  4. Creating D50 substrate rasters for each visit.
  5. Creating a habitat project
  6. Creating Habitat simulations
  7. Running Habitat simulations
  8. Post-processing results

Regretably, this is a complex workflow. But when all goes well, it’s possible to run the CHaMP habitat model for all CHaMP data (roughly 2,000 visits) in one run!

Requirements

The workflow actually involves running several, separate CHaMP software tools and so it’s important that you have the latest versions of all the necessary tools before processing any data. The Habitat and Workbench software both have Check For Updates features located under the Help menus that can be used to ensure you are update.

updates

  1. Microsoft .Net Framework 4 or newer (only required if you have windows 7 or earlier. Windows 8 onwards already comes with this).
  2. Habitat Model Software.
  3. CHaMP Workbench software.

If you are want to run either the FIS habitat model or the HSI habitat model for the spawner life stage, then you will need rasters that represent the in-channel bed substrate (i.e. D50). These rasters are generated by merging the CHaMP topographic data with the aux ocular grain size measurements. There is a special batch tool that generates these D50 rasters that is part of the CHaMP Topo Toolbar. This toolbar is an ArcGIS AddIn and has its own set of requirements:

  1. ArcGIS Desktop (it’s critical that your system match all the specific ArcGIS requirements below)
  2. Version 10.1
  3. Service Pack 1
  4. Spatial Analyst (enabled)
  5. Advanced license level (formerly ArcInfo)
  6. CHaMP Topo Toolbar

Folders

Pick a location on your computer where you want to store all the inputs and outputs of the model analysis. This should be a folder high up somewhere on your computer and must have no spaces in the path. Something like C:\CHaMP is optimal. For the remainder of these instructions this will be called your home folder. Inside this folder you should create the following folders

C:\CHaMP\MonitoringData
C:\CHaMP\d50Rasters
C:\CHaMP\Temp

This entire process uses a very rigorous and consistent folder structure inside the MonitoringData and d50Rasters folders. It’s absolutely critical during this process that you end up with the following folder structure:

<Home>\MonitoringData\<FieldSeason>\WatershedName\SiteName\VISIT_XXXX\Topo\SurveyGDB.gdb
<Home>\MonitoringData\<FieldSeason>\WatershedName\SiteName\VISIT_XXXX\Hydro\HydraulicResults\dem_grid

If you have some or all the CHaMP data that you need already on your computer, it must be reorganized into this folder structure for the various tools to find all the necessary files. Note that this folder structure uses the ByYear copy of the CHaMP data. It does not work with the ByWatershed version of the data.

CHaMP Data Download

This habitat modeling process requires the results of the Delft hydraulic model runs for each visit. These data are available on the CHaMP FTP site. Follow the process below to obtain both the topo data (needed to generate D50 rasters later) and the hydraulic model result data for a particular watershed in a particular year.

  1. Open Your FTP Client of choice (e.g. FileZilla)
  2. Connect to the CHaMP FTP site (site = ftp.geooptix.com, port = 21)
  3. Browse the remote site into the ByYear folder.
  4. Browse your local site into the <Home>\MonitoringData folder.
  5. For each field season in which you want data:
    1. Browse the remote site into the field season in question.
    2. Create the relevant field season folder in the local site (if it doesn’t already exists).
    3. Drag the watershed(s) folders from the remote to the local site.
    4. Wait for the field season to finish and then repeat for other field seasons.

Again, the resultant folder structure is critical! Pause at this point and review the files on your system match the overall folder structure described above. None of the following processes will work if you get this step wrong.

UnZipping the CHaMP Data

The CHaMP data downloaded from the FTP site are stored as a series of individual compressed zip files. The workbench is capable of uncompressing all the files in one operation:

  1. Open the workbench software.
  2. Click Data - Unpack Monitoring Data...
  3. Click browse next to the top level folder and choose your <Home>\MonitoringData folder.
  4. Leave all other options as they are.
  5. Click OK. Note that it can take upwards of an hour to unzip several field seasons of data in just one watershed.

unzip

Workbench Configuration

  1. Download and/or update the CHaMP Workbench software to the latest version.
  2. Download the latest Workbench database (use the link below the software download).
  3. Extract the workbench database zip file into your analysis home folder.
  4. Start the Workbench software and choose File - Open. Browse to the workbench database that you just unzipped (Workbench.accdb).
  5. Point to your analysis folders:
    1. Click Tools - Options to open the Options form.
    2. Click browse next to the MonitoringData folder path and point to the <Home>\MonitoringData folder created above.
    3. Click browse next to the InputsOutputs folder path and point to the <Home>\d50 folder created above.
    4. Click browse next to the Temp folder path and point to the Home\Temp folder created above.

options

At this stage you should have all the survey and hydraulic data on your computer in an uncompresssed format, and in a consistent folder structure. You are finally ready to start some processing…

Generating D50 Rasters

… but not habitat modeling! You first need to generate a D50 substrate raster for each visit. This process is done in the CHaMP Topo Toolbar.

  1. Launch ArcMap, the desktop software that is part of the ArcGIS.
  2. Make sure that you have the latest CHaMP Topo Toolbar installed.
  3. Click Advanced - Substrate Raster Generation (Batch).
  4. Browse to the workbench database (should be in your folder from the step above)
  5. Browse to your \MonitoringData folder
  6. Select the desired grain size percentile. The CHaMP standard is D50.
  7. Next to Raster name enter AssocRough.tif.
  8. For the Top level output folder path enter the path to your <Home>\D50Rasters folder.
  9. Check the Overwrite existing substrate rasters option.
  10. Check the Clear substrate raste paths... option.
  11. Clear the Filter for annual sites option.
  12. Clear the Filter for primary visits option.
  13. Set which field seasons to process by navigating to the Field Seasons tab, select only the years to be processed.
  14. Set which watersheds to process by navigating to the Watersheds tab, select only the watersheds to be processed.
  15. Click OK and then wait. This form does not have a progress box or wait cursor. Simply wait for it to finish processing. Generating each substrate raster could take more than a minute. Generating a whole watershed could take two hours. Be patient even though there is no indication of progress. A message box will appear when it has finished processing that reports how many D50 rasters were produced.

d50

Create a Habitat Model Project

Before you proceed, you should think about the types of simulations that you want to run. Each habitat simulation represents a particular combination of:

  • Species
  • Life stage
  • Habitat model (either HSI or FIS)
  • input data (i.e. CHaMP visit)
  • Field season

For example if you have 10 visits and you want to run HSI and FIS models for both Chinook and Steelhead and both juvenile and spawner, you would have 80 simulations! This can add up quickly, especially if you add in repeat visits across field seasons. Some planning is required. All the CHaMP tools involved are designed to work in batch mode and the habitat model software has been tested on running hundreds of simulations in one operation. In reality, you will want to bundle your simulations into smaller runs, if for no other reason than output file management and validating that each run is successful before proceeding to the next.

We recommend that you breakdown your simulations into batches based on the model, because models are tied to species and lifestage. So by running all visits for one particular model you are essentially running all your data for one combination of species and life stage.

There are three steps to running the habitat model. First you need to create a habitat model project. Second you need to populate it with the individual simulations that you want to run. Third and finally, you have to run all the simulations. This is the first step, creating the project:

  1. Download and install the habitat model software.
  2. Click Help - Check For Updates to ensure that you have the latest version.
  3. Click File - New Project to create a new project.
    1. Provide a meaningful project name. Ideally this would describe your particular batch of simulations so using the model, species and lifestage is a good idea. e.g. HSI_Ch_Jv would be good for juvenile Chinook HSI simulations. Or FIS_St_Sp for FIS Steelhead spawners.
    2. Ensure that your project is created in you folder using something like `Home\HabSims` as the folder.
  4. Leave the habitat software open at this point.

Create Habitat Model Simulations

Habitat model simulations are actually created using the CHaMP Workbench software. A special tool in the Workbench will create and write the specifications of the simulations to the project that you just created in the previous step:

  1. Open the CHaMP Workbench software.
  2. Click Tools - Habitat - Generate Batch Habitat Project...
  3. Switch to the Other Criteria tab and turn un-check the Must be a primary visit checkbox.
  4. Review the Watershed, Visit Types and Species Present tabs and change the filters to match your needs. Note that unchecking all boxes on a particular list effectively turns off that particular list and stops it from having a filtering effect.
  5. Review the list of filtered visits at the bottom of the screen.
  6. Either check the box next to each visit that you want to run or click “Select All” to check the box next to all the visits listed.
  7. Remember your visit filter settings because you’re going to need to run this tool again for subsequent simulation batches.
  8. Switch to the Habitat Model tab.
  9. Browse and select your habitat model project file (*.xml) created in the previous step.
  10. Check the box next to the habitat model that you want to run. Remember, it’s recommended that you separate models into separate projects for the purpose of batch processing.
  11. Browse and set the Hydraulic Model Resutls top level folder to your <Home>\MonitoringData folder.
  12. Browse and set the D50 Substrate top level folder to your <home>\D50Rasters folder.
  13. Click OK.
  14. You can close the workbench software.

This process is fairly quick. It’s simply writing the definition of your input data and desired simulations to the habitat probject created in the previous step. When it has finished, you should see a message reporting the number of simulations created.

Note that any visits that are missing hydraulic results and/or missing D50 substrate rasters will not be added to the habitat project. This should explain why the number of simulations reported might differ from your total numner of visits selected.

habitat1

habitat2

Running Habitat Model Simulations

Finally!

  1. Open the Habitat Model Software if you don’t have it open already.
  2. Make sure that the correct project is open (containing the batch of simulations that you just created).
  3. Right click on the project name at the top of the project tree on the left of the screen and choose Reload Project.
  4. Click on Simulations and you should see all the simulations that the workbench software added to your project.
  5. Click Configuration - Run Queued Simulations.
  6. Click Run to start the batch processing of all the simulations. The processing time is very different depending on the type of model being run (i.e. whether there are only CSV inputs or rasters involved. Or whether it’s HSI or FIS).

run

Repeating For Different Species / Life stages

At this point you should have completed the entire process of running the habitat model on a single species/life stage/model. You will now want to repeat the last three steps for the reamining combinations of these three elements. Remember to name each project with an indication of the model, species and life stage (e.g. “FIS_St_Sp”). This will really help when trying to identify results.

  1. Create a Project
  2. Create Habitat Model Simulations
  3. Running Habitat Model Simulations

Post-Processing And Using the Results

The habitat model produces two types of results:

  • Cell-wise results - habitat utilization values for every cell in the input data. If one or more data inputs to the simulation were rasters then the habitat utilization results will also be in raster format (and can be displayed in GIS etc). If all the data inputs to a simulation were CSV files then the habitat utilization results will be in CSV format (with each output row corresponding to a row in the input data).
  • Summary Metrics - the habitat model summarizes the cell-wise results into several numerical metrics. Each metric has one value per simulation. Weighted usable area (WUA) and normalized weighted usable area are the two most useful. See the results online documentation for an explanation.

The current habitat model software does not currently include a way to view results. This feature is planned for the next release.

You can retrieve the cell-wise results by browsing the project folder structure (in the software, right click on the project node at the top of the project tree and choose Browse Project Folder, then navigate to the Simulations folder.)

browse project

The summary metrics are stored in the habitat software project file. This file is in XML format, which is useful if you know how a scripting language, but it’s typically easiest to get the data into Microsoft Access and from there to whatever analysis package you want - R or Excel etc. The steps to get the data into Access are described below:

  1. Run the relevant habitat simulations.
  2. Open Microsoft Access and create a new, blank Access database (using any Access file format).
  3. Switch to the External Data tab and click to import from an XML File access xml
  4. Browse to the habitat project XML file (using the instructions above. The file will have the name of your project and end with .xml)
  5. Click OK to open the file.
  6. Click OK to accept the default import options (which should be structure and data)
  7. Access should now contain the contents of the habitat project as a series of tables.
  8. Use the following SQL query to retrieve the simulation results:
SELECT S.Title AS Simulation, T.Title AS [Result Type], R.ResultValue AS [Result Value]
FROM (Simulations AS S INNER JOIN SimulationResults AS R
  ON S.SimulationID = R.SimulationID)
  INNER JOIN SimulationResultTypes AS T ON R.ResultTypeID = T.ResultTypeID;

Running this query will produce a table similar to the one below. See the results online documentation for an explanation of how these values are calculated.

access results