Python Tools for AFSC GAP
Python-based tool chain ("Pyafscgap.org") for working with the public bottom trawl data from the NOAA AFSC GAP. This provides information from multiple survey programs about where certain species were seen and when under what conditions, information useful for research in ocean health.
Taking your first step is easy!
Explore the data in a UI: To learn about the datasets, try out an in-browser visual analytics app at https://app.pyafscgap.org without writing any code.
Try out a tutorial in your browser: Learn from and modify an in-depth tutorial notebook in a free hosted academic environment (all without installing any local software).
Jump into code: Ready to build your own scripts? Here's an example querying for Pacific cod in the Gulf of Alaska for 2021:
import afscgap # install with pip install afscgap query = afscgap.Query() query.filter_year(eq=2021) query.filter_srvy(eq='GOA') query.filter_scientific_name(eq='Gadus macrocephalus') results = query.execute()
Continue your exploration in the developer docs.
Ready to take it to your own machine? Install the open source tools for accessing the AFSC GAP via Pypi / Pip:
$ pip install afscgap
The library's only dependency is requests and Pandas / numpy are not expected but supported. The above will install the release version of the library. However, you can also install the development version via:
$ pip install git+https://github.com/SchmidtDSE/afscgap.git@main
Installing directly from the repo provides the "edge" version of the library which should be treated as pre-release.
- Pythonic access to the official NOAA AFSC GAP API service.
- Tools for inference of the "negative" observations not provided by the API service.
- Visualization tools for quickly exploring and creating comparisons within the datasets, including for audiences with limited programming experience.
Note that GAP are an excellent collection of datasets produced by the Resource Assessment and Conservation Engineering (RACE) Division of the Alaska Fisheries Science Center (AFSC) as part of the National Oceanic and Atmospheric Administration's Fisheries organization (NOAA Fisheries).
Please see our objectives documentation for additional information about the purpose, developer needs addressed, and goals of the project.
This library provides access to the AFSC GAP data with optional zero catch ("absence") record inference.
Examples / tutorial
One of the best ways to learn is through our examples / tutorials series. For more details see our usage guide.
Full formalized API documentation is available as generated by pdoc in CI / CD.
Detailed information about our data structures and their relationship to the data structures found in NOAA's upstream database is available in our data model documentation.
Absence vs presence data
By default, the NOAA API service will only return information on hauls matching a query. So, for example, requesting data on Pacific cod will only return information on hauls in which Pacific cod is found. This can complicate the calculation of important metrics like catch per unit effort (CPUE). That in mind, one of the most important features in
afscgap is the ability to infer "zero catch" records as enabled by
set_presence_only(False). See more information in our inference docs.
Data quality and completeness
There are a few caveats for working with these data that are important for researchers to understand. These are detailed in our limitations docs.
We are happy to make this library available under the BSD 3-Clause license. See LICENSE for more details. (c) 2023 Regents of University of California. See the Eric and Wendy Schmidt Center for Data Science and the Environment at UC Berkeley.
Intersted in contributing to the project or want to bulid manually? Please see our build docs for details.
- Thank you to Carl Boettiger and Fernando Perez for advice in the Python library.
- Thanks also to Maya Weltman-Fahs, Brookie Guzder-Williams, Angela Hayes, David Joy, and Magali de Bruyn for advice on the visual analytics tool.
- Lewis Barnett, Emily Markowitz, and Ciera Martinez for general guidance.
This is a project of the The Eric and Wendy Schmidt Center for Data Science and the Environment at UC Berkeley where Kevin Koy is Executive Director. Please contact us via firstname.lastname@example.org.
We are happy to be part of the open source community.
In addition to Github-provided Github Actions, our build and documentation systems also use the following but are not distributed with or linked to the project itself:
- mkdocs under the BSD License.
- mkdocs-windmill under the MIT License.
- mypy under the MIT License from Jukka Lehtosalo, Dropbox, and other contributors.
- nose2 under a BSD license from Jason Pellerin and other contributors.
- pdoc under the Unlicense license from Andrew Gallant and Maximilian Hils.
- pycodestyle under the Expat License from Johann C. Rocholl, Florent Xicluna, and Ian Lee.
- pyflakes under the MIT License from Divmod, Florent Xicluna, and other contributors.
- sftp-action under the MIT License from Niklas Creepios.
- ssh-action under the MIT License from Bo-Yi Wu.
Next, the visualization tool has additional dependencies as documented in the visualization readme.
Thank you to all of these projects for their contribution.
Annotated version history:
1.0.4: Minor documentation fypo fix.
1.0.3: Documentation edits for journal article.
1.0.2: Minor documentation touch ups for pyopensci.
1.0.1: Minor documentation fix.
1.0.0: Release with pyopensci.
0.0.9: Fix with issue for certain import modalities and the
0.0.8: New query syntax (builder / chaining) and units conversions.
0.0.7: Visual analytics tools.
0.0.6: Performance and size improvements.
0.0.5: Changes to documentation.
0.0.4: Negative / zero catch inference.
0.0.3: Minor updates in documentation.
0.0.2: License under BSD.
0.0.1: Initial release.