As part of our ongoing work to improve the use of environmental data in agricultural research, we recently published a prototype tool that integrates multiple agrometeorological data sources into a unified access and querying system. This tool demonstrates how general-purpose extract-transform-load (ETL) systems can reduce overhead and improve data usability for digital agriculture workflows.

Researchers often spend a lot of time downloading, cleaning, and reformatting the same climate datasets from multiple providers. Each data source has its own format, API conventions, and spatial-temporal structures. Our prototype simplifies this by offering a standardized, open-source interface that harmonizes disparate sources and automates many of the most common processing tasks.

Built with extensibility and usability in mind, the system includes separate ETL managers for each dataset and supports spatial and temporal aggregation across user-defined parameters. Outputs can be downloaded as CSV or JSON, and users can access the tool through either a graphical user interface or a RESTful API. The system currently runs on Windows, Mac, and Linux and is designed to be lightweight and usable by researchers without extensive programming backgrounds.

We used Minnesota as the case study for this prototype. We were able to explore how researchers might customize queries for specific cropping systems, field experiments, or landscape-scale assessments. We believe this kind of data interface is especially important in the context of changing climates, where the ability to quickly assemble regionally and temporally relevant data can support adjustments to cropping calendars, irrigation schedules, and other time-sensitive decisions.

The project was supported by the Minnesota Environment and Natural Resources Trust Fund and the Legislative-Citizen Commission on Minnesota Resources. We hope this prototype serves not only as a useful tool for others but also as an example of how modular ETL architectures can be applied more broadly to agricultural and environmental research challenges.

The code is available under an open-source license. 

To see related activities by Bryan Runck, check out my Lab Page.

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

GEMS at the Intersection of Technology and Agriculture

Hi, I’m Joe Axberg, an Application Engineer at the Minnesota Supercomputing Institute (MSI).  I am part of the Development and Operations team that builds and maintains our GEMS applications and services. It is through that backend “IT” lens that I will be writing my first post on our blog site.  Others on the GEMS team will be blogging about all the great things being done with GEMS - in other words - how GEMS is being used and how it is making a difference.  In this post though, I’d like to talk a little bit about the technology that powers GEMS.

Here at the University of Minnesota, GEMS is a collaboration between the College of Food, Agricultural, and Natural Resource Sciences (CFANS) and Research Computing, including the Minnesota Supercomputer Institute (MSI) and U Spatial.  The MSI is responsible for providing a variety of high performance computing resources to researchers across the University and beyond. The kind of high performance computing power needed to solve the big problems and crunch the big data.

Utilizing high performance computing can be challenging, complex, and intimidating.  In the agri-food research space, adoption of high performance computing has historically not been very high. Agri-food researchers are not computer scientists - but neither should we need them to be.   Applications and tools should exist that lower the barrier to access high performance computing to the agri-food researcher.

Hence the development of the GEMS Informatics and GEMS Learning Platforms.

What is Informatics? 

A quick Google search revealed this definition:

“the science of processing data for storage and retrieval”

“...processing data for storage and retrieval” - that certainly does sum up what the GEMS platform does - in a perhaps over-simplified way.  At GEMS, we expand on that definition: “turning…data into actionable information for farmers, scientists, governments or companies…” (Check it out at our About Page)

These days it is all about the data and the amount of data is immense - especially in the agri-food sector.  Challenges abound in the form of sustainability, changing climate, distribution, and more.  The data is out there and there is a lot of it.

The volume and complexity of this data often means that traditional methods for gathering, cleaning, organizing, storing, and analyzing data run out of steam. 

The aim of the GEMS Informatics platform is to make the job of the agri-food researcher easier.  An application that allows the data to become actionable more quickly.  Present within GEMS are applications and services that can handle the entire lifecycle of agri-food data.

Coupled with the GEMS platform is GEMS Learning.  Provided are a series of training modules and courses for upskilling agri-food researchers in the use of technology.  GEMS Learning is constantly developing and looking at new training opportunities for agric-food researchers in the area of high performance computing.

What Powers GEMS Informatics?

At risk of sounding cliche’, it really is about the people. Please visit the GEMS website to learn more about the passionate group of researchers, professors, technologists, and administrators who make GEMS possible.

On the technical side, the GEMS platform is a thoroughly modern platform developed using some of the latest technologies and techniques.  The various components of the GEMS platform leverage both private and public cloud services. Docker containers form the basic infrastructure of the platform.  Modern web development technologies are used to implement the platform.

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota