Revolutionizing Agriculture

In recent years, the fusion of artificial intelligence (AI), machine learning (ML), and the Internet of Things (IoT) has opened up unprecedented opportunities for agricultural innovation. One groundbreaking development in this arena is the implementation of real-time geoinformation systems, which promise to revolutionize agri-environment research by enhancing data quality, scalability, and cost-efficiency.

The Rise of Spatial IoT in Agriculture

As the agricultural sector increasingly relies on AI and ML for knowledge discovery, the need for large, high-quality datasets has become paramount. Spatial IoT technologies, which involve deploying internet-connected sensors throughout agricultural environments, have emerged as a crucial tool in this data-driven landscape. These sensors collect real-time, high-resolution geospatial and temporal data, enabling researchers to monitor and analyze agricultural systems with unprecedented precision.

Challenges in IoT Implementation

Despite its potential, the implementation of IoT in agriculture presents significant challenges. Managing large fleets of devices while maintaining data quality is a complex task. Scientists often start with one-off prototypes, but scaling these to thousands of internet-connected devices requires overcoming numerous technical and logistical hurdles.

Case Studies in IoT System Development

The University of Minnesota’s Real-Time GeoInformation Systems Lab has been at the forefront of addressing these challenges. Since 2019, the lab has developed and deployed over 2,727 IoT devices across four continents. This extensive deployment has provided valuable insights into creating a generalizable, open-source spatial IoT system tailored for agricultural research. This work was summarized in a recent pre-print on Arxiv.com (Runck et al. 2024).
One key aspect of the lab's work has been the iterative development of the IoT system, progressing through three major and fourteen minor versions. Each iteration has refined the system's capabilities, from improving sensor accuracy to enhancing data transmission reliability. The current version of the system is designed to be scalable, ensuring that it can be deployed widely while maintaining high data quality.

Practical Applications

The applications of these IoT systems are diverse and impactful. For instance, in irrigation management, real-time data on soil moisture and temperature help optimize water usage, crucial in regions facing water scarcity. Similarly, in plant winterkill research, sensors monitor microclimates to understand the conditions leading to crop damage in cold environments. These insights enable farmers to adopt preventive measures, safeguarding crop yields.
Another notable application is in meteorological observations. Deploying IoT systems for weather monitoring provides granular data that enhance the accuracy of weather forecasts, which is vital for agricultural planning and risk management. For example, in Minnesota and Malawi, extensive networks of weather stations equipped with IoT sensors collect data that support both local farmers and broader agricultural research initiatives. However, paying attention to data quality, access and interoperability matters, often coupled with fit-for-purpose analytic pipelines, is key to ensuring real-time, geo-sensed data lead to actionable, data-driven informatics products.

The Role of Open Source in Scaling IoT

Open-source technology plays a crucial role in the scalability of IoT systems. By making design files and code publicly available, researchers can build on existing work, ensuring broader adoption and continuous improvement. This collaborative approach aligns with the scientific principles of transparency and reproducibility, fostering innovation across the agricultural research community.

Moving Forward: GEMS Sensing Service

To support the ongoing development and deployment of IoT systems, the University of Minnesota has established GEMS Sensing, a service organization within its GEMS Informatics Center. This initiative aims to provide turnkey IoT solutions for researchers, ensuring that the technology is accessible and sustainable. By offering both internal and external sales models, GEMS Sensing facilitates public-private partnerships, driving further advancements in digital agriculture.

Conclusion

The integration of real-time geoinformation systems into agricultural research marks a significant leap towards smarter, more sustainable farming practices. By harnessing the power of spatial IoT, researchers can collect and analyze data at an unprecedented scale and resolution, paving the way for innovative solutions to some of agriculture's most pressing challenges. As these technologies continue to evolve, the future of agriculture looks increasingly data-driven and resilient, promising enhanced productivity and sustainability for the global food system.

Image: Generated with Firefly. of A modern agricultural field with IoT sensors placed at various points.

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

APIs: Now well-documented and much easier to use

One of the big frustrations in using computing to solve large, multidisciplinary challenges is managing data sets from different disciplines. Often, you have to go to each individual site and download the entire dataset. Then you have to parse out the subset of data fields you want within the geographies, spatial resolutions and time periods you care about. It is still the case that a few groups provide their data in the form of an Application Programmer Interface (API), where the data are served in a structured form with clear metadata documentation. Data can be sliced and diced how you like, selecting subsets of geography, time, and variables of interest. Once you sign up and obtain an API key, it just takes a few lines of code in Python or R to establish a connection and query at will!

GEMS has been building out a portfolio of APIs since 2021 across a range of useful datasets seeking to span the full Genetics x Environment x Management x Socioeconomic data landscape. Those who tried GEMS Exchange before will know that we used to have a middle layer managed by RapidAPI. Users found that cumbersome and confusing, so we are now using our own Apache APISIX server within our own web pages to serve you your key and monitor usage. We’re confident that your experience will be super easy this time around. Let’s get you started!

First, check out which APIs might interest you at our GEMS Exchange page. To obtain your API key simply click here for key. (Note you will need to have a Globus.org account, which is free – or you can connect via your academic institution, Google account, or ORCID). Once you know which APIs interest you, explore our collection of Jupyter notebooks in Github that give you practical guidance on how to use them. Many of the APIs we offer use the GEMS Grid which help ensure they are interoperable. And the GEMS Grid itself has recently been made open source, so you can place your own data sets on the Grid and interoperate with the community.

We are always happy to hear of useful datasets that could be added to the GEMS gridded collection in Exchange, so by all means reach out with suggestions or queries here.

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

We are delighted to announce that we have just released the GEMS Grid code library, where the code is under the open source Apache 2.0 license, which allows anyone to use the code for commercial or non-commercial purposes – you simply need to provide attribution to GEMS Informatics when you use or modify it.

Just before we at GEMS Informatics started developing Application Programmer Interfaces (APIs) in agriculture for GEMS Exchange, GEMS geospatial expert Jeffery Thompson worked with others in the GEMS team and colleagues at NSIDC to develop the GEMS Grid, a hierarchical discrete global gridding system. This Grid has allowed us to provide data sets at different resolutions ranging from 36 km to 1 m, and still have them remain functionally interoperable. The interoperability is possible because we have written the code to allow users to project data onto the grid, aggregate data to coarser resolutions, and, notably, also disaggregate data to finer resolutions. The latter operation is ordinarily a difficult problem, but is made easier, as I discuss below, since we enable the users of our code to thoughtfully address it in a standardized, replicable way.

Many problems in agriculture (e.g., understanding the spatial location of crop production) require equal area parcels of land to do proper calculations. Working with strict lat-lon coordinates won’t suffice as areas near the equator are significantly different in size as areas near the poles. The GEMS Grid preserves equal-area assumptions as it divides land, so you can do these calculations with confidence, and preserve aggregation-disaggregation consistency in the data, even if you are not a GIS expert.

So let’s look at the 5 options for disaggregation that GEMS geospatial developer Olena Boiko included in the GEMS grid toolbox, schematically described in the figure she developed above.

Option 1. Value transference. 
In this case, if you were to subdivide a 3 km² resolution grid cell into 9 x 1 km² cells, this option would be appropriate for any value that is deemed roughly constant throughout the area applied. Examples would be rainfall in inches or grain yield in bushels / acre.

Option 2. Even value division. 
Sometimes the quantity measured in a cell represents a cumulative value for the area in which it is reported. In this case, if the parent cell is homogenous, then splitting it up into 9 equal-area pieces would require that you divide the value in each equivalent cell by a factor of 9. Examples where this selection makes sense include grain production in bushels, crop acreage, and population.

Option 3. Value transference with a mask. 
This one is similar to Option 1 except we are no longer making the assumption that the distribution of values in the parent cell is spatially homogeneous. For example suppose you were measuring grain yield, but you knew that 3 of your nine cells had buildings occupying them (see white areas in the Figure). In this case you only transfer your values to 6 remaining cells (colored peach) that have arable land. Cells are binary with this option (i.e., either allowed a value or not).

Option 4. Even value division with a mask. 
Analogously, you can mask out cells in the value division case when you know that your daughters cells are not all equal. This is just like the case in Option 3, except you divide your parent-cell value evenly by the number of viable daughter cells. In this pictorial example, there are 6 viable daughter cells, so each gets a value of 900/6 = 150. This would be appropriate if you were computing grain production in bushels and you had a total value that needed to be split up across the 6 arable daughter parcels.

Option 5. Flexible division with a mask. 
This scenario is the most flexible, and allows the user to create a master mask with arbitrary weights at each daughter cell. It allows you to block off daughter cells entirely, and prescribe the relative weights of all remaining daughter cells. This is ideal for situations where you are allocating crop distributions and you want to avoid certain land use features (e.g., lakes, forests, housing) and probabilistically distribute the remaining crop areas (e.g., with higher probability near soils with a high SSURGO National Commodity Crop Productivity Index).

I’m confident these flexible disaggregation tools will provide much easier, more accurate, and replicable solutions for your particular spatial analytic problem. So please give them a try!

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

Researchers that are transitioning from doing analysis locally on their laptop to remote or large scale analysis often fall into a few common traps. Moving from local analysis isn’t necessarily difficult, but there are some good habits to develop that will make your transition more productive.

Establish a small test set

The set should be representative of the variety of data in the full data set if possible, but doesn’t necessarily need to produce results similar to that of the full analysis. There are two key wins that come from having a small test set predefined.

Development loop speed 

There is a temptation, especially for those who are used to working on smaller datasets, to write code and then run it against the complete dataset. However, code is almost never perfect on the first, second, or even third attempt.  If running a test takes several minutes or more that can force you to focus your attention elsewhere while you wait, reducing your efficiency and causing even longer delays. It is best to have a small, even if nonsensical, test set that can run in 10 seconds or less. Rapid iteration is key, especially during early development phases.

Validating changes before large runs

The point of this set is to establish a set of unit tests that exercise the code which can be used to verify core functionality. There is little worse than starting a long running task, getting most of the way through the task, and then encountering an error due to a typo or other simple oversight that requires starting the process over again. Having tests that check for basic functionality at the edges of expected input will save a lot of time by preventing waiting for executions that can never successfully complete.

Additionally, as new errors are encountered based on real data make sure to extend both the code and the tests to appropriately handle the new cases that were improperly handled.  This does not necessarily mean to solve data problems during execution, but when faults or exceptions occur due to type errors, etc, catch the error and log it in a way that can be presented to the user as part of a list of data cleaning tasks to perform before the next attempt. In a large dataset, only returning the first encountered error is a sure way to make the task take far too long. Instead, log each case as they are encountered, while ensuring that the program keeps going all of the way to the end generating an easy-to-understand error log.

Execution environment

One of the keys to ensuring reproducible behavior is tracking which external libraries are used in a program and more specifically which versions. In modern software development we are very dependent on others and their contributions to make our own development processes tractable. As with everything there are pros and cons to the way software is currently being developed. Further challenges and opportunities will arise as AI generated code, or AI assisted development becomes more common.

For now, ensure familiarity with the concept of semantic versioning. In brief, the first number is the major version, the next version is the minor version, and the third version is the patch version. For most purposes, when setting up an execution environment a good rule of thumb is to use the minor version of a library as the one to base an environment on. This should allow for fixes to be applied at the patch level, but more substantive changes can be adopted as needed.

Required libraries

In this case, required libraries only refer to the libraries that are directly imported and invoked by the code under development. Each required library may also include subsequent libraries, but trying to enumerate these or their versions will make building an execution environment much more challenging. The tradeoff is that these dependencies of dependencies can introduce unexpected changes.  

Packaging systems

Once a requirements list has been established, most programming environments have tools that can be used to create an environment that includes the contents of the requirements list. In Python, this can be achieved with the pip command. However, it is important to note that some libraries will be C or C++ based and require an appropriate compiler to build.

This is where a tool like conda comes into play. Where a tool like pip will install dependencies from their source code, conda maintains repositories with prebuilt binaries instead. Conda also supports multiple languages. The tradeoff is that conda adds a significant amount of disk usage to an environment.

If the intent is to rebuild an environment on every system where the code will be executed this is not much of a problem. If, however, ensuring the exact same version of required libraries are available and the environment itself will be packaged for distribution the additional gigabytes of storage can be more of a disadvantage. 

Container Images w/Docker, Apptainer, or Kubernetes

Over the last decade Docker images and other container infrastructure providers have become more popular as a way to provide a lightweight way to build a frozen, all inclusive, execution environment. This allows a researcher to deploy an identical execution environment and code from their laptop to a High Performance Computing or Cloud Computing environment.

Many CyberInfratructure providers such as the Minnesota Supercomputing Institute and the NSF’s ACCESS provide facilities for running containers from provided images using Apptainer. In cloud computing Kubernetes or other tools might be more readily available. 
 

Putting it all together

With some planning, a researcher could develop a good test suite to ensure that their code produces expected results, create a reproducible environment to the degree of their choice, and get the best of both rapid development locally using a minimal data set as well as the ability to then run the full data set on appropriately scaled hardware elsewhere. Each case will have unique circumstances that are difficult to provide a standard solution for. However, hopefully this introduction to some of the possibilities will help researchers choose a path that will help them develop a robust approach to working with big data.

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

We’re thrilled to announce that GEMS is leading a novel 2-year project funded by the Minnesota Invasive Terrestrial Plants and Pests Center (MITPPC) with support from the Minnesota Environment and Natural Resources Trust Fund.  "Linking soybean aphid losses to technology investment decisions"  a transformative project aimed at developing a flexible, evidence-based, bio-economic evaluation workflow to characterize the long-term, probabilistic extent of soybean aphid damage throughout Minnesota.

Our Mission: A Repeatable and Extensible Pest Risk Assessment Tool to Inform Decision Making

Soybean aphids have emerged as a significant arthropod pest impacting soybeans in North America since 2000. However, a systematic effort to collect, analyze, and report on yield losses caused by this pest in farmers' fields has been notably absent, particularly concerning the longer-term, state-wide perspectives that are crucial for strategic R&D and policy decisions.

Our goal is to comprehensively assess the risk posed by the soybean aphid for the state of Minnesota to help inform decisions regarding research investment and pest mitigation strategies. Our evaluation framework breaks new ground by factoring in both the spatially- and temporally-variable pest risk elements that many prior efforts have overlooked. Our flexible evaluation approach is designed to deal with either data-poor or data-rich scenarios while taking into account the geographical extent, frequency, and severity of damage to undertake regional and meso-scale risk assessments. 

Our project, 'Linking Soybean Aphid Losses to Technology Investment Decisions,' is breaking new ground by developing a flexible, evidence-based framework for estimating crop losses that account for the dynamic nature of pest risks. Through interdisciplinary collaboration, we're forging a path to innovative solutions in agriculture.

-Yuan Chai

Forward-Looking Impact: Shaping Tomorrow's Risk Management Strategies

This project entails close collaboration between the GEMS Informatics Center (PI Dr. Philip Pardey and co-PI Dr. Yuan Chai) and the Department of Entomology (co-PI Dr. Robert Koch), leveraging a spectrum of expertise in pest, crop, and socio-economics. With cutting-edge data and analytics support, we're developing a data-driven, replicable, and extensible framework for meso-scale ex-ante pest risk evaluation of soybean aphids. The project's impact stretches far beyond the laboratory, resonating with stakeholders invested in the field of crop pest management. Our findings will empower MITPPC, state government agencies, academic units, and crop commodity groups to strategically allocate resources for targeted investments in pest management strategies. Moreover, our approach is designed to be extendable to a wide range of crop pest and disease challenges in Minnesota and beyond.

Join Us on this Journey! 

Join us on this transformative journey, where our collaborative effort is poised to shape the future of agricultural resilience and resource optimization decisions. Stay tuned for updates on findings, methodologies, and the broader implications of our work by signing up for the MITPPC newsletter and by visiting our project page. Together, let's revolutionize the landscape of agricultural research and pest management! ??

Photo Attribution: Christina DiFonzo, Michigan State University / © Bugwood.org

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

Assessing BMP Effectiveness for Water Quality Outcomes with Remote Sensing

Virtually everyone in Minnesota and bordering states whose profession has a direct stake in Water Conservation assembled in Downtown St. Paul’s RiverCentre October 17-18 for the annual Water Resources Conference. I had the honor of chairing a Special Session at the conference titled “Assessing BMP Effectiveness for Water Quality Outcomes with Remote Sensing.” The session featured two very foundational efforts that we are attempting to bridge together in partnered research at GEMS: (1) The Minnesota Department of Agriculture’s (MDA) Minnesota Agricultural Water Quality Certification Program (MAWQCP), directed by Brad Jordahl Redlin, and (2) the Department of Forest Resources’ remote-sensed Water Quality assessment for our 10,000+ lakes, led by Leif Olmanson.

The 90-minute session was organized so that each of the three speakers (one had dropped out last minute for logistical reasons) had 20 minutes to speak plus two minutes for burning questions. At the end the audience had 15 minutes for questions with the entire panel of speakers. There were about 30 very inquisitive people in the audience, who peppered the speakers with questions throughout.

Updates on the MDA’s certification program

Brad Jordahl Redlin kicked off the session with an outstanding talk that outlined the principles on which the certification program he founded and directs was created, and highlighted the progress that has been made. Literally over a million acres have now been certified, spread across over 1400 producers. In exchange for regulatory certainty for a period of time, these producers allow his government agency to (1) assess their farm practices, (2) suggest new management practices (sometimes with additional incentives such as subsidized equipment from a grant), and (3) carry out audits of their new practices. The program, which has been operational for nearly a decade, is well beyond the stage of corralling early-adopters. New participants span the spectrum of producer demographics in age, farm size, and more.
 

Scaling lake quality measurement

Leif Olmanson stepped up to review an activity that he first spearheaded twenty years ago – using remote sensing to measure the water quality of Minnesota lakes. In the interim, he has overseen steady progress using new generations of satellites (from Landsat to Sentinel), multiple metrics (clarity, chlorophyll, organic matter), and the frequency of data reporting (from once every 5 years to ~weekly lake snapshot and monthly pixel-level composites of all 10,000+ lakes). Data is made available to the casual user via a friendly Lake Browser interface, and in numeric form pixel-by-pixel for data wizards on GEMS Exchange.
 

GEMS bridging the two programs

David Porter rounded out the session discussing the improvements he has made in creating algorithms that properly remove clouds and aerosols from the satellite images. This is a prerequisite for feeding all of the regularly spaced (~5 day) images of a field through a growing season into a sophisticated Recurrent Neural Network (RNN, a type of Machine Learning algorithm) that his colleague Anubha Agrawal has been preparing. As described in this case study, David, Anubha, and I intend to use the RNN trained on known, certified management practices (e.g., strip till / no till, cover crops) from the MAWQCP to make a prediction of what practices growers are making on uncertified farmland. We then will trace these producer fields to the downstream lakes in each watershed, and see what covariates (e.g., slope, soil type, distance to waterway) affect the correlation of % farm practice adoption vs. downstream lake quality in a watershed.

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

GEMS’s new NSF award

Training ag and food scientists how to make the most out of supercomputers

Do you write code (maybe in R, maybe in Python)? Maybe you’re an agri-food scientist in a lab, or in a similar field in industry. Perhaps you’ve tried to write some code to get your work done, figuring you can run it on your laptop. But when you churn on that data you find after two weeks of keeping your laptop open to run the code that it’s probably not going to finish any time soon… Epiphany. You need to do this in a smarter way! Well you’re in luck – GEMS was just awarded a grant from NSF titled “Cyber Training: Pilot -- Breaking the Compute Barrier, Upskilling Agri-Food Researchers to Utilize HPC Resources” and you are exactly the audience we wish to reach with the courses we are developing.

The Team

The course material is being developed by an interdisciplinary team including:

• Instructor Joe Axberg, who works on supercomputing infrastructure by day but is also a part-time instructor after hours in the IT Infrastructure Program in the College of Continuing and Professional Studies.
• Project mastermind, Ali Joglekar, an agricultural economist by trade with real-world experience working with smallholder farmers in East Africa
• Kevin Silverstein, a cofounder of GEMS Informatics and expert bioinformaticist with specialties in molecular plant-microbe interactions and large-scale informatics problems
• Jesse Erdmann, systems architect and guiding force behind much of the inner workings of GEMS infrastructure.
• Ben Lynch, Director of MSI, with a long track record of innovating in the scientific computing domain

Scope

The proposal identifies 7 separate course modules that each contain a significant number of concepts, representing a broad swath of content to help up-skill agri-food researchers to use HPC:

1. Introduction to High-Performance Computing (HPC) for Agri-Food Researchers
2. Introduction to Cloud Computing for Agri-Food Researchers
3. Hands-On: Use HPC to Analyze More Data and Faster
4. Leveling-Up I: Expanding HPC Skills for Agri-Food Research
5. Computer Science for the Agri-Food Researcher
6. Leveling-Up II: Advanced HPC Concepts
7. High Performance Workstations and Servers for Agri-Food Researchers

It is expected that the number of modules proposed and the content within each module may change as we build out the course and modules during the development phase(s). The goal of our iterative course development process is to refine this broad set of concepts and craft cohesive modules each with a duration of 2-6 hours of lecture materials (depending on the module). Some modules will span multiple weeks. Several rounds of purposeful learner feedback is integral to our course development process. 

The course modules will also be “stackable” allowing learners to have some ability to “pick and choose” modules in line with their competencies and interests.

Timeline for delivery

As the diagram indicates, we are planning to offer each HPC for Agri-Food Researchers course over a 10-12 week period of time. The proposed course structure, in line with our other GEMS Learning offerings, includes asynchronous and synchronous instructional elements. Lecture materials will be delivered via videos and readings, which students are expected to review ahead of the weekly 1.5 hour class session. This instructor-led class time is primarily dedicated to synthesizing and reviewing the content being taught. To solidify learning objectives, students will be expected to engage in regular assessments and hands-on learning exercises outside of the weekly class session. We estimate that students will be responsible for 6+ hours/week of self-directed learning activities, depending on their skill-levels.

You can influence our content!

The good news is we are still in development, so you can influence these course offerings. Please fill out this survey now to make your voice heard!

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

Behavioral economists are studying “nudges” to prompt behavior change in agriculture. Accounting for production economics can help make nudges more effective for reducing agricultural run-off.

Water pollution caused by agricultural nitrogen runoff is a pervasive problem worldwide. In a recently published paper (Food Policy), GEMS researchers (Yuan Chai and Philip Pardey) teamed up with University of Western Australia collaborator (David Pannell) to provide a new slant on this age-old problem. Combining insights from both production economics and behavioral science opens up new and practical avenues for nudging farmers towards eco-friendly practices that can mitigate water pollution. 

Production Economics Insights

The authors highlight three key empirical findings that offer opportunities to reshape farming practices and reduce nitrogen pollution. First, many farmers tend to apply more nitrogen than necessary, driven by yield maximizing or other motives. This can lead farmers to use more fertilizer than is economically optimal, which not only incurs avoidable costs but also exacerbates the water pollution problem. Second, farmers may imagine that, contrary to the evidence, using more fertilizer is a risk-reducing strategy. Thirdly, the profit-to-fertilizer-rate relationship is flat near the optimum, suggesting that farmers can curtail their fertilizer usage without incurring significant, if any, private costs (through foregone income).

Nudges

Leveraging these three insights, the authors draw on emerging findings from behavioral economics to identify a number of potentially transformative approaches to curbing agricultural pollution. The authors make specific suggestions for ways in which behavioral science could contribute in these areas, including understanding farmers’ perceptions about nitrogen, yield and profit (detecting biases), understanding the influences of these biases on behavior, understanding farmers’ perceptions of and preferences for different policy responses, and designing behavioral nudges that capitalize on the specific insights we have highlighted regarding the production economics of nitrogen fertilizer. Nudges could include strategies that aim to correct biases in perceptions (e.g., about yield increases or risk reductions from applying N),or tap into social and psychological factors that influence farmers’ decision making.

Behavioral changes that are deemed nudgeable (i.e., where farmers bear little or no cost) have a lot of merit. The nudges revealed through this research could go a long way to cleaning up water bodies in ways that do not undermine food security or farmer livelihood concerns: a rare win-win result.

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

In 2019, we began working on a project in partnership with the Centre for Agricultural Transformation to accelerate the transformation of Malawian agricultural production away from tobacco to other sustainable agri-food value chains that can provide positive economic and livelihood outcomes. Growing agri-food markets requires farmers to have more affordable and timely access to the right farm inputs, financial services, and expanding fresh and processed commodity markets at home and abroad. The obvious initial questions were which value chains to prioritize, and where in the country does one target interventions to best meet growing urban and export opportunities? But where does one find the requisite market intelligence?

While there are plenty of one-off, glossy, consultancy reports available for specific value chains in Malawi, they left us frustrated. The types of data reported are useful in providing broad contextual information but make it difficult to reach actionable conclusions. By their very nature, these reports provide static snapshots that don’t lend themselves to more user-driven questions.

Not All Farmers Are the Same

Farmers, and the farms they manage, vary in a myriad of ways that have important market development implications. Many Malawian farmers are smallholders, but there are mid-to-larger sized farms as well. They grow different crops, in different locations, with different degrees of input and output market participation, and vary in their resilience to climate and market risks. Identifying and then targeting appropriate technology, innovations and other investments towards the right market segment is key to agricultural transformation.
 

The Market Segmentation Tool (otherwise known as MST) is a user-driven, digital “market intelligence” tool to better inform value-chain investments. It is designed for all sorts of agri-food market participants in Malawi, notably farmer-based organizations, non-profit agencies, for-profit firms, startups and established enterprises, NGOs, government, and research organizations. This first-in-class tool was designed by the GEMS Informatics team at the University of Minnesota to fill an important agricultural development void. It allows users to better tailor their products and services to the unique needs of the particular on- and post-farm market segments they seek to serve.

Intuitive Interactive Intelligence

Available through a publicly available URL, MST uses sophisticated on-the-fly analytics to allow non-technical users to quickly interrogate large, complex, and disparate data on the environment, income, market accessibility and agricultural production. MST draws on more than 100 relevant variables in a spatially explicit way and presents insights through simple charts, graphs and mapped representations. These customizable figures can be saved via a screenshot and the underlying data are available for download in a .csv format at a district-level.

For example, if a user was interested in exploring the production practices of smallholders, rather than simply looking at farmers who cultivate less than 1 hectare of land, users can create a more nuanced smallholder definition. Within MST agricultural households can be readily categorized based on up to 6 segmentation variables: poverty, cropland assets, livestock assets, output market orientation, input market orientation, and income diversity. For instance, users could characterize a market cohort defined by agricultural households that cultivate between 0.5 and 1 hectare and have an average per capita consumption expenditure between 400 and 800 Kwacha per day.

Once the market cohorts are defined, the user then navigates through the dashboard to explore the historic production, market orientation and income diversification practices of households that meet the criteria defined by our poverty and cropland thresholds. Data are displayed in multi-level geographical maps as well as infographic charts, many of which are interactive, and all of which can be filtered by farm type, gender, and tobacco producers across three World Bank LSMS survey waves (2010-11, 2015-16, and 2019-20).

Spatially explicit market intelligence is further enhanced with grid-based data on market access and agro-ecological information such as time to market, road density, population density, agro-ecological zones, climate variables (including temperature and precipitation) and soil types, which can be accessed through the environment and market access modules.

Bottom Line. Our tool enables users to identify which farmers, for which markets, in which locales, make the most sense for their particular value-chain investments.

MST can be explored at agroinformatics and user help videos are available at our YouTube channel.

Feel free to contact us if you have any questions or comments on the tool, which has been designed to be extensible to additional variables and other countries with the ability to accommodate open and private data.

This posting was produced as part of the Centre for Agricultural Transformation, an effort led by Land O’Lakes Venture37 and funded with a grant from the Foundation for a Smoke-Free World, Inc. (“FSFW”), a US nonprofit 501(c)(3) private foundation. The contents, selection and presentation of facts, as well as any opinions expressed herein, are the sole responsibility of the authors and under no circumstances should they be regarded as reflecting the positions of FSFW.

• Centre for Agricultural Transformation
• Land O’Lakes Venture37
• Foundation for a Smoke-Free World

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota

Making Sense of Agriculture from Minnesota to Malawi

GEMS Sensing has coupled world-class engineering design and build expertise with first-rate agricultural science smarts to realize a robust, extensible sensing system for agri-food research anywhere in the world.

Since early 2020, GEMS Sensing has partnered with the Centre for Agricultural Transformation (CAT) in Malawi to pilot the local deployment of GEMS research-grade weather stations and its associated data visualization and sharing tool. Piloting a network of weather stations across Malawi involves much more than working through the logistics of manufacturing, delivering and deploying the necessary hardware and software. It also involves identifying partners willing to trial the weather stations on their research stations/farms and training local CAT personnel to enable the deployment, particularly in remote rural areas with uncertain cellular connectivity. Working through the CAT provides a unique opportunity to engage with private and public agribusiness throughout Malawi to test GEMS Sensing technologies in a low-income, smallholder farmer, tropical environment. 

Over the past five years, the GEMS Sensing team has used feedback from our field partners to develop ever-improving scientific-grade, plug-and-play sensing devices coupled with a secure web-based informatics service. Weather stations aren’t new, but GEMS Sensing has created a highly customizable system that is built specifically for the varied sensing needs of researchers and their collaborators. The system's core logger box can be used for scientists with vastly different sensing requirements, from a full-stack meteorological station supporting a host of on-station or in-field crop performance studies to more specialized applications aimed at monitoring the below ground conditions that affect plant growth and health. The system's configurable and easy-to-use dashboard enables the integration, exploration, and sharing of real-time sensor data from multiple, perhaps distant, sites through a single portal.

Tracking the real-time, micro-level weather and edaphic patterns impacting agricultural production is crucial for a whole host of reasons, including advancements in breeding, agronomy, and extension efforts and the livelihoods of the farmers responsible for this production. These types of data are used for modeling weather events over large areas (e.g., rainfall and temperature) or informing management decisions like when and how much to irrigate a crop. However, being able to collect, store, clean, explore, share, and analyze these data from multiple sites, isn’t always straightforward. With this motivation in mind, the team has coupled world-class engineering design and build expertise with first-rate agricultural science smarts to realize a robust, extensible sensing system for agri-food research anywhere in the world.

GEMS Sensing in sub-Saharan Africa

For the Malawi pilot project, we worked with a South African-based manufacturer to produce the GEMS Sensing stations. The full-stack weather stations are configured to collect eight parameters every 15 minutes: air temperature, humidity, barometric pressure, rainfall, soil moisture, soil temperature, wind speed, and solar radiation. Data are sent via cellular network to a cloud-based database, processed for quality, and made accessible to partners via a user-friendly, web-based portal.

The primary objectives of the pilot exercise were to better understand:

1. Tropical environmental impacts on the sensing hardware;

2. Total cost of ownership from a user perspective (i.e., costs of acquisition, installation, and maintenance) involved in deploying digital ag weather sensors in a low-income context with limited infrastructure support; and

3. Malawian agribusiness’ demand, willingness to pay and potential use cases for networked weather stations (or related real-time sensing needs).

To date, the CAT team has deployed nearly 40 solar-powered weather stations across the spatially and temporally variable agro-ecologies that affect agricultural outcomes throughout the country. The stations continue to stream real-time data from experimental research stations, commercial farms, and smallholder farms. The pilot project is set to wrap-up in December 2023, though the CAT may continue to operate some of the weather stations beyond that time.

Learnings and Lessons

Scaling up a pilot project in Malawi from our operations based in Minnesota during the height of the global pandemic was not without its challenges. The original game plan was to travel with a handful of sensors to Malawi so that our technical team could work closely with CAT personnel and deployment partners to scope out the entire constellation of technical, human capital, cell connectivity, local logistics, device assembly and deployment partner protocols and other factors that could likely affect the odds of a successful outcome. Based on this test deployment we had envisaged tweaking the technical specs, revising our training and related deployment protocols before scaling up deployment throughout the country.

COVID-19 had other plans. It shut down all travel to Malawi, disrupted supply chains for computer chips that were critical to fabricating the GEMS designed sensors, and complicated shipping logistics. Undeterred, we pivoted our whole operation to enable a remote deployment, which, despite our best efforts, was not without its own complications.

Persistence ultimately paid. As we began scaling field deployment in April 2022, our technical team became adept at diagnosing issues long distance, ably supported by a great ground team in Malawi. Fluky cell connectivity issues seemed to be associated with variable soil properties that were vastly improved by simply raising the logger box further off the ground. Unexpected down-time issues were eventually tracked to battery depletion problems that arose from locally assembled devices that sat turned on for lengthy time before travel resumption enabled field deployment to proceed. The fix, change out the run-down batteries.

All told, our GEMS Sensing devices exceeded our expectations, even withstanding Tropical Cyclone Freddy that hit Malawi in February 2023!

Our team visited Malawi in May and traveled with the local CAT sensing team throughout the Southern and Central regions to get first-hand feedback from a host of public and private partners who participated in the GEMS Sensing pilot project. We made site visits with DARS research scientists, commercial agrifood businesses such as Bayer, Pxyus and Global Seeds, and One Acre Fund which services smallholder farmers directly. These partners were uniformly enthusiastic about the ready access to automatically streamed and easily accessible localized weather data. They also highlighted numerous opportunities for leveraging the streaming weather data streams with various backend analytics to tailor informatics solutions to the particular and varied agri-food production problems they faced.

GEMS sensors are currently being deployed with success across four continents. However, this represented our first widespread test of the system in a logistically challenged, low-income African agricultural setting. Prior to the pilot we had all sorts of questions as to whether or not the system would even work, and if so, how reliably and securely. Importantly, would anyone find practical value in such a sophisticated, real-time sensing system in a country dealing with many infrastructure and technological constraints.

Weather matters a lot for agriculture, whether you are farming in Minnesota or Malawi. While Minnesota farmers have access to all sorts of weather related data products, locally accurate, real-time data is still relatively scarce. In Malawi, rurally-relevant weather data of any sort is largely absent. The last time we checked, NOAA (National Oceanic and Atmospheric Administration) live stream weather data from just one site in Malawi, the Blantyre International airport! This pilot demonstrated the potential for changing this reality for Malawian farmers and the numerous public and private agencies that support their operations.

Even with its many hurdles and holdups, this pilot exercise revealed the feasibility and the latent demand for access to locally sensed weather data. With the lessons learned from our pilot deployment we are excited about the possibility of using GEMS Sensing’s data generation and analytical capabilities to support data-driven decisions that strengthen existing and emerging value chains in Malawi, as well as unlock new agri-food value-chains throughout the world.

This posting was produced as part of the Centre for Agricultural Transformation, an effort led by Land O’Lakes Venture37 and funded with a grant from the Foundation for a Smoke-Free World, Inc. (“FSFW”), a US nonprofit 501(c)(3) private foundation. The contents, selection and presentation of facts, as well as any opinions expressed herein, are the sole responsibility of the authors and under no circumstances should they be regarded as reflecting the positions of FSFW.

• Centre for Agricultural Transformation
• Land O’Lakes Venture37
• Foundation for a Smoke-Free World

​This activity supported in part by MnDRIVE Global Food Ventures, University of Minnesota