Research
The GMAI team is committed to deliver the most valuable synthetic data for computer vision tasks. As a team of high-profile scientists we conduct research toward enabling synthetic data and generative AI.
State-of-the-Art Wildfire Simulations
Wildfires are a complex physical phenomenon that involves the combustion of a variety of flammable materials ranging from fallen leaves and dried twigs to decomposing organic material and living flora. All these materials can potentially act as fuel with different properties that determine the progress and severity of a wildfire. In this paper, we propose a novel approach for simulating the dynamic interaction between the varying components of a wildfire, including processes of convection, combustion and heat transfer between vegetation, soil and atmosphere. We propose a novel representation of vegetation that includes detailed branch geometry, fuel moisture, and distribution of grass, fine fuel, and duff. Furthermore, we model the ignition, generation, and transport of fire by firebrands and embers. This allows simulating and rendering virtual 3D wildfires that realistically capture key aspects of the process, such as progressions from ground to crown fires, the impact of embers carried by wind, and the effects of fire barriers and other human intervention methods. We evaluate our approach through numerous experiments and based on comparisons to real-world wildfire data.
A. Kokosza, H. Wrede, D. G. Esparza, M. Makowski, D. Liu, D. L. Michels, S. Pirk, W. Pałubicki
Scintilla: Simulating Combustible Vegetation for Wildfires
ACM Transactions on Graphics (Proceedings of SIGGRAPH), 2024
Simulating Cyclones and Tornadoes
Cyclones are large-scale phenomena that result from complex heat and water transfer processes in the atmosphere, as well as from the interaction of multiple hydrometeors, i.e., water and ice particles. When cyclones make landfall, they are considered natural disasters and spawn dread and awe alike. We propose a physically-based approach to describe the 3D development of cyclones in a visually convincing and physically plausible manner. Our approach allows us to capture large-scale heat and water continuity, turbulent microphysical dynamics of hydrometeors, and mesoscale cyclonic processes within the planetary boundary layer. Modeling these processes enables us to simulate multiple hurricane and tornado phenomena. We evaluate our simulations quantitatively by comparing to real data from storm soundings and observations of hurricane landfall from climatology research. Additionally, qualitative comparisons to previous methods are performed to validate the different parts of our scheme. In summary, our model simulates cyclogenesis in a comprehensive way that allows us to interactively render animations of some of the most complex weather events.
J. A. Amador Herrera, J. Klein, D. Liu, W. Pałubicki, S. Pirk, Dominik, L. Michels
Cyclogenesis: Simulating Hurricanes and Tornadoes
ACM Transactions on Graphics (Proceedings of SIGGRAPH), 2024
Understanding Ecosystems
Due to the enormous amount of detail and the interplay of various biological phenomena, modeling realistic ecosystems of trees and other plants is a challenging and open problem. Previous research on modeling plant ecologies has focused on representations to handle this complexity, mostly through geometric simplifications, such as points or billboards. In this paper we describe a multi-scale method to design large-scale ecosystems with individual plants that are realistically modeled and faithfully capture biological features, such as growth, plant interactions, different types of tropism, and the competition for resources. Our approach is based on leveraging inter- and intra-plant self-similarities for efficiently modeling plant geometry. We focus on the interactive design of plant ecosystems of up to 500K plants, while adhering to biological priors known in forestry and botany research. The introduced parameter space supports modeling properties of nine distinct plant ecologies while each plant is represented as a 3D surface mesh. The capabilities of our framework are illustrated through numerous models of forests, individual plants, and validations.
M. Makowski, T. Hädrich, J. Scheffczyk, D. L. Michels, S. Pirk, W. PaÅ‚ubicki
Synthetic Silviculture: Multi-scale Modeling of Plant Ecosystems
ACM Transactions on Graphics (Proceedings of SIGGRAPH), 2019
Augmenting Datasets with GenAI
We introduce a specialized procedural model for generating synthetic agricultural scenes, focusing on soybean crops, along with various weeds. The model simulates distinct growth stages of these plants, diverse soil conditions, and randomized field arrangements under varying lighting conditions. The integration of real-world textures and environmental factors into the procedural generation process enhances the photorealism and applicability of the synthetic data. We validate our model's effectiveness by comparing the synthetic data against real agricultural images, demonstrating its potential to significantly augment training data for machine learning models in agriculture. This approach not only provides a cost-effective solution for generating high-quality, diverse data but also addresses specific needs in agricultural vision tasks that are not fully covered by general-purpose models.
M. Cieslak, U. Govindarajan, A. Garcia, A. Chandrashekar, T. Hädrich, A. Mendoza-Drosik, D. Michels S. Pirk, C.-C. Fu, W. Palubicki
Generating Diverse Agricultural Data for Vision-Based Farming Applications
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Vision for Agriculture, 2024
AI-Powered Growth Parameter Estimation of Plants
We introduce LAESI, a Synthetic Leaf Dataset of 100K synthetic leaf images on millimeter paper, each with semantic masks and surface area labels. This dataset provides a resource for leaf morphology analysis aimed at beech and oak leaves. We evaluate the applicability of the dataset by training machine learning models for leaf surface area prediction and semantic segmentation, using real images for validation. Our validation shows that these models can be trained to predict leaf surface area with a relative error not greater than an average human annotator. LAESI also provides an efficient framework based on 3D procedural models and generative AI for the large-scale, controllable generation of data with potential further applications in agriculture and biology. We evaluate the inclusion of generative AI in our procedural data generation pipeline and show how data filtering based on annotation consistency results in datasets allows training the highest performing vision models.
​J. Kaluzny, Y. Schreckenberg, K. Cyganik, P. Annighöfer, S. Pirk, D. L. Michels, M. Cieslak, F. Assaad, B. Benes, W. Palubicki
LAESI: Leaf Area Estimation with Synthetic Imagery
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Synthetic Data for Computer Vision, 2024
​
Paradigms for Designing Synthetic Data
The rise of artificial intelligence (AI) and in particular modern machine learning (ML) algorithms during the last decade has been met with great interest in the agricultural industry. While undisputedly powerful, their main drawback remains the need for sufficient and diverse training data. The collection of real datasets and their annotation are the main cost drivers of ML developments, and while promising results on synthetically generated training data have been shown, their generation is not without difficulties on their own. In this paper, we present a paradigm for the iterative, cost-efficient generation of synthetic training data. Its application is demonstrated by developing a low-cost early disease detector for tomato plants (Solanum lycopersicum) using synthetic training data. A neural classifier is trained by exclusively using synthetic images, whose generation process is iteratively refined to obtain optimal performance. In contrast to other approaches that rely on a human assessment of similarity between real and synthetic data, we instead introduce a structured, quantitative approach. Our evaluation shows superior generalization results when compared to using non-task-specific real training data and a higher cost efficiency of development compared to traditional synthetic training data
J. Klein, R. E. Waller, S. Pirk, W. Pałubicki, M. Tester, and D. L. Michels
Synthetic Data at Scale: A Paradigm to Efficiently Leverage Machine Learning in Agriculture
Frontiers in Plant Science, 2024
Synthetic-Data Enabled Robot-Plant-Interactions
Robotic harvesting has the potential to positively impact agricultural productivity, reduce costs, improve food quality, enhance sustainability, and to address labor shortage. In the rapidly advancing field of agricultural robotics, the necessity of training robots in a virtual environment has become essential. Generating training data to automatize the underlying computer vision tasks such as image segmentation, object detection and classification, also heavily relies on such virtual environments as synthetic data is often required to overcome the shortage and lack of variety of real data sets. However, physics engines commonly employed within the robotics community, such as ODE, Simbody, Bullet, and DART, primarily support motion and collision interaction of rigid bodies. This inherent limitation hinders experimentation and progress in handling non-rigid objects such as plants and crops. In this contribution, we present a plugin for the Gazebo simulation platform based on Cosserat rods to model plant motion. It enables the simulation of plants and their interaction with the environment. We demonstrate that, using our plugin, users can conduct harvesting simulations in Gazebo by simulating a robotic arm picking fruits and achieve results comparable to real-world experiments.
J. Deng, S. Marri, J. Klein, W. Palubicki, S. Pirk, G. Chowdhary, D. L. Michels
Gazebo Plants: Simulating Plant-Robot Interaction with Cosserat Rods
International Conference on Robotics and Automation Workshop - Robotics And Sustainability, 2024
Simulations of Weather
Due to the complex interplay of various meteorological phenomena, simulating weather is a challenging and open research problem. In this contribution, we propose a novel physics-based model that enables simulating weather at interactive rates. By considering atmosphere and pedosphere we can define the hydrologic cycle – and consequently weather – in unprecedented detail. Specifically, our model captures different warm and cold clouds, such as mammatus, hole-punch, multi-layer, and cumulonimbus clouds as well as their dynamic transitions. We also model different precipitation types, such as rain, snow, and graupel by introducing a comprehensive microphysics scheme. The Wegener-Bergeron-Findeisen process is incorporated into our Kessler-type microphysics formulation covering ice crystal growth occurring in mixed-phase clouds. Moreover, we model the water run-off from the ground surface, the infiltration into the soil, and its subsequent evaporation back to the atmosphere. We account for daily temperature changes, as well as heat transfer between pedosphere and atmosphere leading to a complex feedback loop. Our framework enables us to interactively explore various complex weather phenomena. Our results are assessed visually and validated by simulating weatherscapes for various setups covering different precipitation events and environments, by showcasing the hydrologic cycle, and by reproducing common effects such as Foehn winds. We also provide quantitative evaluations creating high-precipitation cumulonimbus clouds by prescribing atmospheric conditions based on infrared satellite observations. With our model we can generate dynamic 3D scenes of weatherscapes with high visual fidelity and even nowcast real weather conditions as simulations by streaming weather data into our framework.
J. A. Amador Herrera, T. Hädrich, W. PaÅ‚ubicki, D. T. Banuti, S. Pirk, D. L. Michels
Weatherscapes: Nowcasting Heat Transfer and Water Continuity
ACM Transactions on Graphics (SIGGRAPH Asia), 2021