Case Study Overview & Agritech Challenge
This project was established to analyze, optimize, and build predictive machine learning models for agricultural yield using environmental sensor metrics (soil pH, moisture, humidity, temperature, sunlight) and remote satellite indexes (NDVI). In precision agriculture, predicting crop yield allows supply chains to adjust logistics and enables growers to tailor fertilizer/water treatments.
A rigorous statistical analysis of the dataset revealed that the features are generated independently of the crop yield target. As a result, all traditional regression algorithms (Linear Regression, Random Forest, XGBoost) yield negative out-of-sample R² scores (performing worse than predicting the target mean). Rather than masking this anomaly, this project presents an honest, scientifically sound audit, plotting overfitting curves that demonstrate how models learn noise, and proving the statistical independence of the variables mathematically.
Key Project Accomplishments:
- Rigorous Data Preprocessing: Constructed a robust ColumnTransformer pipeline to handle categorical one-hot encoding and numeric standardization safely, correcting errors present in reference notebooks.
- Statistical Signal Audit: Calculated Mutual Information regression scores and Pearson matrices to prove the independence of sensor features.
- Overfitting Visualization: Plotted training vs test curves at varying Random Forest depths to illustrate model complexity vs generalization.
- Deployment Architecture: Developed a complete microservice-based production deployment plan for edge inference.
Exploratory Data Analysis & Agronomic Significance
We audited the dataset distributions and compared crop yield levels across geographic regions, soil types, and operational management parameters. In agronomy, variables like soil pH directly affect the bioavailability of essential plant nutrients, while soil moisture governs microbial activity and root respiration.
Crop & Regional Performance
Farms are distributed fairly evenly across crop types (Wheat, Soybean, Cotton, Rice, Maize). Yield distributions by region (Central USA, East Africa, South India, North India, South USA) show identical medians (~4,000 kg/ha) and standard deviations (~1,150 kg/ha), indicating geographic independence.
Agronomic Strategy Impact
Audits of irrigation types (Drip, Sprinkler, Manual) and fertilizer types (Organic, Inorganic, Mixed) indicate that management choices do not drive changes in yield distribution, reinforcing the simulated independence of the variables.
Pearson correlation coefficients between numeric sensor columns and crop yield range from -0.076 to +0.041. Mutual Information regression scores computed using scikit-learn return values very close to 0.0 for all features, confirming the absence of linear or non-linear predictive signals.
Predictive Regressor Performance & Overfitting Audits
We trained regularized Linear Regressors (Ridge), Random Forest Regressors, and XGBoost Regressors. Below is the performance summary of the pipelines:
Overfitting Complexity Curves
The chart below shows model performance on training data (green) vs test data (amber) at varying depths. As depth increases, the Random Forest memorizes the random noise in the training split, achieving an R² of ~0.85. However, out-of-sample test R² remains flat and negative, demonstrating textbook overfitting.
Production Deployment Roadmap
In a production environment, models must be packaged, served, and monitored reliably. Below is our proposed architecture to serve the XGBoost regressor pipeline:
joblib to maintain state consistency between feature scaling and model
weights.
Digital Twin Crop Yield Estimator
Enter environmental parameters below. The estimator implements the exact coefficients learned by our Ridge regressor model pipeline.
Note: Due to variable independence in this simulated dataset, predictions converge around the global mean of ~4,000 kg/ha, adjusted by the marginal coefficients (e.g., Soybean: +256, Moderate Disease: -84) learned by the Ridge regressor model pipeline.