Case Study Overview & Agritech Challenge
This project was established to analyze, optimize, and build predictive machine learning models for agricultural yield using environmental sensor metrics (soil pH, moisture, humidity, temperature, sunlight) and remote satellite indexes (NDVI). In precision agriculture, predicting crop yield allows supply chains to adjust logistics and enables growers to tailor fertilizer/water treatments.
A rigorous statistical analysis of the dataset revealed that the features are generated independently of the crop yield target. As a result, all traditional regression algorithms (Linear Regression, Random Forest, XGBoost) yield negative out-of-sample R² scores (performing worse than predicting the target mean). Rather than masking this anomaly, this project presents an honest, scientifically sound audit, plotting overfitting curves that demonstrate how models learn noise, and proving the statistical independence of the variables mathematically.
Key Project Accomplishments:
- Rigorous Data Preprocessing: Constructed a robust ColumnTransformer pipeline to handle categorical one-hot encoding and numeric standardization safely, correcting errors present in reference notebooks.
- Statistical Signal Audit: Calculated Mutual Information regression scores and Pearson matrices to prove the independence of sensor features.
- Overfitting Visualization: Plotted training vs test curves at varying Random Forest depths to illustrate model complexity vs generalization.
- Deployment Architecture: Developed a complete microservice-based production deployment plan for edge inference.
Exploratory Data Analysis & Agronomic Significance
We audited the dataset distributions and compared crop yield levels across geographic regions, soil types, and operational management parameters. In agronomy, variables like soil pH directly affect the bioavailability of essential plant nutrients, while soil moisture governs microbial activity and root respiration.
Crop & Regional Performance
Farms are distributed fairly evenly across crop types (Wheat, Soybean, Cotton, Rice, Maize). Yield distributions by region (Central USA, East Africa, South India, North India, South USA) show identical medians (~4,000 kg/ha) and standard deviations (~1,150 kg/ha), indicating geographic independence.
Agronomic Strategy Impact
Audits of irrigation types (Drip, Sprinkler, Manual) and fertilizer types (Organic, Inorganic, Mixed) indicate that management choices do not drive changes in yield distribution, reinforcing the simulated independence of the variables.
Pearson correlation coefficients between numeric sensor columns and crop yield range from -0.076 to +0.041. Mutual Information regression scores computed using scikit-learn return values very close to 0.0 for all features, confirming the absence of linear or non-linear predictive signals.
Predictive Regressor Performance & Overfitting Audits
We trained regularized Linear Regressors (Ridge), Random Forest Regressors, and XGBoost Regressors. Below is the performance summary of the pipelines:
Overfitting Complexity Curves
The chart below shows model performance on training data (green) vs test data (amber) at varying depths. As depth increases, the Random Forest memorizes the random noise in the training split, achieving an R² of ~0.85. However, out-of-sample test R² remains flat and negative, demonstrating textbook overfitting.
Production Deployment Roadmap
A design exercise, included because the engineering is reusable even when the model is not. Nothing in this project should be served: all three regressors lose to predicting the mean, so there is no model here worth deploying. The steps below describe how the pipeline would be packaged if it had predictive value.
joblib to maintain state consistency between feature scaling and model
weights.
Digital Twin Crop Yield Estimator
A demonstration widget, not a forecasting tool. The offsets below are illustrative values in the spirit of the Ridge fit; they are not read from the serialised pipeline, and because the underlying features are statistically independent of yield, any such coefficients are noise. Change the inputs and watch the output barely move — that is the point.
Note: these Ridge coefficients are fitted to features that are statistically independent of the target, which means they are noise. Predictions converge on the global mean of ~4,000 kg/ha with small arbitrary adjustments (e.g. Soybean: +256, Moderate Disease: -84). The adjustments carry no agronomic meaning and should not be read as marginal effects.