Abstract
This study investigates whether historical search and engagement metrics can be used to identify content pages that are likely to experience declining search performance. The analysis was conducted using the anonymized FlyRank ML Internship dataset containing approximately 30,000 content records. Logistic Regression and Random Forest models were evaluated under both random and grouped validation strategies, with grouped validation providing a more realistic estimate of performance on previously unseen clients. The Random Forest model achieved the strongest measured performance while careful validation and leakage checks ensured honest evaluation. The resulting ranked recommendations are intended to support editorial decision-making and should be used as decision-support rather than automated publishing instructions.
Table of Contents
1. Introduction
Search performance changes continuously because of evolving user behavior, search engine updates and content freshness.As the number of published pages grows, manually identifying content that requires attention becomes increasingly difficult for editorial teams.
The objective of this research is to investigate whether historical search and engagement metrics can help identify pages that are more likely to experience declining search performance. Rather than replacing editorial judgment, the proposed model is intended to support content teams by providing a ranked list of pages that may benefit from manual review.
This work was completed as part of the FlyRank ML Internship using an anonymized production dataset. Throughout the study, careful validation and leakage checks were performed to ensure that the reported results represent realistic model performance.
This research addresses the FlyRank content refresh problem by helping content teams identify pages that may require manual review before search performance declines further. Rather than replacing editorial decision-making, the model provides ranked recommendations based on historical search and engagement signals to support content prioritization.
2. Data
This research was conducted using the anonymized FlyRank ML Internship dataset built from large-scale production search and engagement data.
The analysis was performed on approximately 30,000 content records. The dataset contains historical search performance, engagement metrics, content characteristics and content freshness indicators.
Features used in this study include search impressions, clicks, pageviews, sessions, average search position, click-through rate (CTR), engagement rate, content age and freshness-related metrics.
The target label was derived from the observed trend_direction field. Throughout this work, private client names, URLs, search queries and other sensitive information were excluded to maintain public-safe reporting.
3. Methodology
The objective of this study was to identify content pages that are more likely to experience declining search performance using historical search and engagement metrics.
Target Label
The target variable was created from the observed trend_direction field, where pages labeled "down" were treated as the positive class.
Features
The models were trained using historical search performance, engagement metrics, content characteristics and freshness indicators, including impressions, clicks, pageviews, sessions, content age, average position, click-through rate (CTR), engagement rate and freshness-related metrics.
Models Evaluated
| Model | Purpose |
|---|---|
| Logistic Regression | Baseline machine learning model |
| Random Forest Classifier | Final model used for evaluation |
Validation Strategy
Two validation strategies were evaluated. First, a random train-test split was used as an initial baseline. The final evaluation used a grouped validation split based on client_id, ensuring that pages from the same client did not appear in both training and testing datasets. This provides a more realistic estimate of model performance on previously unseen clients.
Leakage Checks
Potential data leakage was carefully reviewed before model training. The trend_direction field was used only as the target label, trend_pct was excluded from the feature set, and no product-generated decision outputs were included as model inputs.
4. Results
Two machine learning models were evaluated for this study: Logistic Regression and Random Forest. Performance was measured using Accuracy, Precision, Recall and F1 Score.
| Validation | Model | Accuracy | Precision | Recall | F1 Score |
|---|---|---|---|---|---|
| Random Split | Random Forest | 0.9098 | 0.8981 | 0.9403 | 0.9187 |
| Grouped Split | Random Forest | 0.8861 | 0.8766 | 0.9044 | 0.8903 |
The Random Forest model achieved the highest measured performance among the evaluated models. Performance decreased slightly when using grouped validation, indicating a more realistic estimate of generalization to previously unseen clients. These results support the use of grouped validation when evaluating content-ranking models.
5. Limitations
This study is based on an anonymized historical dataset and should be interpreted within that context. The models were evaluated using historical search and engagement metrics and cannot account for future search engine algorithm updates, changes in user behavior or new content strategies.
The results demonstrate measured performance on this dataset under grouped validation and should not be interpreted as proof that the same performance will generalize to every website or content portfolio.
The ranked recommendations are intended to support human decision-making and should not replace editorial expertise or business judgment.
6. Ranked Recommendations
Based on the validated model and the action playbook developed during this study, the following recommendations are proposed for content teams. These recommendations are intended to support manual decision-making rather than automate editorial actions.
| Priority | Reason Code | Recommended Action |
|---|---|---|
| High | HIGH_VISIBILITY | Review and refresh high-traffic content to maintain search performance. |
| Medium | STALE_CONTENT | Update outdated content and verify information accuracy. |
| Medium | LOW_CTR | Improve page titles and meta descriptions to increase click-through rate. |
| Low | MODERATE_PRIORITY | Continue monitoring performance before taking further action. |
These recommendations were derived from historical search and engagement patterns observed in the dataset. They should be interpreted as decision-support guidance and not as automatic publishing or content management instructions.
7. Reproducibility
This research was developed as part of the FlyRank ML Internship. All experiments, preprocessing steps, validation procedures and action playbooks are documented in the accompanying notebooks available in the project repository.
Week 5: Machine Learning Model Development
Week 6: Validation and Leakage Audit
Week 7: Content Action Playbook
Capstone: Research Paper
The complete source code, notebooks and supporting files are available in the GitHub repository, allowing the workflow to be reviewed and reproduced using the publicly provided project structure.
8. Acknowledgments
This project was completed as part of the FlyRank ML Internship. The internship provided the opportunity to work with a large-scale anonymized dataset and apply machine learning techniques to a real-world content analytics problem.
The research, analysis, validation and action playbook presented in this paper were developed using the resources and datasets provided during the internship while following responsible machine learning practices and public-safe reporting guidelines.
Data Credit: Built on the FlyRank ML Internship Dataset.
Learn more about FlyRank: https://flyrank.ai