Autonomous Operation of HBTracker

Publication date: 2025-03-06 | Source: PubMag

It's time to report on February, which turned out to be a productive month despite its brevity. The main focus was on process automation, database optimization, and increasing the volume of processed domains.

✅ Achievements in February

Fully automated the HBTracker process — the service now operates without manual intervention, ensuring continuous system operation even in my absence.
Improved website statistics display speed — optimized some complex queries, and work continues. A query caching system is being developed to reduce database load and speed up page loading.
Explored solutions to increase report collection speed — identified a database overload issue due to a high number of queries. No solution yet. Hosting support is unresponsive, so a local database solution or an intermediate buffer might be required.
Added the ability to register on the site — registered users will gain extended functionality. Additional features are currently in development.
Expanded the domain database to ±300k through search engine parsing — however, the current architecture may not handle further growth.
Launched work on publisher cataloging. No automation solution for manual work yet. The IAB 3.1 taxonomy is used for categorization.
Developed Battleboard — adapter monitoring has been ongoing for three months. Added a new adapter, DynoTech, bringing the total number of adapters to 61.

❌ Challenges in February

Failed to optimize the scanning algorithm — high database load and excessive traffic usage remain critical bottlenecks.
Automated site categorization based on collected search queries — still under development.
Developing a scalable cataloging architecture — automation is technically feasible but requires significant resources.
Addressing rapid database growth — the current database is already experiencing performance issues.
There is an issue with Cyrillic and mixed-language domains. No solution has been found yet.

🚀 Goals for March

Automate the semantic data collection process — this will improve the categorization system with an additional layer of information.
Create a domain search feature — to simplify site management.
Enhance the domain details page — improve the interface and functionality.
Enhance Luminex functionality — add search result parsing for Yandex and Google.
Further orchestration of scanning processes — optimize data processing.
Refactoring LibTracker — the library tracking module has not been updated since December. It needs a bot upgrade and expanded capabilities.
Developing product strategy and backlog — address issues from previous months.
Add new features for registered users — expand available data.

February marked a new level of progress through automation. However, I have hit some limitations again, requiring either significant changes or a focus on new functionality. March will undoubtedly involve a lot of optimization of existing features, and I will also try to introduce many new ones!