What happened during the January holidays? Work was carried out to improve scaling mechanisms and accelerate website crawling, along with some minor fixes.

This reduced the overall task execution time and increased system performance during parallel operations. The queue mechanism was also enhanced, allowing for more efficient workload distribution across system components.

I set an ambitious goal for myself — to crawl websites monthly to check for changes in their auction composition. This will enable timely tracking of adapter updates, ensuring data relevance, as well as collecting an archive for subsequent analysis. The first rechecks were conducted in December, during which the mechanism for launching multiple service instances and their interaction with each other and the database was debugged.

In January, error handling and recovery mechanisms were improved, enhancing system resilience. Memory optimization also helped reduce the overall load.

Currently, the service operates on a local server. This decision ensures rapid testing of changes and gradual scaling. I don’t plan to move it to the cloud in the near future as it would entail unjustified costs at this stage.

The global goal is to cover 8 million websites, nearly the entire Runet. To achieve this, plans include expanding system capabilities through increased computing power and implementing more efficient analysis algorithms. The first milestone is 200–250 thousand sites per month.

Since the service is at the MVP stage, I decided to purchase additional hardware for the home server to increase its speed. The new equipment will not only enhance performance but also lay the groundwork for future growth. An additional server will boost capacity.

Although there are still many unresolved issues, everything possible has been done at this stage, and the system is running stably. Now it’s time to switch to other tasks.

So, the tasks for January for HBTracker and LibTracker are:

• Continuing to optimize service operations in two directions:
  • Reducing traffic usage
  • Increasing performance
• Adding a scaling scheme to LibTracker, similar to HBTracker
• Adding dashboards for LibTracker on PubMag
• Creating a monthly report on adapter and library changes in table format. This will make analysis more visual and help identify key changes in the system more quickly.

These tasks aim to improve both the technical aspects of the project and the user experience. The implementation of gamification looks particularly promising, making tracking changes more visual and engaging.