Store title flavour download statistics #560

This is a subissue / pre-requisite of #449

CMS will store for each title flavour the number of downloads done per day.

It will source this data from our matomo instance at https://stats.kiwix.org/

Goal is to fetch website 24 (lb.download.kiwix.org) download statistics: https://stats.kiwix.org/index.php?module=CoreHome&action=index&date=yesterday&period=day&idSite=24#?idSite=24&period=day&date=yesterday&category=General_Actions&subcategory=General_Downloads

Proper API way to do this is to use this kind of query and paginate with filter_limit:

curl -X POST "https://stats.kiwix.org/index.php" \
  -d "module=API" \
  -d "method=Live.getLastVisitsDetails" \
  -d "idSite=24" \
  -d "period=day" \
  -d "date=yesterday" \
  -d "segment=actionType==downloads" \
  -d "filter_limit=100" \
  -d "filter_offset=0" \
  -d "format=json"

Only four kind extensions should be taken into account: .zim, .zim.torrent, .zim.magnet and .zim.meta4. Sum of all these downloads will be saved in DB.

Based on URL in these stats, it will find the matching title flavour in CMS DB. A given title flavour can theoretically receive downloads from two distinct paths when associated with two collections which are both served by lb.download.kiwix.org, this is expected and sum of download count must be saved. We do not store downloads at book level because this is not relevant from a stats PoV.

The CMS will run a background job (in the mill) to save these statistics. It will fetch every missing days in its DB, from 365 days ago up to yesterday. On a given day, yesterday statistics can only be fetched after a given time (4am UTC by default). Previous days can always be fetched. CMS will store in DB the list of days it already fetched up to completion, in order to know which data still has to be fetched. When fetching a given day, all data existing in DB is replaced. Admins might manually remove a day from the list of days already fetched to force stats re-download.

I propose we store these statistics in a new DB table with these columns: title_id, flavour, year, lb_downloads, where xxx_downloads is an integer[], i.e. a list of 365 (or 366) values. The list must hence contains sufficient null or 0 values (not sure which one is correct in PG terms) to feed a given day. I'm not certain this is the best schema, but it looks like it allows to limit the number of DB rows (avoiding foreign key duplication on each day and making foreign key maintenance a nightmare for the DB) all while giving sufficient performance capabilities. It also provide extensibility with the single lb_downloads column which might evolve into multiple columns if needed. This schema might of course evolve in the future once we have real data, more needs and better feedback.

CMS will automatically purge data older than previous year: e.g. if purge runs today, it will purge data from 2024 and before, keeping the whole 2025 year + ongoing 2026. This is meant to ensure we always have one full year available and simplify purge logic. I do expect not all this data will be used for now, but this seems to be a balanced decision.