Overview

MetaMAL is composed of two main and interconnected projects: - Crawler: it collects and analyzes statistical data taken from MyAnimeList; - Website: it converts the data gathered by the crawler into a user-friendly format and makes it accessible through the web.

This repository contains the code for a web crawler which extracts some data from MyAnimeList and produces statistics that can be feed to the Website.

Technical details

The Crawler is written in Python 3 (3.6+) and is supported by a few pip packages. It can be run by executing the Crawler.py file.

The data collection process is executed every day at midnight UTC and requires about 20 minutes to complete with a good internet connection.

The application makes use of a single thread for two main reasons: - because the server might return an HTTP 429 Error: Too many requests in case two or more requests are sent almost at the same time; - because too many new connections may cause the crawler's IP address to be blacklisted.

The following statistics are collected daily for each seasonal anime: - score; - members; - genres; - characters' favourites counter; - seiyuu favourites counter; - IDs, timestamps and other minor fields.

At the end of each data collection cycle, five more operations are executed: - the data is stored in a SQLite database which can be found in the location pointed by the DATABASE_PATH variable; - refined statistics are generated for being consumed by the Website (see below); - a spreadsheet containing the seasonal statistics is generated; - the pictures for the most popular characters and seiyuu are downloaded and stored locally so that the Website can show them in the seasonal page; - the SQLite database is backed up in a cloud archive.

With “refined statistics” I refer to three different JSON files which are used by the Website to correctly render the template for the seasonal pages. In particular: - the seasons.json file contains info about the start and end dates for each analyzed season (example); - the [year]-[season].json files hold the actual data used to draw the diagrams and charts on the seasonal webpage (example); - the [year]-[season]-full.json files have the same internal structure as their [year]-[season].json counterpart, but the formers contain more data.

Copies of the SQLite database containing the raw data collected by the Crawler can be found here.

Python3 dependencies

The following pip3 packages are required to execute the programme: - boto3 - lxml - openpyxl - requests