Principles of polite crawling for collecting weather data from websites

Abstract

The issue of obtaining large data from publicly available weather forecast sites is under consideration. The advantages of using polite crawling when working with sites are shown. The proposed approach allows us to reduce the number of requests to sites. This work will also be useful for downloading data from other types of sites.