download

Get a remote file, checking the HTTP response code, if it has changed since the last time a download was tried.

If the ETag does not match, it could be caused by the previous download of the same file coming from a mirror running on a different webserver, e.g. Apache vs Nginx. Content-Length and Last-Modified are used to check whether the file has changed since those are more standardized than ETag. Plus, Nginx and Apache 2.4 defaults use only those two values to generate the ETag anyway. Unfortunately, other webservers and CDNs have totally different methods for generating the ETag. And mirrors that are syncing using a method other than rsync could easily have different Last-Modified times on the exact same file. On top of that, some services like GitHub's raw file support raw.githubusercontent.com and GitLab's raw file support do not set the Last-Modified header at all. So ultimately, then ETag needs to be used first and foremost, then this calculated ETag can serve as a common fallback.

In order to prevent the ETag from being used as a form of tracking cookie, this code never sends the ETag to the server. Instead, it uses a HEAD request to get the ETag from the server, then only issues a GET if the ETag has changed.

This uses a integer value for Last-Modified to avoid enabling the use of that value as some kind of "cookieless cookie". One second time resolution should be plenty since these files change more on the time space of minutes or hours.

See also