Understanding HTTP Redirects

Introduction

If you want to program an HTTP request module (to make HTTP requests, to download web pages, or other data) there are a number of relevant concepts relating to HTTP that you must understand.

One of these concepts is HTTP redirects. What are they, why do they exist, and what a "reasonable" number of redirects looks like when sending an HTTP request to a server.

What is a redirect - technically speaking?

First the technical part. HTTP redirects occur when a server responds to a client request with the reply 3XX (status code). This is coupled with a location header of the URL it wants the client to load instead.

The browser or other HTTP client (which could be a Python program, for example) then follows the redirect, issuing a new request to the URL in the location header.

Why are they needed?

Examples:

301 Permanent redirect - e.g. for site reorganization, domain migration, SEO link preservation

Also temporary redirects - e.g. for load balancing

HTTP to HTTPS redirect

Redirects can be configure in Apache .htaccess and in nginx.

Pythons requests module allows by default 30 redirects.