Understanding HTTP State Management

HTTP state management is covered in RFC6265.  

Python's request module however refers to its predecessor RFC2965 within the class DefaultCookiePolicy in cookiejar.py. Some programmers say the implementation is quite strict.

One difference in the RFCs is the use of Augmented Backus-Naur form (ABNF) in the latter, not used in the former. ABNF for specifications is formalised in RFC5234 (an Internet standard, proposed to be augmented by RFC7405).

Understanding HTTP Redirects

Introduction

If you want to program an HTTP request module (to make HTTP requests, to download web pages, or other data) there are a number of relevant concepts relating to HTTP that you must understand.

One of these concepts is HTTP redirects. What are they, why do they exist, and what a "reasonable" number of redirects looks like when sending an HTTP request to a server.

What is a redirect - technically speaking?

First the technical part. HTTP redirects occur when a server responds to a client request with the reply 3XX (status code). This is coupled with a location header of the URL it wants the client to load instead.

The browser or other HTTP client (which could be a Python program, for example) then follows the redirect, issuing a new request to the URL in the location header.

Why are they needed?

Examples:

301 Permanent redirect - e.g. for site reorganization, domain migration, SEO link preservation

Also temporary redirects - e.g. for load balancing

HTTP to HTTPS redirect

Redirects can be configure in Apache .htaccess and in nginx.

Pythons requests module allows by default 30 redirects.

Revisiting Port 443

Port 443 is primarily known for handling HTTPS traffic over TCP, enabling secure communication between browsers and servers.  

It can be used with UDP in specific scenarios like QUIC (for lower latency secure communications) and HTTP/3.

When diagnosing why an HTTPS connection is failing, an in depth knowledge of port 443 is needed.

RDAP is the new whois

You may see the message on websites "Use of the RDAP service is limited to lawful business purposes only". 

RDAP is the Registration Data Access Protocol developed by the IETF as the successor to whois.

Key difference: 

whois returns free text, RDAP returns JSON, making it machine readable and easier to automate. It also supports RESTful web services, allowing for HTTP based queries, error codes, authentication and access control.

RDAP also supports Internationalized Domain Names (IDNs), which are domain names utilizing non-Latin characters. Languages can include Arabic, Chinese, Cyrillic or Devanagari. As DNS is limited to ASCII characters, an ASCII encoding called Punycode (deliberately designed to rhyme with Unicode) is used for name translation.

All that said, whois is still probably more frequently used than RDAP.

The HTTP 400 Error

The HTTP 400 error translates to Bad Request. It can sometimes be seen when attempting a logon to a website unsuccessfully.

In short, the server cannot process your request - potentially due to malformed URLs, corrupted cookies or outdated DNS data.

You can try flushing the DNS cache as well:

ipconfig /flushdns

which should yield the output "Successfully flushed the DNS Resolver Cache" if successful.

Debugging Web Access Issues with Microsoft Edge

Edge comes with Developer Tools (Control-Shift-I).  

These tools are surprisingly powerful. The tools appear right next to the rendered webpage in the browser.

Suppose you are trying to log in to a service called "Microsoft New Service" but it doesn't work.  The tab text says "Sign in to Microsoft New Service". Open up Dev Tools. Click on Network (the wifi-style icon). 

Now reload the webpage.  

You will see entire flow of HTTP requests and responses. A 200 response is status code OK, a 204 means No Content. In fact, any status code between 200 and 299 is a form of success. 300-399 are redirection messages, and anything about that represents an error.

You may be surprised by the number of conditional access-related HTTP requests that are involved in an authentication attempt.

OHTTP

OHTTP (Oblivious HTTP) is an IETF network protocol to enable anonymous HTTP transactions over the Internet. 

Its primary goal is to enable users (browsers, agents, other software) to send HTTP requests without revealing their IP address.

It is defined by RFC9458.

Introducing the Azure SRE Agent

The new Azure SRE agent (announced May 2025) and demonstrated at MS Build, is designed to make it easier to "sustain production environments". This includes taking toil away from checking log files, analyzing historical changes and augmenting this with LLMs. Incident and infrastructure management is set to be transformed, with the Azure SRE agent able to partner in incident investigation and root cause analysis. An example prompt may be: "visualize HTTP request and 500 errors for last week for my app".

WinJoe's Guide to (E)nginx

nginx ("Engine-X") is an HTTP and reverse proxy server (used by Dropbox, Netflix and Zynga).

What is a reverse proxy server I hear you cry?

A reverse proxy typically sits behind the firewall and routes requests to internal servers (thereby potentially acting as "traffic cop" to external traffic).

Unsurprisingly, load balancing is one of its main applications. They can also reduce network traffic by compressing inbound and outbound data (this is sometimes remarked very untechnically in marketing as "web acceleration") as well as caching frequently requested content.

The RP can also add to your security defence.

nginx is written by Igor Sysoev.

It can also do generic proxying of TCP and UDP requests.

Debugging 401 Errors in Internet Information Services

HTTP, an application level protocol for distributed hypermedia systems, came into existence in 1990 with HTTP/0.9, a simple protocol for raw data transfer over the Internet.

With HTTP 1.x came a built-in mechanism for authentication, fully documented in RFC 2617. HTTP 1.1 supports two modes of bulit-in authentication "Basic" and "Digest". Basic authentication transmits the username/password pair in unencrypted form from browser to server. It should only be used over an encrypted medium e.g. SSL (Secure Sockets Layer recently renamed by the IETF to TLS, or Transport Layer Security). Digest authentication sends a one-way hash of the username/password pair, calculated with a time-sensitive salt value from the server. The purpose of the salt value is that the username-password hash is always unique, and protected from replay attacks. Note though that only the authentication details are protected by the hash and any data sent following successful authentication will be visible to any party with access to the network traffic. To overcome this, an encrypted stream is needed.

Attempting to retrieve a page using HTTP authentication can result in an HTTP 401 (Unauthorized) error (some other interesting error codes are listed here, including the currently unused 402 error - Payment Required). The response from the server will include a "WWW-authenticate" header of the form (WWW-Authenticate: AUTH_SCHEME realm="REALM_NAME"). A valid value for AUTH_SCHEME would be Basic, to denote basic authentication. The "realm" is the section of the website protected by the authentication scheme. Since HTTP is stateless, servers don't remember who is logged in and clients must send correct headers for each protected page they wish to access.

IIS provides substatus codes to analyse the underlying causes of 401 errors. The codes are: 1) invalid credentials, 2) wrong authentication method, 3) ACL issue, 4) ISAPI (Internet Server API)/CGI issue, and, 5) access denied by URL authorization policy on server.