I am trying to use urllib’s HTTPBasicAuthHandler to send user & password credentials to an endpoint. The real code runs in a Docker container entrypoint script and targets Wazuh running in another Docker container. For the purpose of this SO question, I’ve created a minimal example script and a test server with FastAPI to test the script against. It appears, judging from logs and the 401 response received, that urllib with a HTTPBasicAuthHandler registered does not send any Authorization header. Adding the header to the Request either on construction of the request or using the add_header method results in the Authorization header being correctly sent with the credentials. This is consistent with the behavior I see sending requests with urllib to the Dockerized Wazuh.

FastAPI test server, toy_server.py

import logging
from typing import Annotated
from fastapi import Depends, FastAPI, Request
from fastapi.security import HTTPBasic, HTTPBasicCredentials

app = FastAPI()

auth = HTTPBasic()

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

@app.get('/login')
def login(creds: Annotated[HTTPBasicCredentials, Depends(auth)], request: Request):
    logger.info('%s %s', creds.username, creds.password)
    logger.info(request.headers)
    return {
        'data': {
            'username': creds.username,
            'password': creds.password
        }
    }

I’ve run this either with FastAPI CLI on Python 3.12, fastapi run toy_server.py, or with Uvicorn on Python 3.9, uvicorn toy_server:app. This FastAPI server doesn’t care what the user & password are, just that they are included in an Authorization header of type Basic. I would expect for any request with Authorization Basic header that includes username and password I would get back a response containing those credentials, as well as see the headers logged on the server. For a request sent without Authorization Basic credentials, I would expect 401 back from the server, and nothing to be logged because FastAPI doesn’t pass the request on to the handler function.

Minimal working example constructing Request object with Authorization header. This also works with adding header after construction using add_header method.

working_example.py

import base64
import urllib.request

SCHEME = 'http'
HOST = 'localhost'
PORT = 8000
DOMAIN = f'{SCHEME}://{HOST}:{PORT}'
ROUTE = 'login'
URL = f'{DOMAIN}/{ROUTE}'
USERNAME = 'me'
PASSWORD = 'P@ssw0rd'

b64creds = base64.b64encode(f'{USERNAME}:{PASSWORD}'.encode()).decode()
# Also works to omit headers arg and do req.add_header('Authorization', f'Basic {b64creds}')
req = urllib.request.Request(URL, headers={
    'Authorization': f'Basic {b64creds}'
})

with urllib.request.urlopen(req) as res:
    print(res.read())

When I run this (I’ve tried Python 3.12 and 3.9) I get the expected response.

# python working_example.py
b'{"data":{"username":"me","password":"P@ssw0rd"}}'

And I can see from the server logs that the expected Authorization header was received.

...
INFO:toy_server:me P@ssw0rd
INFO:toy_server:Headers({'accept-encoding': 'identity', 'host': 'localhost:8000', 'user-agent': 'Python-urllib/3.9', 'authorization': 'Basic bWU6UEBzc3cwcmQ=', 'connection': 'close'})
...

I can also curl the server with creds, and get the expected response

curl -v -u hey:buddy http://localhost:8000/login
...
< HTTP/1.1 200 OK
...
{"data":{"username":"hey","password":"buddy"}}

Or without creds, and get the expected 401

# curl -v http://localhost:8000/login
...
< HTTP/1.1 401 Unauthorized
...
{"detail":"Not authenticated"}

Now I try to send the same request, but using urllib’s HTTPBasicAuthHandler. This example is taken from https://proxiesapi.com/articles/accessing-protected-resources-with-urllib-and-realm-authentication

broken_example.py

import urllib.request

SCHEME = 'http'
HOST = 'localhost'
PORT = 8000
DOMAIN = f'{SCHEME}://{HOST}:{PORT}'
ROUTE = 'login'
URL = f'{DOMAIN}/{ROUTE}'
USERNAME = 'me'
PASSWORD = 'P@ssw0rd'

password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, URL, USERNAME, PASSWORD)

handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
opener = urllib.request.build_opener(handler)

urllib.request.install_opener(opener)

with urllib.request.urlopen(URL) as res:
    print(res.read())

When I run this, it raises an exception.

Traceback (most recent call last):
  File "/Users/colinking-bailey/Projects/sparta/docker-wazuh/broken_example.py", line 20, in <module>
    with urllib.request.urlopen(URL) as res:
  File "/Users/colinking-bailey/.pyenv/versions/3.9.11/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/colinking-bailey/.pyenv/versions/3.9.11/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/Users/colinking-bailey/.pyenv/versions/3.9.11/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/Users/colinking-bailey/.pyenv/versions/3.9.11/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/Users/colinking-bailey/.pyenv/versions/3.9.11/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Users/colinking-bailey/.pyenv/versions/3.9.11/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

And no logs get printed. If I remove the auth Dependency, creds: Annotated[HTTPBasicCredentials, Depends(auth)], from the route (and replace the access of creds attributes in the return statement with placeholder strings), and send the request from broken_example.py again, then I can see from the logs that no Authorization header was sent for the broken_example request.

INFO:toy_server:Headers({'accept-encoding': 'identity', 'host': 'localhost:8000', 'user-agent': 'Python-urllib/3.9', 'connection': 'close'})

I have also tried the examples from the Python docs, https://docs.python.org/3.12/library/urllib.request.html#examples, and this SO answer, /a/47200746/6715241, with the same result of 401. This is all non-critical to me since I’ve got it working with adding the header directly to the Request object, but I’m truly surprised and perplexed that these minimal examples do not work for me.

I have tried various permutations of the broken example code, opening the url with opener.open(req), constructing the HTTPBasicAuthHandler without arguments and adding the url and creds to the created object with handler.add_password, and passing the url as a string instead of a Request object to both opener.open and urllib.request.urlopen. These all get 401 as well.

Am I missing something in my usage of HTTPBasicAuthHandler?