flask-log-request-id is a small extension for Flask to forward a request ID from HTTP requests to logging. By default, it works with X-Request-ID, X-Correlation-ID, and AWS’s X-Amzn-Trace-Id. I recently wanted to propagate the request ID to a worker that does not use HTTP and thus does not receive any of the above headers.

My workflow was as follows: The application receives an HTTP request. Since this is a long-running request, it will not be processed immediately. Instead, it is put into a message queue. A worker process on another node fetches items from the queue and works on them. Both the API and the backend worker use logging, but they have different log targets. To be able to correlate logs from the API and the worker I wanted to write the same request ID into all logs.

Normally, flask-log-request-id should be initialized with a flask app and it will then fetch the request ID with a before_request handler before each HTTP request. The fetched request ID is stored to a location in flask’s g object. During logging the ID will be retrieved from g.

In my worker, there is no flask app to initialize and no HTTP request. However, we can quite easily write our own fetcher that does not load the value from g but instead from somewhere else. This fetcher can be registered with register_fetcher. flask-log-request-id will then iterate all fetchers and use the first available value for logging.

In my case this was as simple as creating a lambda function:

import flask_log_request_id
import logging
from typing import Optional

handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - level=%(levelname)s - request_id=%(request_id)s - %(message)s"))
handler.addFilter(flask_log_request_id.RequestIDLogFilter())
logging.getLogger().addHandler(handler)

# body comes from the message queue, I hard-coded it here to show its contents
body = {'request_id': 'my-id-123456', 'data': {'some': 'payload'}}

request_id: Optional[str] = body['request_id'] if 'request_id' in body else None
flask_log_request_id.request_id.current_request_id.register_fetcher(lambda: request_id)

logging.error('My test error')

For each log message flask-log-request-id will call the lambda and fetch the (fixed) request ID. If you run the script above, you will see:

2020-11-25 20:20:39,906 - root - level=ERROR - request_id=my-id-123456 - My test log

Comments

The comments section is a test at the moment. I might decide to remove it again. If you have any questions you can also send an e-mail to blog@stefan-koch.name. The software (Commento) is running on my own server.