Happy New Year
Happy New Year, wishing everyone a year filled with joy, happiness and achieving everything they wish for.
Preface
This article is only for the purpose of translating and rewriting according to my own understanding, so if there is anything missing, please give me more suggestions.
If you’ve ever coded Python, chances are high that you’ve had to use quite a few print statements to see how your code runs, is it correct and ok or not, or to find out where the error is. But using print so much is extremely overwhelming so today I will show you how you can start using Log as an aka provip in python.
Logging is a very, very important thing in software development. It helps developers better understand program execution, unexpected errors and their reasons. Logging can store information such as the current state of the program or where the program is running. If an error occurs, developers can quickly find the line of code that is causing the problem and fix it. Finding these can be made easier using Python’s built-in logging tools.
Log Levels
It is a side note that the loggers as well as the underlying processing layers all use a so-called logging levels. This is a log hierarchy of how things are handled.
For example: When running the project locally when you use DEBUG level, it will receive debug commands and all levels on it. These levels are used to filter data by its importance. For example, when selecting the DEBUG log level, the system will log everything related to DEBUG , INFO , WARNING , ERROR , CRITICAL .
| Level | Value |
|---|---|
| CRITICAL | 50 |
| ERROR | 40 |
| WARNING | 30 |
| INFO | 20 |
| DEBUG | ten |
| NOTSET | ten |
Logger
Logger is simply a named instance of the logging object. All loggers are “global” in the program. Eg.
1 2 3 4 | # logging/app.py import logging logger = logging.getLogger("app") |
The above code defined a logger named app in app.py .
Now if I want, I can import into another file and have access to the same logger instance I created earlier. It will look like this:
1 2 3 4 | # logging/common.py import logging logger = logging.getLogger("app") |
Now common.py has access to the same logger instance . This means I can declare all the loggers I want before I need them.
Note: the best practice is to use __name__ to name the logger like this: logging.getLogger(__name__) . This way, each module has its own logger name.
Logger ‘s core job is to send LogRecords to different handlers. LogRecords records details of what happened in the system at runtime.
1 2 3 | logger.info("Emitting Info LogRecord") logger.error("Emitting Error LogRecord") |
The above is an example of what it means to fire an event. Events are fired with an associated level . This allows to know the urgency of the LogRecord (Log Level). The problem with the above code is that the logger is completely unaware of the destination of these logs such as console or file.
Handlers
Handlers control where LogRecords return. Handlers are independent objects that can be attached to logger instances.
1 2 3 4 | logger = logging.getLogger(__name__) fileHandle = logging.FileHandler('logging.log') logger.addHandler(fileHandle) |
The above code tells the logger that, whenever a LogRecord is created, it will be sent to the file logging.log . The problem right now is that logging.log will be crammed with a lot of logs.
Setting the log level that Handlers have to manage helps me filter my logs by level.
1 2 3 4 5 6 7 | logger = logging.getLogger(__name__) fileHandle = logging.FileHandler('logging.log') fileHandle.setLevel(logging.WARNING) logger.addHandler(fileHandle) logger.info("Nó sẽ không hiển thị logging.log") logger.error("Nó sẽ hiển thị") |
Above is how to filter the log using log level . As in the log levels , ERROR is higher than WARNING so it will be processed and INFO is lower than WARNING so it will be discarded.
There are several types of handlers that can be used for example StreamHandler, FileHandler . For a complete list, see the handlers document in python: handlers
StreamHandler will send output to console. FileHandler sends information to an output file. The most dynamic handlers are QueueHandler and QueueListener , but we’ll talk about that later.
Formatters
Assuming you only have a message saying: “An error has occurred”, this message is almost meaningless. So formatters were born to solve this problem. Formatters exist on Handlers when Handlers process log records.
You can easily add some additional information such as where, what, why, when and how.
1 2 3 4 5 6 7 8 9 10 11 12 13 | import logging logger = logging.getLogger(__name__) fileHandle = logging.FileHandler('app.log') fileHandle.setLevel(20) formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') fileHandle.setFormatter(formatter) logger.addHandler(fileHandle) logger.warning("Warning Message") logger.error("Error Message") >>> 2023-01-24 22:31:43,182 - __main__ - WARNING - Warning Message >>> 2023-01-24 22:31:43,182 - __main__ - ERROR - Error Message |
The above is the output in the generated console.
Formatters in the above example have been inserted with the time, logger name, record level and message.
QueueHandler and QueueListener
The problem is that whenever the program runs to the code logger.info(“message”) , your program will be blocked until it finishes processing the logging. It won’t be a big deal if you just output the log to the console but when it’s running fine and the server crashes, the log will sing “yes, don’t keep it, don’t look”.
If you send the log to the database, waiting for the record to be created may take some time, your program will be blocked every time the log record is created. QueueHandler and QueueListener will solve this problem. QueueHandler places the message in a queue that exists on a separate Thread . Then the QueueListener sends the Log Record to all other handler . This way your program is not blocked waiting for the Log Record to be created. To learn more about using QueueHandler and QueueListener, you can see more at this link
Using File to initialize loggers
The last thing to do is create a File to launch all the loggers. This will help avoid code duplication, easy management and changes, saving time and effort. You can completely create and manage all loggers in a single place. There are many ways to store the loggers configuration for the project such as using dictionary , JSON or yaml file to initialize the loggers. Here is an example with yaml file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | # Logger Config.yaml version: 1 objects: queue: class: queue.Queue maxsize: 1000 formatters: simple: format: '%(asctime)s - %(levelname)s - %(name)s - %(message)s' handlers: discord: class: utilities.log_utils.logger_util.DiscordHandler level: ERROR formatter: simple queue_listener: class: utilities.log_utils.logger_util.QueueListenerHandler handlers: - cfg://handlers.console - cfg://handlers.file - cfg://handlers.discord queue: cfg://objects.queue loggers: __main__: level: WARNING handlers: - queue_listener propagate: false ieddit: level: WARNING handlers: - queue_listener propagate: false |
In the end you just need to write a file to read the config from a yaml file, dictionary or JSON to initialize loggers and then just import and run it once in the project. Examples are as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 | # logger_init.py import config import yaml import logging with open('logger_config.yaml', 'r') as stream: try: logging_config = yaml.load(stream, Loader=yaml.SafeLoader) except yaml.YAMLError as exc: print("Error Loading Logger Config") pass logging.config.dictConfig(logging_config) |
People can refer to the sample code on using logging with python at the following link: GITHUB
tags: Basic Python, Python, Programming, Logging, Loggers, FileHandlers, Handlers, Formatters, QueueHandler , QueueListener, Log Levels
Source: Mediummmm