If the Python code seems perfectly fine and the memory still increases when you loop through the isolated function, the leak might be located on the C side. This happens, for instance, when a Py_DECREF macro is missing in the critical part of some imported C extension.
The C code of CPython interpreter is pretty robust and tested for the existence of memory leaks, so it is the last place to look for memory problems. But if you use packages that have custom C extensions, they might be a good place to look first. Because you will be dealing with code operating on a much lower level of abstraction than Python, you need to use completely different tools to resolve such memory issues.
Memory debugging is not easy in C, so before diving into extension internals, make sure that you properly diagnose the source of your problem. It is a very popular approach to isolate a suspicious package with code similar in nature to unit tests. To diagnose the source of your problem, you should consider the following actions:
- Write a separate test for each API unit or functionality of an extension you are suspecting to leak memory
- Perform the test in a loop for an arbitrarily long time in isolation (one test function per run)
- Observe from outside which of the tested functionalities increases memory usage over time
By using such an approach, you will eventually isolate the faulty part of the extension and this will reduce the time required later to inspect and fix its code. This process may seem burdensome because it requires a lot of additional time and coding, but it really pays off in the long run. You can always ease your work by reusing some of the testing tools that were introduced in Chapter 12, Test-Driven Development. Utilities such as pytest and tox were perhaps not designed exactly for this case, but can at least reduce the time required to run multiple tests in isolated environments.
If you have successfully isolated the part of the extension that is leaking memory, you can finally start actual debugging. If you're lucky, a simple manual inspection of the isolated source code section may give the desired results. In many cases, the problem is as simple as adding the missing Py_DECREF call. Nevertheless, in most cases, your work won't be that simple. In such situations, you need to bring out some bigger guns. One of the notable generic tools for fighting memory leaks in compiled code that should be in every programmer's toolbelt is Valgrind. It is a whole instrumentation framework for building dynamic analysis tools. Because of this, it may not be easy to learn and master, but you should definitely acquaint yourself with the basics of its usage.
Profiling network usage is explained in the next section.