In this chapter, we will leverage databases in our scripts so that we can accomplish meaningful tasks when working with large quantities of data. Using a simple example, we will demonstrate the capabilities and benefits of using a database backend in our Python scripts. We will store file metadata that has been recursively indexed from a given root directory into a database and then query it to generate reports. Although this may seem like a simple feat, the purpose of this chapter is to showcase the ways we can interact with a database in Python by creating an active file listing.
In this chapter, we will delve into the following topics:
- The basic design and implementation of SQLite3 databases
- Working with these databases in Python using built-in and third-party modules
- Understanding how to recursively iterate through directories in Python
- Understanding filesystem metadata and the methods for accessing it using Python
- Crafting CSV and HTML reports for easy review by our end user
The code for this chapter was developed and tested using Python 2.7.15 and Python 3.7.1. The file_lister.py script was developed to work with Python 3.7.1. The file_lister_peewee.py script was developed and tested using both Python 2.7.15 and Python 3.7.1.