Learning Python · 5th Edition by Lutz, Mark -- Read -- Imperial Library of Trantor

Index

Learning Python

Mark Lutz

Dedication Special Upgrade Offer Preface This Book’s “Ecosystem” About This Fifth Edition The Python 2.X and 3.X Lines

The 2.X/3.X Story Today Coverage for Both 3.X and 2.X Which Python Should I Use?

Note

This Book’s Prerequisites and Effort This Book’s Structure

Note

What This Book Is Not

It’s Not a Reference or a Guide to Specific Applications It’s Not the Short Story for People in a Hurry It’s as Linear as Python Allows

Note

This Book’s Programs

Python Versions Platforms Fetching This Book’s Code Using This Book’s Code

Font Conventions

Note Warning

Book Updates and Resources Acknowledgments

The Backstory Python Thanks Personal Thanks

Part I. Getting Started Chapter 1. A Python Q&A Session Why Do People Use Python?

Software Quality Developer Productivity

Is Python a “Scripting Language”? OK, but What’s the Downside? Who Uses Python Today? What Can I Do with Python?

Systems Programming GUIs Internet Scripting Component Integration Database Programming Rapid Prototyping Numeric and Scientific Programming And More: Gaming, Images, Data Mining, Robots, Excel...

How Is Python Developed and Supported?

Open Source Tradeoffs

What Are Python’s Technical Strengths?

It’s Object-Oriented and Functional It’s Free It’s Portable It’s Powerful It’s Mixable It’s Relatively Easy to Use It’s Relatively Easy to Learn It’s Named After Monty Python

How Does Python Stack Up to Language X? Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 2. How Python Runs Programs Introducing the Python Interpreter Program Execution

The Programmer’s View Python’s View

Byte code compilation The Python Virtual Machine (PVM) Performance implications Development implications

Execution Model Variations

Python Implementation Alternatives

CPython: The standard Jython: Python for Java IronPython: Python for .NET Stackless: Python for concurrency PyPy: Python for speed Note

Execution Optimization Tools

Cython: A Python/C hybrid Shed Skin: A Python-to-C++ translator Psyco: The original just-in-time compiler

Frozen Binaries Future Possibilities?

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 3. How You Run Programs The Interactive Prompt

Starting an Interactive Session The System Path New Windows Options in 3.3: PATH, Launcher Where to Run: Code Directories What Not to Type: Prompts and Comments

Note

Running Code Interactively Why the Interactive Prompt?

Experimenting Testing

Usage Notes: The Interactive Prompt

Entering multiline statements

System Command Lines and Files

A First Script Running Files with Command Lines Command-Line Usage Variations Usage Notes: Command Lines and Files

Unix-Style Executable Scripts: #!

Unix Script Basics The Unix env Lookup Trick The Python 3.3 Windows Launcher: #! Comes to Windows

Clicking File Icons

Icon-Click Basics Clicking Icons on Windows The input Trick on Windows

Note

Other Icon-Click Limitations

Module Imports and Reloads

Import and Reload Basics

Note

The Grander Module Story: Attributes

Modules and namespaces Note

Usage Notes: import and reload

Using exec to Run Module Files

Note

The IDLE User Interface

IDLE Startup Details IDLE Basic Usage IDLE Usability Features Advanced IDLE Tools Usage Notes: IDLE

Other IDEs Other Launch Options

Embedding Calls Frozen Binary Executables Text Editor Launch Options Still Other Launch Options Future Possibilities?

Which Option Should I Use? Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part I Exercises

Warning

Part II. Types and Operations Chapter 4. Introducing Python Object Types The Python Conceptual Hierarchy

Note

Why Use Built-in Types? Python’s Core Data Types Numbers Strings

Sequence Operations Immutability Type-Specific Methods Getting Help Other Ways to Code Strings Unicode Strings Pattern Matching

Lists

Sequence Operations Type-Specific Operations Bounds Checking Nesting Comprehensions

Dictionaries

Mapping Operations Nesting Revisited Missing Keys: if Tests Sorting Keys: for Loops Iteration and Optimization

Tuples

Why Tuples?

Files

Binary Bytes Files Unicode Text Files Other File-Like Tools

Other Core Types

How to Break Your Code’s Flexibility User-Defined Classes And Everything Else

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 5. Numeric Types Numeric Type Basics

Numeric Literals Built-in Numeric Tools Python Expression Operators

Mixed operators follow operator precedence Parentheses group subexpressions Mixed types are converted up Note Preview: Operator overloading and polymorphism

Numbers in Action

Variables and Basic Expressions Numeric Display Formats Comparisons: Normal and Chained Division: Classic, Floor, and True

Supporting either Python Floor versus truncation Why does truncation matter?

Integer Precision Complex Numbers Hex, Octal, Binary: Literals and Conversions Bitwise Operations Other Built-in Numeric Tools

Other Numeric Types

Decimal Type

Decimal basics Setting decimal precision globally Decimal context manager

Fraction Type

Fraction basics Numeric accuracy in fractions and decimals Fraction conversions and mixed types

Sets

Set basics in Python 2.6 and earlier Set literals in Python 3.X and 2.7 Immutable constraints and frozen sets Set comprehensions in Python 3.X and 2.7 Why sets?

Booleans

Numeric Extensions Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 6. The Dynamic Typing Interlude The Case of the Missing Declaration Statements

Variables, Objects, and References Types Live with Objects, Not Variables Objects Are Garbage-Collected

Shared References

Shared References and In-Place Changes Shared References and Equality

Dynamic Typing Is Everywhere Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 7. String Fundamentals This Chapter’s Scope

Unicode: The Short Story

String Basics String Literals

Single- and Double-Quoted Strings Are the Same Escape Sequences Represent Special Characters

Note

Raw Strings Suppress Escapes

Note

Triple Quotes Code Multiline Block Strings

Strings in Action

Basic Operations Indexing and Slicing

Extended slicing: The third limit and slice objects

String Conversion Tools

Character code conversions

Changing Strings I

Note

String Methods

Method Call Syntax Methods of Strings String Method Examples: Changing Strings II String Method Examples: Parsing Text Other Common String Methods in Action The Original string Module’s Functions (Gone in 3.X)

String Formatting Expressions

Formatting Expression Basics Advanced Formatting Expression Syntax Advanced Formatting Expression Examples Dictionary-Based Formatting Expressions

String Formatting Method Calls

Formatting Method Basics Adding Keys, Attributes, and Offsets Advanced Formatting Method Syntax Advanced Formatting Method Examples Comparison to the % Formatting Expression

Note

Why the Format Method?

Extra features: Special-case “batteries” versus general techniques Flexible reference syntax: Extra complexity and functional overlap Explicit value references: Now optional and unlikely to be used Named method and context-neutral arguments: Aesthetics versus practice Functions versus expressions: A minor convenience Note

General Type Categories

Types Share Operation Sets by Categories Mutable Types Can Be Changed in Place

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 8. Lists and Dictionaries Lists Lists in Action

Basic List Operations List Iteration and Comprehensions Indexing, Slicing, and Matrixes Changing Lists in Place

Index and slice assignments List method calls More on sorting lists Note Other common list methods Other common list operations

Dictionaries Dictionaries in Action

Basic Dictionary Operations Changing Dictionaries in Place More Dictionary Methods

Note

Example: Movie Database

Preview: Mapping values to keys

Dictionary Usage Notes

Using dictionaries to simulate flexible lists: Integer keys Using dictionaries for sparse data structures: Tuple keys Avoiding missing-key errors Nesting in dictionaries

Other Ways to Make Dictionaries Dictionary Changes in Python 3.X and 2.7

Dictionary comprehensions in 3.X and 2.7 Dictionary views in 3.X (and 2.7 via new methods) Dictionary views and sets Sorting dictionary keys in 3.X Dictionary magnitude comparisons no longer work in 3.X The has_key method is dead in 3.X: Long live in!

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 9. Tuples, Files, and Everything Else

Note

Tuples

Tuples in Action

Tuple syntax peculiarities: Commas and parentheses Conversions, methods, and immutability

Why Lists and Tuples? Records Revisited: Named Tuples

Files

Opening Files Using Files Files in Action

Note

Text and Binary Files: The Short Story Storing Python Objects in Files: Conversions Storing Native Python Objects: pickle

Note

Storing Python Objects in JSON Format

Note

Storing Packed Binary Data: struct File Context Managers Other File Tools

Note

Core Types Review and Summary

Object Flexibility References Versus Copies Comparisons, Equality, and Truth

Python 2.X and 3.X mixed-type comparisons and sorts Python 2.X and 3.X dictionary comparisons

The Meaning of True and False in Python

The None object The bool type

Python’s Type Hierarchies Type Objects

Note

Other Types in Python

Built-in Type Gotchas

Assignment Creates References, Not Copies Repetition Adds One Level Deep Beware of Cyclic Data Structures Immutable Types Can’t Be Changed in Place

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part II Exercises Part III. Statements and Syntax Chapter 10. Introducing Python Statements The Python Conceptual Hierarchy Revisited Python’s Statements A Tale of Two ifs

What Python Adds What Python Removes

Parentheses are optional End-of-line is end of statement End of indentation is end of block

Why Indentation Syntax? A Few Special Cases

Statement rule special cases Block rule special case

A Quick Example: Interactive Loops

A Simple Interactive Loop

Note

Doing Math on User Inputs

Note

Handling Errors by Testing Inputs Handling Errors with try Statements

Supporting floating-point numbers Note

Nesting Code Three Levels Deep

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 11. Assignments, Expressions, and Prints Assignment Statements

Assignment Statement Forms Sequence Assignments

Advanced sequence assignment patterns

Extended Sequence Unpacking in Python 3.X

Extended unpacking in action Boundary cases A useful convenience Application to for loops

Multiple-Target Assignments

Multiple-target assignment and shared references

Augmented Assignments

Augmented assignment and shared references

Variable Name Rules

Naming conventions Names have no type, but objects do Note

Expression Statements

Expression Statements and In-Place Changes

Print Operations

The Python 3.X print Function

Call format The 3.X print function in action

The Python 2.X print Statement

Statement forms The 2.X print statement in action

Print Stream Redirection

The Python “hello world” program Manual stream redirection Automatic stream redirection

Version-Neutral Printing

2to3 converter Importing from __future__ Neutralizing display differences with code Note

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 12. if Tests and Syntax Rules if Statements

General Format Basic Examples Multiway Branching

Handling switch defaults Handling larger actions

Python Syntax Revisited

Block Delimiters: Indentation Rules

Avoid mixing tabs and spaces: New error checking in 3.X

Statement Delimiters: Lines and Continuations A Few Special Cases

Truth Values and Boolean Tests The if/else Ternary Expression Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 13. while and for Loops while Loops

General Format Examples

break, continue, pass, and the Loop else

General Loop Format pass

Note

continue break Loop else

Note More on the loop else

for Loops

General Format Examples

Basic usage Other data types Tuple assignment in for loops Python 3.X extended sequence assignment in for loops Nested for loops

Loop Coding Techniques

Counter Loops: range Sequence Scans: while and range Versus for Sequence Shufflers: range and len Nonexhaustive Traversals: range Versus Slices Changing Lists: range Versus Comprehensions Parallel Traversals: zip and map

map equivalence in Python 2.X Note Dictionary construction with zip

Generating Both Offsets and Items: enumerate

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 14. Iterations and Comprehensions Iterations: A First Look

Note The Iteration Protocol: File Iterators

Note

Manual Iteration: iter and next

The full iteration protocol Manual iteration

Other Built-in Type Iterables

List Comprehensions: A First Detailed Look

List Comprehension Basics Using List Comprehensions on Files

Note

Extended List Comprehension Syntax

Filter clauses: if Nested loops: for Note

Other Iteration Contexts New Iterables in Python 3.X

Impacts on 2.X Code: Pros and Cons The range Iterable

Note

The map, zip, and filter Iterables Multiple Versus Single Pass Iterators Dictionary View Iterables

Other Iteration Topics Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 15. The Documentation Interlude Python Documentation Sources

# Comments The dir Function

Note

Docstrings: __doc__

User-defined docstrings Docstring standards and priorities Built-in docstrings

PyDoc: The help Function PyDoc: HTML Reports

Python 3.2 and later: PyDoc’s all-browser mode Python 3.2 and earlier: GUI client Note

Beyond docstrings: Sphinx The Standard Manual Set Web Resources Published Books

Common Coding Gotchas Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part III Exercises Part IV. Functions and Generators Chapter 16. Function Basics Why Use Functions? Coding Functions

def Statements def Executes at Runtime

A First Example: Definitions and Calls

Definition Calls Polymorphism in Python

A Second Example: Intersecting Sequences

Definition Calls Polymorphism Revisited Local Variables

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 17. Scopes Python Scope Basics

Scope Details Name Resolution: The LEGB Rule

Other Python scopes: Preview

Scope Example The Built-in Scope

Redefining built-in names: For better or worse Note

The global Statement

Program Design: Minimize Global Variables Program Design: Minimize Cross-File Changes

Note

Other Ways to Access Globals

Scopes and Nested Functions

Nested Scope Details Nested Scope Examples Factory Functions: Closures

A simple function factory Note Closures versus classes, round 1 Note

Retaining Enclosing Scope State with Defaults

Nested scopes, defaults, and lambdas Loop variables may require defaults, not scopes Arbitrary scope nesting

The nonlocal Statement in 3.X

nonlocal Basics nonlocal in Action

Using nonlocal for changes Boundary cases

Why nonlocal? State Retention Options

State with nonlocal: 3.X only State with Globals: A Single Copy Only State with Classes: Explicit Attributes (Preview) State with Function Attributes: 3.X and 2.X

State with mutables: Obscure ghost of Pythons past?

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 18. Arguments Argument-Passing Basics

Arguments and Shared References Avoiding Mutable Argument Changes Simulating Output Parameters and Multiple Results

Note

Special Argument-Matching Modes

Argument Matching Basics Argument Matching Syntax The Gritty Details

Note

Keyword and Default Examples

Keywords Defaults Combining keywords and defaults Note

Arbitrary Arguments Examples

Headers: Collecting arguments Calls: Unpacking arguments Note Applying functions generically The defunct apply built-in (Python 2.X)

Python 3.X Keyword-Only Arguments

Ordering rules Why keyword-only arguments?

The min Wakeup Call!

Full Credit Bonus Points The Punch Line...

Generalized Set Functions

Note

Emulating the Python 3.X print Function

Using Keyword-Only Arguments

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 19. Advanced Function Topics Function Design Concepts Recursive Functions

Summation with Recursion Coding Alternatives Loop Statements Versus Recursion Handling Arbitrary Structures

Recursion versus queues and stacks Cycles, paths, and stack limits More recursion examples

Function Objects: Attributes and Annotations

Indirect Function Calls: “First Class” Objects Function Introspection Function Attributes Function Annotations in 3.X

Anonymous Functions: lambda

lambda Basics Why Use lambda?

Multiway branch switches: The finale

How (Not) to Obfuscate Your Python Code Scopes: lambdas Can Be Nested Too

Functional Programming Tools

Mapping Functions over Iterables: map Selecting Items in Iterables: filter Combining Items in Iterables: reduce

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 20. Comprehensions and Generations List Comprehensions and Functional Tools

List Comprehensions Versus map Adding Tests and Nested Loops: filter

Formal comprehension syntax

Example: List Comprehensions and Matrixes Don’t Abuse List Comprehensions: KISS

On the other hand: performance, conciseness, expressiveness Note

Generator Functions and Expressions

Generator Functions: yield Versus return

State suspension Iteration protocol integration Note Generator functions in action Why generator functions? Extended generator function protocol: send versus next

Generator Expressions: Iterables Meet Comprehensions

Why generator expressions? Generator expressions versus map Generator expressions versus filter

Generator Functions Versus Generator Expressions Generators Are Single-Iteration Objects Generation in Built-in Types, Tools, and Classes

Generators and library tools: Directory walkers Generators and function application Preview: User-defined iterables in classes

Example: Generating Scrambled Sequences

Scrambling sequences Simple functions Generator functions Generator expressions Tester client Permutations: All possible combinations

Don’t Abuse Generators: EIBTI

On the other hand: Space and time, conciseness, expressiveness

Example: Emulating zip and map with Iteration Tools

Coding your own map(func, ...) Coding your own zip(...) and map(None, ...) Note

Comprehension Syntax Summary

Scopes and Comprehension Variables Comprehending Set and Dictionary Comprehensions Extended Comprehension Syntax for Sets and Dictionaries

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 21. The Benchmarking Interlude Timing Iteration Alternatives

Timing Module: Homegrown Timing Script Timing Results

The impact of function calls: map Note

Timing Module Alternatives

Using keyword-only arguments in 3.X

Other Suggestions

Timing Iterations and Pythons with timeit

Basic timeit Usage

Interactive usage and API calls Command-line usage Timing multiline statements Other usage modes: Setup, totals, and objects

Benchmark Module and Script: timeit Benchmark Script Results More Fun with Benchmarks

A win for map and a rare loss for PyPy The impact of function calls revisited Comparing techniques: Homegrown versus batteries Room for improvement: Setup

Other Benchmarking Topics: pystones Function Gotchas

Local Names Are Detected Statically Defaults and Mutable Objects Functions Without returns Miscellaneous Function Gotchas

Enclosing scopes and loop variables: Factory functions Hiding built-ins by assignment: Shadowing

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part IV Exercises Part V. Modules and Packages Chapter 22. Modules: The Big Picture Why Use Modules? Python Program Architecture

How to Structure a Program Imports and Attributes Standard Library Modules

How Imports Work

1. Find It 2. Compile It (Maybe) 3. Run It

Byte Code Files: __pycache__ in Python 3.2+

Byte Code File Models in Action

The Module Search Path

Configuring the Search Path Search Path Variations The sys.path List Module File Selection

Module sources Selection priorities Import hooks and ZIP files Note Optimized byte code files

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 23. Module Coding Basics Module Creation

Module Filenames Other Kinds of Modules

Module Usage

The import Statement The from Statement The from * Statement

Note

Imports Happen Only Once

Initialization code

import and from Are Assignments

Changing mutables in modules Cross-file name changes

import and from Equivalence Potential Pitfalls of the from Statement

When import is required

Module Namespaces

Files Generate Namespaces Namespace Dictionaries: __dict__ Attribute Name Qualification Imports Versus Scopes Namespace Nesting

Reloading Modules

Note reload Basics reload Example

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 24. Module Packages Package Import Basics

Packages and Search Path Settings Package __init__.py Files

Note Package initialization file roles Note

Package Import Example

from Versus import with Packages

Why Use Package Imports?

A Tale of Three Systems

Package Relative Imports

Note Changes in Python 3.X Relative Import Basics Why Relative Imports?

The relative imports solution in 3.X Relative imports versus absolute package paths

The Scope of Relative Imports Module Lookup Rules Summary Relative Imports in Action

Imports outside packages Imports within packages Imports are still relative to the CWD Selecting modules with relative and absolute imports Relative imports search packages only Imports are still relative to the CWD, again

Pitfalls of Package-Relative Imports: Mixed Use

The issue Fix 1: Package subdirectories Fix 2: Full path absolute import Example: Application to module self-test code (preview) Note

Python 3.3 Namespace Packages

Namespace Package Semantics

The import algorithm

Impacts on Regular Packages: Optional __init__.py Namespace Packages in Action Namespace Package Nesting Files Still Have Precedence over Directories

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 25. Advanced Module Topics Module Design Concepts Data Hiding in Modules

Minimizing from * Damage: _X and __all__

Enabling Future Language Features: __future__ Mixed Usage Modes: __name__ and __main__

Unit Tests with __name__

Note

Example: Dual Mode Code

Note Currency Symbols: Unicode in Action Docstrings: Module Documentation at Work

Changing the Module Search Path The as Extension for import and from Example: Modules Are Objects Importing Modules by Name String

Running Code Strings Direct Calls: Two Options

Example: Transitive Module Reloads

A Recursive Reloader

Testing recursive reloads

Alternative Codings

Testing reload variants

Module Gotchas

Module Name Clashes: Package and Package-Relative Imports Statement Order Matters in Top-Level Code from Copies Names but Doesn’t Link from * Can Obscure the Meaning of Variables reload May Not Impact from Imports reload, from, and Interactive Testing Recursive from Imports May Not Work

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part V Exercises Part VI. Classes and OOP Chapter 26. OOP: The Big Picture Why Use Classes? OOP from 30,000 Feet

Attribute Inheritance Search Classes and Instances Method Calls Coding Class Trees Operator Overloading OOP Is About Code Reuse

Polymorphism and classes Programming by customization

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 27. Class Coding Basics Classes Generate Multiple Instance Objects

Class Objects Provide Default Behavior Instance Objects Are Concrete Items A First Example

Classes Are Customized by Inheritance

Note A Second Example Classes Are Attributes in Modules

Classes Can Intercept Python Operators

A Third Example

Returning results, or not

Why Use Operator Overloading?

The World’s Simplest Python Class

Records Revisited: Classes Versus Dictionaries

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 28. A More Realistic Example Step 1: Making Instances

Coding Constructors Testing As You Go Using Code Two Ways

Step 2: Adding Behavior Methods

Coding Methods

Step 3: Operator Overloading

Providing Print Displays

Step 4: Customizing Behavior by Subclassing

Coding Subclasses Augmenting Methods: The Bad Way Augmenting Methods: The Good Way Polymorphism in Action Inherit, Customize, and Extend OOP: The Big Idea

Step 5: Customizing Constructors, Too

OOP Is Simpler Than You May Think Other Ways to Combine Classes

Step 6: Using Introspection Tools

Special Class Attributes A Generic Display Tool Instance Versus Class Attributes Name Considerations in Tool Classes Our Classes’ Final Form

Step 7 (Final): Storing Objects in a Database

Pickles and Shelves

The pickle module The shelve module

Storing Objects on a Shelve Database Exploring Shelves Interactively Updating Objects on a Shelve

Future Directions Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 29. Class Coding Details The class Statement

General Form Example

Methods

Method Example Calling Superclass Constructors Other Method Call Possibilities

Note

Inheritance

Attribute Tree Construction Specializing Inherited Methods Class Interface Techniques Abstract Superclasses

Abstract superclasses in Python 3.X and 2.6+: Preview

Namespaces: The Conclusion

Simple Names: Global Unless Assigned Attribute Names: Object Namespaces The “Zen” of Namespaces: Assignments Classify Names Nested Classes: The LEGB Scopes Rule Revisited Namespace Dictionaries: Review

Note

Namespace Links: A Tree Climber

Documentation Strings Revisited Classes Versus Modules Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 30. Operator Overloading The Basics

Constructors and Expressions: __init__ and __sub__

Note

Common Operator Overloading Methods

Note

Indexing and Slicing: __getitem__ and __setitem__

Intercepting Slices Slicing and Indexing in Python 2.X But 3.X’s __index__ Is Not Indexing!

Index Iteration: __getitem__ Iterable Objects: __iter__ and __next__

Note User-Defined Iterables

Single versus multiple scans Classes versus generators

Multiple Iterators on One Object

Classes versus slices

Coding Alternative: __iter__ plus yield

Multiple iterators with yield

Membership: __contains__, __iter__, and __getitem__ Attribute Access: __getattr__ and __setattr__

Attribute Reference Attribute Assignment and Deletion

Note

Other Attribute Management Tools Emulating Privacy for Instance Attributes: Part 1

String Representation: __repr__ and __str__

Why Two Display Methods? Display Usage Notes

Right-Side and In-Place Uses: __radd__ and __iadd__

Right-Side Addition

Reusing __add__ in __radd__ Propagating class type

In-Place Addition

Call Expressions: __call__

Function Interfaces and Callback-Based Code

Comparisons: __lt__, __gt__, and Others

The __cmp__ Method in Python 2.X

Boolean Tests: __bool__ and __len__

Boolean Methods in Python 2.X

Object Destruction: __del__

Destructor Usage Notes

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 31. Designing with Classes Python and OOP

Polymorphism Means Interfaces, Not Call Signatures

OOP and Inheritance: “Is-a” Relationships OOP and Composition: “Has-a” Relationships

Stream Processors Revisited

OOP and Delegation: “Wrapper” Proxy Objects

Note

Pseudoprivate Class Attributes

Name Mangling Overview Why Use Pseudoprivate Attributes?

Methods Are Objects: Bound or Unbound

Note Unbound Methods Are Functions in 3.X

Note

Bound Methods and Other Callable Objects

Other callables

Classes Are Objects: Generic Object Factories

Why Factories?

Multiple Inheritance: “Mix-in” Classes

Coding Mix-in Display Classes

Listing instance attributes with __dict__ Listing inherited attributes with dir Note Listing attributes per object in class trees Running the tree lister Usage variation: Showing underscore name values Note Usage variation: Running on larger modules Collector module Room for improvement: MRO, slots, GUIs

Other Design-Related Topics Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 32. Advanced Class Topics

Note

Extending Built-in Types

Extending Types by Embedding Extending Types by Subclassing

The “New Style” Class Model

Just How New Is New-Style?

New-Style Class Changes

Note Attribute Fetch for Built-ins Skips Instances

Why the lookup change? Implications for attribute interception Proxy coding requirements For more details Note

Type Model Changes

Implications for type testing

All Classes Derive from “object”

Implications for defaults

Diamond Inheritance Change

Implications for diamond inheritance trees Explicit conflict resolution Note Scope of search order change

More on the MRO: Method Resolution Order

The MRO algorithm Tracing the MRO

Example: Mapping Attributes to Inheritance Sources

Note

New-Style Class Extensions

Slots: Attribute Declarations

Slot basics Note Slots and namespace dictionaries Multiple __slot__ lists in superclasses Handling slots and other “virtual” attributes generically Slot usage rules Example impacts of slots: ListTree and mapattrs What about slots speed?

Properties: Attribute Accessors

Property basics

__getattribute__ and Descriptors: Attribute Tools Other Class Changes and Extensions

Static and Class Methods

Why the Special Methods? Static Methods in 2.X and 3.X Static Method Alternatives Using Static and Class Methods Counting Instances with Static Methods Counting Instances with Class Methods

Counting instances per class with class methods Note

Decorators and Metaclasses: Part 1

Function Decorator Basics A First Look at User-Defined Function Decorators A First Look at Class Decorators and Metaclasses For More Details

The super Built-in Function: For Better or Worse?

The Great super Debate Traditional Superclass Call Form: Portable, General Basic super Usage and Its Tradeoffs

Odd semantics: A magic proxy in Python 3.X Pitfall: Adding multiple inheritance naively Limitation: Operator overloading Use differs in Python 2.X: Verbose calls

The super Upsides: Tree Changes and Dispatch Runtime Class Changes and super Cooperative Multiple Inheritance Method Dispatch

The basics: Cooperative super call in action Constraint: Call chain anchor requirement Scope: An all-or-nothing model Flexibility: Call ordering assumptions Customization: Method replacement Coupling: Application to mix-in classes Customization: Same-argument constraints

The super Summary

Class Gotchas

Changing Class Attributes Can Have Side Effects Changing Mutable Class Attributes Can Have Side Effects, Too Multiple Inheritance: Order Matters Scopes in Methods and Classes Miscellaneous Class Gotchas

Choose per-instance or class storage wisely You usually want to call superclass constructors Delegation-based classes in 3.X: __getattr__ and built-ins

KISS Revisited: “Overwrapping-itis”

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part VI Exercises Part VII. Exceptions and Tools Chapter 33. Exception Basics Why Use Exceptions?

Exception Roles

Exceptions: The Short Story

Default Exception Handler Catching Exceptions

Note

Raising Exceptions User-Defined Exceptions Termination Actions

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 34. Exception Coding Details

Note

The try/except/else Statement

How try Statements Work try Statement Clauses

Catching any and all exceptions Catching all: The empty except and Exception Note

The try else Clause Example: Default Behavior Example: Catching Built-in Exceptions

The try/finally Statement

Note Example: Coding Termination Actions with try/finally

Unified try/except/finally

Unified try Statement Syntax Combining finally and except by Nesting Unified try Example

The raise Statement

Note Raising Exceptions Scopes and try except Variables Propagating Exceptions with raise Python 3.X Exception Chaining: raise from

Note

The assert Statement

Example: Trapping Constraints (but Not Errors!)

with/as Context Managers

Basic Usage The Context Management Protocol Multiple Context Managers in 3.1, 2.7, and Later

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 35. Exception Objects

Note

Exceptions: Back to the Future

String Exceptions Are Right Out! Class-Based Exceptions Coding Exceptions Classes

Why Exception Hierarchies? Built-in Exception Classes

Built-in Exception Categories

Note

Default Printing and State

Custom Print Displays

Note

Custom Data and Behavior

Providing Exception Details Providing Exception Methods

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 36. Designing with Exceptions Nesting Exception Handlers

Example: Control-Flow Nesting Example: Syntactic Nesting

Exception Idioms

Breaking Out of Multiple Nested Loops: “go to” Exceptions Aren’t Always Errors Functions Can Signal Conditions with raise Closing Files and Server Connections Debugging with Outer try Statements

Note

Running In-Process Tests More on sys.exc_info Displaying Errors and Tracebacks

Note

Exception Design Tips and Gotchas

What Should Be Wrapped Catching Too Much: Avoid Empty except and Exception Catching Too Little: Use Class-Based Categories

Core Language Summary

The Python Toolset Development Tools for Larger Projects

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part VII Exercises Part VIII. Advanced Topics Chapter 37. Unicode and Byte Strings String Changes in 3.X String Basics

Character Encoding Schemes How Python Stores Strings in Memory Python’s String Types

Why the different string types?

Text and Binary Files

Coding Basic Strings

Python 3.X String Literals

Python 2.X Unicode literals in Python 3.3

Python 2.X String Literals String Type Conversions

Coding Unicode Strings

Coding ASCII Text Coding Non-ASCII Text Encoding and Decoding Non-ASCII text Other Encoding Schemes Byte String Literals: Encoded Text Converting Encodings Coding Unicode Strings in Python 2.X

Mixing string types in 2.X

Source File Character Set Encoding Declarations

Note

Using 3.X bytes Objects

Method Calls Sequence Operations Other Ways to Make bytes Objects Mixing String Types

Using 3.X/2.6+ bytearray Objects

bytearrays in Action Python 3.X String Types Summary

Using Text and Binary Files

Text File Basics Text and Binary Modes in 2.X and 3.X Type and Content Mismatches in 3.X

Using Unicode Files

Reading and Writing Unicode in 3.X

Manual encoding File output encoding File input decoding Decoding mismatches

Handling the BOM in 3.X

Dropping the BOM in Notepad Dropping the BOM in Python

Unicode Files in 2.X Unicode Filenames and Streams

Filenames: Text versus bytes Stream content: PYTHONIOENCODING

Other String Tool Changes in 3.X

The re Pattern-Matching Module The struct Binary Data Module The pickle Object Serialization Module XML Parsing Tools

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 38. Managed Attributes Why Manage Attributes?

Inserting Code to Run on Attribute Access

Properties

The Basics A First Example Computed Attributes Coding Properties with Decorators

Setter and deleter decorators

Descriptors

The Basics

Descriptor method arguments Read-only descriptors Note

A First Example Computed Attributes Using State Information in Descriptors How Properties and Descriptors Relate

Descriptors and slots and more Note

__getattr__ and __getattribute__

The Basics

Avoiding loops in attribute interception methods Note

A First Example

Using __getattribute__

Computed Attributes

Using __getattribute__

__getattr__ and __getattribute__ Compared Management Techniques Compared Intercepting Built-in Operation Attributes

Delegation-based managers revisited

Example: Attribute Validations

Using Properties to Validate

Testing code

Using Descriptors to Validate

Option 1: Validating with shared descriptor instance state Option 2: Validating with per-client-instance state

Using __getattr__ to Validate Using __getattribute__ to Validate

Chapter Summary Test Your Knowledge: Quiz

Test Your Knowledge: Answers

Chapter 39. Decorators What’s a Decorator?

Managing Calls and Instances Managing Functions and Classes Using and Defining Decorators Why Decorators?

Note

The Basics

Function Decorators

Usage Implementation Supporting method decoration

Class Decorators

Usage Implementation Supporting multiple instances

Decorator Nesting Decorator Arguments Decorators Manage Functions and Classes, Too

Coding Function Decorators

Tracing Calls Decorator State Retention Options

Class instance attributes Enclosing scopes and globals Enclosing scopes and nonlocals Function attributes

Class Blunders I: Decorating Methods

Using nested functions to decorate methods Using descriptors to decorate methods

Timing Calls

Decorators versus per-call timing Note Testing subtleties

Adding Decorator Arguments

Timing with decorator arguments Note

Coding Class Decorators

Singleton Classes

Coding alternatives

Tracing Object Interfaces

Tracing interfaces with class decorators Applying class decorators to built-in types Note

Class Blunders II: Retaining Multiple Instances Decorators Versus Manager Functions Why Decorators? (Revisited)

Note

Managing Functions and Classes Directly Example: “Private” and “Public” Attributes

Implementing Private Attributes Implementation Details I

Inheritance versus delegation Decorator arguments State retention and enclosing scopes Using __dict__ and __slots__ (and other virtual names)

Generalizing for Public Declarations, Too Implementation Details II

Using __X pseudoprivate names Breaking privacy Decorator tradeoffs

Open Issues

Caveat: Implicitly run operator overloading methods fail to delegate under 3.X Approaches to redefining operator overloading methods for 3.X

Inline definition Mix-in superclasses Coding variations: Routers, descriptors, automation

Should operator methods be validated? Implementation alternatives: __getattribute__ inserts, call stack inspection

Python Isn’t About Control

Example: Validating Function Arguments

The Goal A Basic Range-Testing Decorator for Positional Arguments Generalizing for Keywords and Defaults, Too Implementation Details

Function introspection Argument assumptions Matching algorithm

Open Issues

Invalid calls Arbitrary arguments Decorator nesting

Decorator Arguments Versus Function Annotations Other Applications: Type Testing (If You Insist!)

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 40. Metaclasses To Metaclass or Not to Metaclass

Increasing Levels of “Magic” A Language of Hooks The Downside of “Helper” Functions Metaclasses Versus Class Decorators: Round 1

The Metaclass Model

Classes Are Instances of type Metaclasses Are Subclasses of Type Class Statement Protocol

Declaring Metaclasses

Declaration in 3.X Declaration in 2.X Metaclass Dispatch in Both 3.X and 2.X

Note

Coding Metaclasses

A Basic Metaclass Customizing Construction and Initialization Other Metaclass Coding Techniques

Using simple factory functions Overloading class creation calls with normal classes Overloading class creation calls with metaclasses

Inheritance and Instance

Metaclass Versus Superclass Inheritance: The Full Story

Python’s inheritance algorithm: The simple version The descriptors special case Python’s inheritance algorithm: The somewhat-more-complete version Assignment inheritance The built-ins special case

Metaclass Methods

Metaclass Methods Versus Class Methods Operator Overloading in Metaclass Methods

Example: Adding Methods to Classes

Manual Augmentation Metaclass-Based Augmentation Metaclasses Versus Class Decorators: Round 2

Decorator-based augmentation Managing instances instead of classes Metaclass and class decorator equivalence?

Example: Applying Decorators to Methods

Tracing with Decoration Manually Tracing with Metaclasses and Decorators Applying Any Decorator to Methods Metaclasses Versus Class Decorators: Round 3 (and Last)

Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Chapter 41. All Good Things The Python Paradox

On “Optional” Language Features Against Disquieting Improvements Complexity Versus Power Simplicity Versus Elitism Closing Thoughts

Where to Go From Here Encore: Print Your Own Completion Certificate! Part IX. Appendixes Appendix A. Installation and Configuration Installing the Python Interpreter

Is Python Already Present? Where to Get Python Installation Steps

Configuring Python

Python Environment Variables How to Set Configuration Options

Unix/Linux shell variables DOS variables (and older Windows) Windows environment variable GUI Windows registry Path files

Python Command-Line Arguments

Running script files with arguments Running code given in arguments and standard input Running modules on the search path Optimized and unbuffered modes Post-run interactive mode Python 2.X command-line arguments

Python 3.3 Windows Launcher Command Lines

For More Help Appendix B. The Python 3.3 Windows Launcher The Unix Legacy The Windows Legacy Introducing the New Windows Launcher A Windows Launcher Tutorial

Step 1: Using Version Directives in Files Step 2: Using Command-Line Version Switches Step 3: Using and Changing Defaults

Pitfalls of the New Windows Launcher

Pitfall 1: Unrecognized Unix !# Lines Fail

Book examples impact and fix

Pitfall 2: The Launcher Defaults to 2.X

Book examples impact and fix

Pitfall 3: The New PATH Extension Option

Conclusions: A Net Win for Windows Appendix C. Python Changes and This Book Major 2.X/3.X Differences

3.X Differences 3.X-Only Extensions

General Remarks: 3.X Changes

Changes in Libraries and Tools

Standard library changes Tools changes

Migrating to 3.X

Fifth Edition Python Changes: 2.7, 3.2, 3.3

Changes in Python 2.7 Changes in Python 3.3 Changes in Python 3.2

Fourth Edition Python Changes: 2.6, 3.0, 3.1

Changes in Python 3.1 Changes in Python 3.0 and 2.6 Specific Language Removals in 3.0

Third Edition Python Changes: 2.3, 2.4, 2.5 Earlier and Later Python Changes Appendix D. Solutions to End-of-Part Exercises Part I, Getting Started Part II, Types and Operations Part III, Statements and Syntax Part IV, Functions and Generators Part V, Modules and Packages Part VI, Classes and OOP Part VII, Exceptions and Tools Index

A note on the digital index Symbols A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

About the Author Colophon Special Upgrade Offer Learning Python

Mark Lutz

Editor

Rachel Roumeliotis

← Prev
Back
Next →

← Prev
Back
Next →