Abstract syntax tree (AST)

The Python syntax is converted into AST before it is compiled into byte code. This is a tree representation of the abstract syntactic structure of the source code. Processing of Python grammar is available thanks to the built-in ast module. Raw ASTs of Python code can be created using the compile() function with the ast.PyCF_ONLY_AST flag, or by using the ast.parse() helper. Direct translation in reverse is not that simple and there is no function provided in the standard library that can do so. Some projects, such as PyPy, do such things though.

The ast module provides some helper functions that allow you to work with the AST, for example:

>>> tree = ast.parse('def hello_world(): print("hello world!")')
>>> tree
<_ast.Module object at 0x00000000038E9588>
>>> ast.dump(tree)
"Module(
    body=[
        FunctionDef(
           name='hello_world', 
           args=arguments(
               args=[], 
               vararg=None, 
               kwonlyargs=[], 
               kw_defaults=[], 
               kwarg=None, 
               defaults=[]
           ), 
           body=[
               Expr(
                   value=Call(
                       func=Name(id='print', ctx=Load()), 
                       args=[Str(s='hello world!')], 
                       keywords=[]
                   )
               )
           ], 
           decorator_list=[], 
           returns=None
       )
    ]
)"  

The output of ast.dump() in the preceding example was reformatted to increase the readability and better show the tree-like structure of the AST. It is important to know that the AST can be modified before being passed to compile(). This gives you many new possibilities. For instance, new syntax nodes can be used for additional instrumentation, such as test coverage measurement. It is also possible to modify the existing code tree in order to add new semantics to the existing syntax. Such a technique is used by the MacroPy project (https://github.com/lihaoyi/macropy) to add syntactic macros to Python using the already existing syntax (refer to Figure 3):

Figure 3: How MacroPy adds syntactic macros to Python modules on import

AST can also be created in a purely artificial manner, and there is no need to parse any source at all. This gives Python programmers the ability to create Python bytecode for custom domain-specific languages, or even completely implement other programming languages on top of Python VMs.