Introspecting Python Functions

Toward the end of the project article for PySketch I pointed out that it would be more useful if it could detect which python modules you are using automatically, rather than guessing with a predefined list. Recently I have been working on this problem using Python's introspection engine.

Note that I am using python 3.5.2 that is packaged in Ubuntu 16.04*

I am just gonna say it now: I do not want to parse Python source, so lets go look at the introspection engine.

Python uses the built in function dir() to do basic object inspection. In case you don't know in Python everything is expressed as an object, so dir(True) is valid code and it will return a list of attributes for the object:

>>> dir(True)
['__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__delattr__', '__dir__',
 '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__',
 '__getattribute__', '__getnewargs__', '__gt__', '__hash__', '__index__', '__init__', '__int__',
 '__invert__', '__le__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__',
 '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__',
 '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__',
 '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__',
 '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'bit_length',
 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes']

Most of the listed attributes are built in default attributes that we typically don't need to worry about, however we can find out which ones are common to all objects by creating an empty class:

>>> class Empty:
...     pass
... 
>>> e = Empty()
>>> dir(e)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__',
 '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__',
 '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
 '__subclasshook__', '__weakref__']

Everything you see here is part of Python's base object type, everything else is inherited. Most of these are handles to built in functions, provided so that the class can override them and/or respond to events. For example __init__ should be familiar to anyone who has written a class in python before; it is the constructor, and it is provided for overriding from the base object definition.

So lets look at a sketch loaded with PySketch:

>>> import sketches
>>> loader = sketches.ModuleLoader("example.pys")
>>> s = loader.sketch
>>> dir(s)
['__builtins__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'cleanup',
 'loop', 'setup'] 

As expected we have the same behaviour as in the class. Listed are the docstring, the file the module was loaded from, it's parent package, and the name for module. In all there is less here than I expected. We will politely ignore __builtins__ as it exposes nothing useful for our goal. We also have the three functions we define in the sketch. Lets inspect the setup function of this sketch:

def setup():
    pin = 18
    clock = 22
    GPIO.setmode(GPIO.BCM)
    GPIO.setup(pin, GPIO.OUT)
    GPIO.setup(clock, GPIO.OUT)
    GPIO.output(pin, GPIO.High)
    time.sleep(1000)
    GPIO.output(pin, GPIO.Low)

Original Code

>>> f = s.setup
>>> dir(f)
['__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__',
 '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__',
 '__globals__', '__gt__', '__hash__', '__init__', '__kwdefaults__', '__le__', '__lt__', '__module__',
 '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__',
 '__setattr__', '__sizeof__', '__str__', '__subclasshook__']

dir() output

I have stripped out the values populated in the empty class from above. While some of them may contain useful data I'm pretty sure most of them are useless inherited values which just allow Python to do it's thing; this is what we are left with:

>>> dir(f) # With the values present in Empty() from above removed 
['__annotations__', '__call__', '__closure__', '__code__', '__defaults__', '__get__', '__globals__',
 '__kwdefaults__', '__name__', '__qualname__']

Ahh that's more manageable

The meanings of all these cryptic attributes are tabulated below. The types were extracted from the python console with the type() command. :

Unfortunately for us most of these attributes are metadata and scope; little of it relates to the contents of the function call. Therefore the only attribute of any real interest to us is __code__.

>>> dir(f.__code__)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
 '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__',
 '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'co_argcount',
 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars',
 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames']

Output from inspecting __code__ object

>>> dir(f.__code__) # With the values present in Empty() from above removed 
['co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags',
 'co_freevars', 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize',
 'co_varnames']

# Oh look all the __*__ values evaporated

Output from inspection of the code object with the base attributes removed

If we strip attributes we find in the empty class again we end up with a code object that contains only fields relevant to a method. While a lot of these can be worked out with context I think it's best to refer to the documentation here. Unfortunately the python documentation is notoriously bad and despite some exhaustive searching the closest I could find to official documentation for the __code__ class comes from the "inspect" module documentation, which can be found here. This information was incomplete, however I was able to fill in the blanks using this stack overflow answer. The types were extracted from the python console with the type() command.

Only two of these are interesting: "co_names" and "co_varnames". Despite their vague definitions in the documentation from testing I haven't seen a variable show up in both varnames and names. "co_varnames" appears to be variables locally defined within the function whereas "co_names" are tokens used with a global scope. So when searching for modules used their values are going to fall into "co_names".

There is however a problem; "co_names" is completely ungrouped with the order determined by the order tokens are used in the source. For example look at the "co_names" field for our setup function:

>>> f.__code__.co_names
('GPIO', 'setmode', 'BCM', 'setup', 'OUT', 'output', 'High', 'time', 'sleep', 'Low')
>>>
>>> f.__code__.co_varnames
('pin', 'clock')

The output of "co_names" is completely useless

This problem is made worse by the fact that these tokens are completely uninspectable (they are just strings). Yes it's objects all the way down in python, but we've reached the bottom of the pile. The "co_code" attribute is actually the bytecode that is directly fed to the virtual machine when the function is called. The code is already compiled at this point, but apparently not linked (again see the stack overflow answer above) so the tokens don't lead to other objects, they're just strings. So the question is... Now What?

This method has revealed the tokens used in the function, so theoretically these tokens could be one by one attempted to be imported by the PySketch interpreter, skipping them if they are not found; however this is a pretty crappy hack. There are some possible further routes to explore, exploiting either the ImportLib or PIP, however as this article has been waiting to be published for nearly a month I think I will for now have to accept defeat.

Stay tuned for the next instalment of "Why won't the f*cking interpreter let me do that???!!!"

- SEGFAULT

*I think it's 16.04; I'm using Linux mint 18 which uses the Ubuntu apt repos.