GarlicSim blog

27th January 2011

Text

`address_tools`: More powerful replacements for `eval` and `repr`

GarlicSim module of the week

This is the third post in the GarlicSim module of the week series. This is a series of blog posts in which I explore different modules from the garlicsim.general_misc package. These are modules which are not specific to computer simulations; they can be relevant to any kind of Python program, and you are welcome to import them from garlicsim.general_misc and use them in your projects.

Part one: caching.cache: A caching decorator that understands arguments

Part two: caching.CachedType: A metaclass for sharing instances

Part three: address_tools: More powerful replacements for eval and repr 

Part four: ContextManager and the manage_context method

Part five: cute_profile: Profile your Python code on the fly

Before we start…

I think that this package is going to get some negative responses from people.

This package was born in the same circumstances that most of the other packages on garlicsim.general_misc were: I had a need for a certain tool; I looked for it online for a while; I couldn’t find it; I asked people on various forums whether they know of some package on PyPI that does this; they said no; I decided to write it myself with documentation and tests; and now I’m sharing it with you.

But when I was in the “asking people on various forums whether they know of some package on PyPI that does this” phase, I discovered that people found my idea revolting. My StackOverflow question got downvoted to -6 points. I was told by various people that what I want is not Pythonic, that it goes against the Zen of Python, that it’s a bad design choice, that it’s impossible, and that I should go program in C.

I don’t like these kinds of internet arguments. It’s easy to criticize someone else’s design choices, but every project needs to make some non-ideal design choices in order to actually deliver working software. I wish I could code all my programs in the purest, most beautiful and mathematical architectures imaginable. I try to get as close to this ideal as possible, but sometimes you need something that just works, so you could deliver your project on time. And when the time comes for you, dear reader, to be in this position, and you will ask on various forums where you could get a tool that solves your problem, I hope that people will help you get the tool you need to solve your problem instead of judging you or trying to educate you.

Of course it’s okay to have a discussion about why something can be a problematic design decision; but after a person or two said that it’s a bad design decision, and the original poster still wants it, then it’s time to stop judging and start helping. Or at least get out of the way of people who actually want to help.

So now we got that out of the way, let’s speak about address_tools and what it does.

address_tools: More powerful replacements for eval and repr

The problem that address_tools was originally designed to solve was getting the “address” of a class, and possibly shortening it to an equivalent but shorter string. But after I implemented that, I realized that this could be generalized into a pair of functions, address_tools.describe and address_tools.resolve, that can replace the built-in repr and eval functions. So I will explain about those first, and after that I will explain about the “class address” and shortening business.

So, Python has two built-in functions called repr and eval. You can say that they are opposites of each other: repr ”describes” a Python object as a string, and eval evaluates a string into a Python object.

When is this useful? This is useful in various cases: For example when you have a GUI program (ahem, GarlicSim) that needs to show the user Python objects and let him manipulate them. (Screenshot.) As a more well-known example, Django uses something like eval to let the user specify functions without importing them, both in settings.py and urls.py.

In some easy cases, repr and eval are the exact converses of each other:

>>> repr([1, 2, 'meow', {3: 4}])
"[1, 2, 'meow', {3: 4}]"
>>> eval(
...     repr(
...         [1, 2, 'meow', {3: 4}]
...     )
... )
[1, 2, 'meow', {3: 4}]

When you put a simple object like that in repr and then put the resulting string in eval, you get the original object again. That’s really pretty, because then we have something like a one-to-one correspondence between objects and strings used to describe them.

In a happy-sunshine world, there would indeed be a perfect one-to-one mapping between Python objects and strings that describe them. You got a Python object? You can turn it into a string so a human could easily see it, and the string will be all the human will need to create the object again. But unfortunately some objects just can’t be meaningfully described as a string in a reversible way:

>>> import threading
>>> lock = threading.Lock()
>>> repr(lock)
'<thread.lock object at 0x00ABF110>'
>>> eval(repr(lock))
Traceback (most recent call last):
  File "", line 1, in 
invalid syntax: , line 1, pos 1

A lock object is used for synchronization between threads. You can’t really describe a lock in a string in a reversible way; A lock is a breathing, living thing that threads in your program interact with, it’s not a data-type like a list or a dict.

So when we call repr on a lock object, we get something like this: '<thread.lock object at 0x00ABF110>'. Enveloping the text with pointy brackets is Python’s way of saying, “you can’t turn this string back into an object, sorry, but I’m still going to give you some valuable information about the object, in the hope that it’ll be useful for you.” This is good behavior on Python’s part. We may not be able to use eval on this string, but at least we got some info about the object, and introspection is a very useful ability.

So some objects, like lists, dicts and strings, can be easily described by repr in a reversible way; Some objects, like locks, queues, and file objects, simply cannot by their nature; And then there are the objects in between.

Classes, functions, methods, modules

What happens when we run repr for a Python class?

>>> import decimal
>>> repr(decimal.Decimal)
"<class 'decimal.Decimal'>"

We get a pointy-bracketed un-evalable string. How about a function?

 >>> import re
>>> repr(re.match)
'<function match at 0x00E8B030>'

Same thing. We get a string that we can’t put back in eval. Is this really necessary? Why not return 'decimal.Decimal' or 're.match' so we could eval those later and get the original objects?

It is sometimes helpful that the repr string "<class 'decimal.Decimal'>" informs us that this is a class; but sometimes you want a string that you can turn back into an object. Although… eval might not be able to find it, because decimal might not be imported.

Enter address_tools:

address_tools.describe and address_tools.resolve

Let’s play with address_tools.describe and address_tools.resolve:

>>> from garlicsim.general_misc import address_tools
>>> import decimal
>>> address_tools.describe(decimal.Decimal)
'decimal.Decimal'

That’s a nice description string! We can put that back into resolve and get the original class:

>>> address_tools.resolve(address_tools.describe(decimal.Decimal)) is decimal.Decimal
True

We can use resolve to get this function, without re being imported, and it will import re by itself:

>>> address_tools.resolve('re.match')
<function match at 0x00B5E6B0>

This shtick also works on classes, functions, methods, modules, and possibly other kinds of objects.

Address shortening

One of the original purposes of address_tools was address-shortening. Sometimes I have a class inside a module inside a package inside my root package. For example, the most important class in garlicsim is garlicsim.asynchronous_crunching.project.Project. But that’s a really long string. And because it’s such an important class, it’s also available at the friendlier address garlicsim.Project.

Using address_tools.describe, we can get the shorter address of the class:

>>> from garlicsim.general_misc import address_tools
>>> import garlicsim
>>> address_tools.describe(garlicsim.Project) # By default we get the full address:
'garlicsim.asynchronous_crunching.project.Project'
>>> address_tools.describe(garlicsim.Project, shorten=True) # Bam, shortening:
'garlicsim.Project'

I recommend using this feature in any  __repr__ methods that you write for your classes, to make the resulting string shorter. Here’s how I do it.

Source and tests for address_tools

Source for address_tools, well-documented. Here are the tests.

There is also a Python 3 version of address_tools, and here are its tests. It’s available with the Python 3 fork of GarlicSim.

Please give feedback!

I’ll be happy to get any opinions, critiques and code reviews on address_tools!