GarlicSim blog

Silly Python riddle

Tue, 20 Dec 2011 14:16:00 +0200

Here’s a silly Python riddle for you.

Today I opened up a Python 2.7 shell, and ran two commands in it.

>>> f = lambda: g(???)
>>> f()

(Note that these are the only commands that I ran. You’re not allowed to run any other commands before them.)

The riddle: What’s the shortest thing you can put instead of ??? so my second command would not raise an exception?

Edit: I just posted it and people are already coming up with creative solutions! So now it’s time to say: The winning solution is 7 characters long. Try to match that!

———————————

Update: An anonymous commenter on Hacker News solved it!

The solution:

(Scroll down…)

The solution:

>>> f = lambda: g((yield))
>>> f()

Funny, isn’t it? I was surprised to see that the yield keyword can be used in a lambda function.

So when you type f(), it just returns a generator. If you’ll try to exhaust it, an exception will be raised because g doesn’t exist, but that’s a new line :)

It’s funny that in this case, Python seems to throw away the value of the lambda function! As we know, the yield keyword actually forms an expression whose value is None, unless you used the generator’s .send instead of .next. So you could also use .send to send in whatever value you want into the lambda function, and Python will just throw it away. Unless I’m missing something.

So that’s the only case I can think of where Python completely throws away the value of a lambda function.

The `coverage` module celebrates 10 years today!

Sun, 04 Dec 2011 21:28:00 +0200

Today, December 4^th 2011, marks the tenth birthday of the excellent coverage module! Here is the changelog entry showing it was created exactly ten years ago, on December 4^th, 2001 by Gareth Rees.

The coverage module is one of the most sturdy and reliable pillars of my development toolset. It’s a hallmark of a great tool that you never have to spend too much with it— When I want to check the statement-coverage of my tests, I just use coverage (invoked via its Nose plugin), it just works, and then I can go back to writing my tests and ensuring they cover a big part of my code. You don’t have to spend too much time with it for the simple reason that it very rarely breaks— Which is not a trivial accomplishment for a tool that acts on every, single, line of code in your project.

Kudos to Ned Batchelder for his excellent maintenance of the coverage module! I’m a relatively new user of coverage, having used it only for about 2 years, if I remember correctly. So long-time users of coverage may have a bigger perspective on this module than I do. But in the relatively short time that I’ve been using it, Ned has been an exemplary maintainer, and the few tickets that I created on the bug tracker received his attention. Ned also made sure that coverage supported Python 3.x before it was cool, and on every release he takes the time to create Windows binaries of coverage to help Windows users avoid having to compile it themselves.

If every maintainer was as dedicated as Ned, the Python community would be in a better place today, and we would have probably already seen Python 3.x going into mainstream.

Thank you, Gareth and Ned, for this wonderful module!

Deadline for GarlicSim 0.7 cancelled

Mon, 10 Oct 2011 13:30:35 +0200

Just when I finished one contract job and thought that I finally had time for some serious work on GarlicSim, I got a new contract job.

Since the deadline for the GarlicSim 0.7 release has been postponed so many times, I’m just going to cancel the deadline altogether. Sorry for the disappointment.

I’m angry at myself for not making the deadlines, but on the other hand I’m happy about having a bunch of contract jobs. This will allow me to spend a lot of time on GarlicSim later on. My contract jobs have already allowed me to do a bunch of upgrades and improvements of my work environment (from hardware upgrades, through time-saving AutoHotKey scripts, to ergonomic upgrades), and that really helps me be more productive, both in my contract work and my open-source work.

GUIs kick CLIs' asses

Tue, 06 Sep 2011 15:17:00 +0300

Or, “Hey programmers, did you know you can use a GUI with the keyboard?”

This post is part 1 of the Unpopular Opinions series. In this series I will present a few opinions of mine that go against what most other software developers think. If you disagree any of these posts, you know they’re working. Please stay civil in the comments— Criticizing my opinions is encouraged, as long as it’s done in a constructive and respectful manner.

One of the big things that have been bugging me ever since I became a professional developer is that most developers shun GUIs.

As a developer, I often get to hang out with other developers and see their tools and work processes— Sometimes in person, and sometimes online. I quickly found that almost all of them hate GUIs (Graphical User Interfaces, i.e. visual programs with windows, buttons, menus, dialogs, etc.) Most programmers much prefer working with CLIs (Command Line Interfaces, i.e. a textual shell.)[1]

I found this baffling. Why would anyone enjoy working in front of a command-line interface, where you can’t see shit and have to remember a ton of commands and flags by heart? Surely the programmer community is intelligent and passionate enough to pick the very best possible tools, so perhaps they have good reasons for preferring CLIs?

After thinking about it for a while, my conclusion is that there are two main reasons that most programmers prefer CLIs to GUIs:

Most GUIs suck. And when it comes to proper keyboard support, most GUIs suck terribly.
Most programmers are ignorant about good GUIs.

But before we get to that:

Let’s compare GUIs and CLIs

There are a bunch of parameters that we can measure GUIs against CLIs on. The two parameters that are the most important for heavy computer users are Visibility and Speed.[2]

Visibility

The first parameter we’re going to look at is visibility. This is the 800-pound gorilla that most programmers seem to ignore.

For the sake of an example, let’s take a simple everyday task: Say you want to copy a bunch of files from one folder on your computer to another.

If you were to do this with a CLI, you would do something like this:

    cd ~/source_dir
    cp file_1 file_2 file_3 ~/target_dir

On the other side of the ring, if you were to do this with a GUI, you would navigate to the source folder in your explorer/nautilus/whatever, select the three files, copy them to the clipboard, navigate to the the target folder and hit paste.

When you take the GUI route to copying files, you get to see a lot of stuff. Some of it may not be relevant to your current needs, but some of it might be. You will get to see all the other files in the source and target folders. You will get to see how much disk space all these files take. You will get to see any sibling folders that the source/target folder might have. You will get to see the file size of each of the files you’re copying.

When you take the CLI route to copying files, you’re not seeing anything except the next shell prompt waiting for your next command. Sure, if you got everything right, you wouldn’t really need to see anything, because your 3 files just got copied and you completed the task. So what do you care about seeing all the data you would have seen in the GUI route?

Because keeping your eyes open is important.

Maybe when looking around the source folder, you’d see that there was another file that you forgot to copy? Maybe you’ll see that for some reason the target folder has more files that it should, which might indicate a problem with your system? Maybe you’ll see that some impolite package-management tool that you previously ran left a bunch of build folders that you should delete?

When you take the GUI route, you’re exposed to all of that stuff. You’re more likely to catch possible problems, or have ideas about how to improve the system.

So the scoreboard for Visibility is:

GUI - Medium to High (depending on GUI quality)
CLI - Low to Medium (depending on CLI quality)

And when our task isn’t to copy files, but to write or debug a computer program, this point becomes crucial. There’s so much shit that’s helpful to see when you’re developing! If you just got an exception, you should see the running code, the entire stack, the values of all variables, documentation for the currently selected function, system I/O, and on and on. Each of these pieces of data might help you solve your problem in seconds instead of hours.

Now sure, you can obtain all of that data on the shell too. But having to explicitly ask for it by running a command is such a chore! And what more, every time you ask for more data, it pushes the previous data up and away, as if that data wasn’t important anymore!

Do you know what this reminds me of? It reminds me of old text-based games, where you would tell the computer “Go west”, “Look around”, “Fight monster” etc., and it would tell you things like “You have killed the monster” or “You are in a room with exits to the north and east.” Aside from the nostalgic feelings, playing a game like this totally sucks! You want to see how many exits the room has! You want to see the monster you’re killing, in all its gory detail! And this is true whether that monster is a grue or a segfault.

Speed

Now we’re getting to our second parameter: Speed. The opposite of speed, when it comes to computing tasks, can be called friction. If you have to click in seven different places to finish a task, then that’s a lot of friction, and that makes for a slow process and a sad user.

Speed is obviously a crucially important parameter. When you spend 8 hours a day doing tasks on the computer, the speed with which you do the tasks makes for a night-and-day difference. Sometimes you manage to shorten a 10-second task to a 2-second task, which is nothing short of an orgasm for a productivity-minded developer. But even cutting a task down from 3 seconds to 2 seconds is a considerable improvement. Remember— We’re sitting in front of that freaking computer all day long.

It’s important to note that speed is important not only because of the time you save, but because computer tasks are usually done in order to achieve some greater goal. That greater goal often requires a bunch of subtasks, and as each of these subtasks becomes more complicated and slow, you lose sight of your original goal. Doing tasks fast allows your train of thought to go faster.

Okay, so how do CLIs fare against GUIs in the parameter of speed?

Speed is the old standby of CLI-loving programmers. “CLIs are so must faster than GUIs! A GUI is so cumbersome, you have to click here and click there, navigate through menus… With a CLI, you just press a few keys, hit enter, and presto, it’s done!”

I beg to differ.

The scoreboard for Speed, which I will promptly explain, is:

GUI - Low to High (depending on GUI quality)
CLI - Medium to High (depending on CLI quality)

The CLIs’ score of “Medium to High” is, on average, better than the GUIs’ score of “Low to High”; but when it comes to software, where copying costs are minimal, the best program tends to make all the other alternatives extinct. So in the long run we care more about the best program rather than the average program, so GUIs and CLIs end up pretty close to each other on the all-important speed parameter.

“But wait,” the Vim fanboy cries, “how the hell could a GUI have anything close to a “high” speed rating?”

If you don’t know the answer to that question, I want you to really listen to what I’m about to say next, and try hard to remember it.

Because GUIs can be controlled with the keyboard!

Phew, that felt good! Now let’s explain.

The reason that programmers see GUIs as so much slower and more cumbersome than CLIs is that they try to operate them with the mouse. Working with the mouse is terribly, terribly slow. When you’re working with the mouse, you don’t have muscle memory. You always have to look at the screen for the next button or next menu item that you want to click. Not to mention that the mouse is more conducive to RSI than the keyboard, and this can be a serious health problem for heavy computer users.

To take a simple example: After you’ve learned to copy-paste by pressing Ctrl-C and Ctrl-V, you don’t need to think about them anymore; those movements become second nature. Your body and mind are good at memorizing these kinds of tasks, and before you know it you’ll be Ctrl-C‘ing and Ctrl-V‘ing without even thinking about it, like a pianist doesn’t even think about which piano keys he’s hitting. But if you were to try this with the mouse, it won’t work: You’ll always have to be on the lookout for the context menu after right-clicking, and then visually search for the “copy” item and click it… Friction friction friction.

Not to mention that when you’re using the mouse, you constantly have to move your right hand from the mouse to the keyboard and back, and that sucks.

And programmers often do tasks much more complicated than copy-pasting, making this point so much stronger.

So we all agree: Mice suck, keyboards rock!

But, but, but… A good GUI can be controlled with the keyboard! A good GUI can be operated completely with the keyboard, and has good keyboard shortcuts for the most common tasks.

For example, let’s look at a program I use everyday and has both a GUI and CLI interface: Git. Say I want to check-out a branch, something I do dozens of times a day. If I’d want to do that with the CLI, I’d type git checkout master. With autocompletion in place, I will have to make exactly 12 keypresses (namely, git chec{TAB}m{Tab}{Enter}.)

What about the GUI? Sure, if I’ll try using a mouse it will be an epic fail, but with the keyboard I can do {Ctrl-O}ma{Enter}{Enter}, which makes for exactly 6 keypresses. It’s twice as fast at the CLI version, i.e. half of the friction!

Now sure, some tasks can turn out to be faster on a GUI and others can be faster on a CLI, but when you pit the best GUIs against the best CLIs, they’re in the same league.

So why do programmers still prefer CLIs?

Now we can get back to the two items I mentioned at the beginning of the article, and expand on them. Those are the two main reasons that programmers prefer CLIs to GUIs:

Most GUIs suck. And when it comes to proper keyboard support, most GUIs suck terribly. It seems that most developers of GUI apps don’t care so much about keyboard-accessibility. It’s all too common to see desktop apps that simply cannot be controlled with the keyboard. Buttons that cannot be pressed, menus that cannot be opened, scrollbars that cannot be moved with the arrow keys.

This is annoying enough in mainstream software that’s mostly intended for non-technical people, but the astounding thing is that many open-source programs that are made for developers by developers are not keyboard-usable!

I find this completely shocking. The same programmers that preach about how Vim is so much more productive than an IDE because you get to use the keyboard, suddenly put on a different face when it’s time for them to make GUIs, for other programmers no less!
Most programmers are ignorant about good GUIs. Because some GUIs are good. Some GUIs have great keyboard support. Microsoft is one company that really shines in this regard. I challenge you to find one action in Windows, or in any other Microsoft product, that can be done with the mouse but not with the keyboard.

So good GUIs exist. They can be hard to find, but hey, a lot of good software is. Google Chrome has good keyboard support. Mac and Ubuntu have okay keyboard support. Wing IDE has good keyboard support. PuTTY has great keyboard support.

And of course, these GUI programs usually have much better visibility than their CLI counterparts, making them more productive programs overall.

Final words

I want to stress out that what I did here is a comparison between the very best GUIs and the very best CLIs. It takes time to find good GUI programs that can be operated with the keyboard. That’s tough. It takes time before you can surround yourself in excellent GUI tools to help you do your job. But it’s possible, and eventually, it’s the optimal way to work.

————————————————————————————————————

[1] - Actually there are some communities of programmers where most of the members like GUIs, but these are usually communities of developers that work in environments like Dot-Net and Java.

[2] - One parameter that is very important for non-hardcore users, and still somewhat important even for hardcore users, is Discoverability, i.e. being able to easily see all the different options and commands that a program gives you. GUIs obviously beat CLIs big-time in this parameter.

Quick riddle

Sun, 04 Sep 2011 22:12:00 +0300

Here’s a quick riddle:

You owe your friend two Dollars. You have a $5 bill, but your friend has only a $1 coin for change.

What would be a fair way to settle your debt that would not require you to get more change?

Answer: You toss the coin. If you win, you don’t give your friend anything. If your friend wins, you give him the $5 and he gives you back $1, so he gets $4. His expectation is $4 * 1/2 = $2, which is exactly fair.

Thanks for all the people who posted their answers in the comments!

A Git user's first (and hopefully last) foray into SVN-land

Tue, 30 Aug 2011 14:36:00 +0300

I belong to a rare breed of programmers: I was born and raised on Git. It’s the only version-control system that I’ve ever used. This is my personal story of how I had to use SVN temporarily for a contract project.

Now, I know there are a million of blog posts out there saying “Ooh, look how much better Git is than SVN!” I’m not trying to tell people something they don’t already know about Git, SVN, and the difference between them— I’ll just be telling my own personal story and my own impressions of SVN.

Beginnings

I started my programming comeback about 3 years ago. When I was a child, I programmed a lot— I was taught Basic in third grade, and I spent a lot of time of my elementary school and junior high making various little programs in C and Pascal. But I took a long hiatus from programming starting at about 10th grade, because I was frustrated with how many stupid technical details were involved in programming.

That hiatus ended about 3 years ago as I started learning Python, programming computer simulations for my Physics research, and about a year after that became a professional programmer. By the way, the main reason I chose Python is because Paul Graham mentioned it favorably in one of his essays.

Starting out with Git

How did I start out with version control? I saw the term “GitHub” being thrown around a lot on Hacker News. It seemed to be something everyone liked, so I figured I’ll try putting my project GarlicSim there. I learned Git. It took a while, because I was still not very technical, but eventually I got the hang of it enough to work in a single branch.

After a while, I understood what branches meant. I understood that I was able to create alternate versions of my code, work on them as much as I want, and decide at my leisure whether they should be merged into master and become an official part of my project.

I was happy. Branches are awesome.

I heard about SVN…

At some point I heard about SVN. I heard that it was something like a predecessor to Git.

What I imagined in my mind was that SVN was to Git what Windows 98 was to Windows XP. I imagined that SVN was just a crappier version of Git. Like, same idea, same workflow, just with more annoyances and less polish.

That perception changed when I got my first programming job.

First contact with SVN

Then I started programming professionally. I was hired by an Israeli startup to work on a Django project. I was kinda surprised to learn that they were using SVN. I figured that a technology startup would, you know, care about technology. But I said, “Meh, so I’ll just have to use Windows 98 for a while instead of XP, I could live with that.”

Oh boy.

So I was just about to learn SVN. This is a dramatization of the conversation I had with that company’s chief developer: (I ask you to read the text in italics in John Cleese’s famous surprised voice.)

Me: Hey man, I wanna make a small change to the code to add [Feature X], I’ll just make a little branch, alright?

Chief developer: What?! You’re gonna make a branch?!

Me (taken back): Eh, yeah, a branch. So, you know, I could make changes in it without affecting our master, or trunk, or whatever?

Chief developer: And how are we supposed to put the changes back into our trunk?!

Me (totally freaked out at this point): Eh, merge them back?

Chief developer: Merge? MERGE?! That’s a mighty big word for someone your size!

And then followed a conversation in which he explained to me that branching-n-merging in SVN is really fucking complicated. I don’t remember exactly what he said was required, but it changed my perception of SVN from:

“SVN is a slightly crappier version of Git.”

To:

“SVN is a version of Git in which every time you want to start a branch or merge it back, you have to sacrifice a young virgin to the source-control gods.”

I was not happy.

Luckily, the chief developer was considerate of me, and the company was small enough (only a handful of people), so he agreed that we’ll just move our entire product to Git.

I was saved at last moment from the atrocities of SVN.

Until now.

SVN strikes back

Recently I got a nice contract job. And, like any other contract job after my initial trauma with SVN, I made sure to ask that the job was in Git before I started the job. Unfortunately, despite being told that the job was in Git, it turned out that the job was in SVN, and that we’d move to Git “after the deadline.”

I was not happy. It turned out that I had to learn SVN after all.

I had my brother give me a crash course on SVN via phone. My brother’s a C++/Java programmer, and he had to use SVN for a while, so he agreed to help me with it. He has experience with Git so he was in a good position to explain the differences between the two. I primarily remember him explaining to me about branches.

Now, I remember seeing the “branches” folder in the SVN repo. I thought it was pretty weird to keep branches in a folder, but whatever. I assumed that the stuff in the “branches” folder was like the stuff in the “.git” folder in Git: Magic data that contains all your branches and allows you to switch to them at will.

Then my brother explained to me that this is not the case. I remember this part of our phone conversation pretty well, because at that point I was sitting on my knees on the carpet, with my head in my hands, looking down into the seat of my office chair. He explained that SVN doesn’t really have branches, it’s just a folder called “branches” that you manually copy your code into.

I was not happy.

Though, on the other hand, it was kind of a relief; it was relieving to know that SVN doesn’t even try to do branches. Because God knows how badly they would fuck it up if they would have tried to. Copy-pasting folders is major bullshit, but at the very least it’s predictable; when you use mv or cp, you know exactly what you’re getting, which is something you can’t say about svn.

That partial relief nonwithstanding, the whole “branches” thing in SVN is utterly ridiculous. In the past when people told me “SVN has branches too!” I knew that something wasn’t right and I just assumed that SVN branches suck, but now with this new knowledge I can answer in confidence, “No, SVN does not have branches, it merely has a folder called “branches”, which is a far cry from having actual branches.”

You know what SVN’s “branches” folder is like? It’s like taking an A4 paper from your printer, drawing a Flux Capacitor on it, hanging it in your car and declaring with a satisfied grin, “My car’s a DeLorean time machine! I can go back in time!”

SVN Ignore

One other thing I had to deal with in SVN was doing “SVN ignore” (I don’t know the SVN jargon for it.) Basically I was looking for the SVN equivalent of .gitignore: A list of file patterns that do not get source-controlled, ever. This is very useful for build products, files containing passwords, user-uploaded static content, etc.

Now in Git, you just have a file called .gitignore in the root of your repo, and in that file you write the file patterns that should be git-ignored, like *.pyc or build. You save the file, and that’s it. I assumed that there would be a similar mechanism in SVN.

You can probably guess where this is going by now.

It turned out that SVN’s ignoring mechanism is far more complicated than that. It basically holds a list of SVN-ignored file for every, single, folder in your repo, and there is no convenient way to edit all these rules for the enitre repo, or even just view them. You have to run this baffling command in order to even view the ignore rules. Also, when you edit the ignore rules for a folder, you’re not just editing a text file like in Git, you have to actually set the rules using svn. I tried actually going into the dreaded .svn folder and edit the text file that holds the ignore rules myself, but it was encoded in a really stupid encoding, and in order to write rules there manually I would have had to count the number of characters in my ruleset and type it as part of the text file.

Back to Git

Eventually my new client saw how bad of an effect SVN was having on me, and he let me move us to Git. So over the weekend I used svn2git to convert our repo to Git and now we manage all our code in Git. Uninstalling TortoiseSVN from my machine was quite a happy moment.

Goodbye SVN, may we never meet again.

Postponing GarlicSim 0.7 yet again

Mon, 08 Aug 2011 23:23:31 +0300

Yeah, the deadline for GarlicSim 0.7 has been an epic fail. A couple of weeks ago I thought I had it under control, but then I got a contract job that’s been taking all my time. Good news for mah bank account, bad news for mah open-source project.

So I’m postponing the 0.7 release to November 1st, 2011. See you then.

Postponing GarlicSim 0.7 again

Wed, 20 Jul 2011 20:43:57 +0300

I’m postponing the deadline of GarlicSim 0.7 again, this time to August 20th.

This is fucked up and I feel really bad about this. I should have been more realistic about my deadlines, but I’ve let my optimism have the better of me. I’ll try to be more accurate in future deadlines.

Why making a cool project is a good idea for an aspiring software developer

Sun, 17 Jul 2011 00:34:31 +0300

As a software developer, I get to know a lot of other software developers. Many of the software developers I know are either running a startup, working as an early employee for a startup, doing contract jobs for a startup, or dreaming of one day starting a startup. My point is that software developers are very interested in startups, either starting them or working for them.

My advice to anyone who’s interested in startups: Before you ever start a startup, try to make at least one cool and useful project. It doesn’t have to make money; in fact, it’s better if your project doesn’t even try to make any kind of profit.

Why is making a cool project important?

When you’re doing a startup, you essentially have to do two things at once:

Create some kind of product or solution that answers your users’ needs; this is usually the software ingredient of your startup; and
Create a business around that product that brings in money for your company.

Both of these things are difficult. Really difficult. It’s easy to mess either of these up. And you have to do them both at the same time. Get one right but mess the other one up, and you still failed.

Here’s a metaphor: Starting a startup is like juggling while riding a unicycle. Riding a unicycle is hard. Juggling is hard. Doing them both at the same time? Really freaking hard.

If your goal in life was to juggle while riding a unicycle, would you get on the unicycle while holding a few juggling clubs and try to juggle while riding the unicycle? Not if you have any common sense. You would first try to ride the unicycle. You’d take your time until you’re sufficiently skilled with that. Ditto for juggling. Then you’d carefully try to do both of them at once.

The same applies to startups. Making a successful product is hard enough; it’s all too common for companies to invest a lot of resources in making an ambitious product, and when it’s launched it’s just “meh”.

Building a business on top of a product is hard as well. There are many places where you can screw it up, both by making technical mistakes or strategic ones.

As you can guess, the chance of you messing either the software side or the business side is very big.

Start a project, don’t try to make money.

That’s the essence of my advice. Start a project which is free of the burden of profit. Sure, you won’t get rich off of it, and you’re gonna have to support yourself financially while you work on it, but it’s your best chance at learning how to make a successful product— It definitely has a much better ROI than going to college.

Not being burdened with having to make a profit will make it easier for you to build a successful product. You’d never have to think about limiting or exploiting your users, which is something that many software businesses have to do. Think about those crippled “starter” versions of Windows that sell for cheap, or DRM, or annoying ads on websites, or desktop software that installs annoying toolbars and spyware. These are all attempts to make money by limiting or exploiting users. Software needs to act like a butler, which is difficult enough; imagine how hard it is to have a good butler who also tries to upsell you some crap as he serves you.

As a person who’s making a project that doesn’t aim to make money, you’ll be free from all of those dangerous practices.

Making a useful product is still hard, mind you, and you still have a good chance of failing; but at least when you crash from the unicycle and try to pick yourself up, you won’t have juggling clubs falling on your head.

`SimpackSelectionDialog` almost done

Tue, 12 Jul 2011 01:13:00 +0300

Hey, only a few weeks remain until the release of GarlicSim 0.7. I just wanted to give you a sneak peak of the new SimpackSelectionDialog, incidentally on a Mac:

(Click to enlarge.)

I’m quite pleased with this widget. It’s almost finished, I might put in a few more touches before the release. The above screenshot shows it on a Mac, but as always, it’s completely cross-platform and works on Windows XP, Windows 7 and Linux as well.

This time I put in more emphasis on keyboard-accessibility, including accelerators for as many buttons as possible. The widget is not 100% keyboard accessible, but I’d say around 90%, and it will get to 100% in a couple of years as the underlying wxPython framework evolves. I also went through all the other dialogs in GarlicSim and gave them as many accelerators as possible.

Going back to work on 0.7!

DreamPie: My favorite Python shell

Sat, 11 Jun 2011 23:56:00 +0300

If you’re not familiar with it, you should really check out DreamPie, which is a graphical Python shell.

It’s a really well-done GUI shell, with great keyboard support to boot. Its most unique feature is the separation of the “history area” from the “current command area”. I think this design makes more sense than the traditional approach of having them together. It’s also notable that the ”current command area” can be used to enter multi-line Python commands (like defining a function or a class) and it’s very easy to move between lines to edit them. (This sounds obvious but many other shells make this kind of editing cumbersome.) The autocompletion is also well done; for example, DreamPie is able to autocomplete modules even if they haven’t been imported yet.

On top of that, I was delighted to find that it works on PyPy out of the box! Check it out:

Why don't people use enum in Python?

Fri, 10 Jun 2011 20:25:00 +0300

A while ago I was introduced to the concept of enum in Java. I never programmed Java, but this looks like the solution for when you want a variable to hold a value from a limited set of possible values. What is usually done in Python is either using a string or an index number, both of which are less elegant solutions in my opinion.

Python doesn’t provide a built-in enum type, but I found a few third-party modules, out of which I chose flufl.enum as being the most promising:

http://packages.python.org/flufl.enum/docs/using.html#creating-an-enum

This looks like a pretty good solution. A tad less native than in Java, but still much better than strings or ints.

So I wonder: Why do I never see Python code that uses flufl.enum, or any other enum package for that matter? Is there some reason not to use enums in Python? Perhaps a large portion of Python programmers are ignorant about enums, (as I was before I saw it recently?) Or perhaps people just don’t really care about making their code elegant?

Anyway I’ll try to start using flufl.enum soon. One awesome application that this could have is to make a Django EnumField (or ChoiceField or whatever you want to call it) instead of the current ugly way of making an IntegerField and defining CHOICES separately.

Nullege: Search engine for Python source code

Wed, 18 May 2011 00:41:00 +0300

Nullege is a service I’ve been using for a while. It’s a search engine for Python code. I’m quite surprised that I don’t hear much talk about it in Python-related blogs; it’s a pretty useful tool:

It shows you info about the class/function/whatever else you were searching for, and also places where it’s used in various Python projects found on the web.

Check it out: http://nullege.com.

Is something blocking you from moving to Python 2.7?

Mon, 16 May 2011 13:45:00 +0300

I didn’t move to Python 2.7 yet. That is, I don’t consider it my main Python version. My main version is Python 2.6. I consider Python 2.5 the past, Python 2.6 the present, Python 2.7 the future, and Python 3.x the distant future. When I package GarlicSim for end-user distribution I do it with Python 2.6.

Now, I’m a big fan of using the latest version of Python, in order to both take advantage of new features and bugfixes, and also to help advance the entire Python community to more advanced versions. If we all adopted new versions of Python as soon as possible, and ensured our packages and services support them, it would be much easier for our fellow Python developers to upgrade as well. This is especially true for Python 2.7 which serves as a bridge to the holy grail, Python 3.x.

I try to do my part: GarlicSim supports Python 2.5, 2.6, 2.7, 3.1 and 3.2. But unfortunately it’s sometimes hard to move to new versions because existing technologies don’t support them yet.

For me the blocker is Psyco. Psyco’s JIT compiler speeds up my code by several times. It’s really awesome. But Psyco doesn’t support Python 2.7, and it’s pretty much the reason that I don’t move to Python 2.7. It seems that all development effort is focused on PyPy, and while PyPy is a super-awesome project, you can’t really use it if you use C extensions. (Unless you’re willing to dig into cpyext.) The cool thing about Psyco is that you just import psyco; psyco.full() and your code becomes faster, regardless of which third-party packages you use. So for me upgrading to Python 2.7 means making my program several times slower, which is a price I’m not willing to pay.

So Psyco is what’s blocking me from using Python 2.7. What’s blocking you?

GarlicSim 0.7 release's scope expanded, deadline postponed

Fri, 13 May 2011 12:51:00 +0300

Yeah, I’m postponing the deadline for GarlicSim 0.7 again. The new deadline for GarlicSim 0.7 is August 1st, 2011. (The old deadline was ~~June 1st, 2011~~.)

The good news is that I’m adding a few more features besides those I promised in my original plans:

I’m vastly improving the SimpackSelectionDialog widget. This is what I’m trying to do with it:

(Thanks for the awesome mockup tool, Balsamiq!)
GarlicSim 0.7 is going to be PyPy-compatible! PyPy is a very exciting Python implementation project and it has a good chance of becoming the future of Python. And I want to be sure from the start that GarlicSim can be part of that future. (Do note that there is no wxPython installer available for PyPy yet, so I wasn’t able to ensure that the GarlicSim GUI works on PyPy; I only ensured the core logic does and that all its tests pass.)

Despite these added features, I feel kind of bad about postponing the release. I should have been more conservative with the deadline. This will definitely be a lesson for future GarlicSim releases. I’ll try to set more realistic deadlines from now on.

GarlicSim continuous-integration website, managed by ShiningPanda

Wed, 27 Apr 2011 17:54:00 +0300

Check out the new “GarlicSim status” website:

http://status.garlicsim.org

It’s a Jenkins continuous-integration server which automatically pulls the most recent version of GarlicSim from the GitHub repo, runs the tests, measures line coverage, and displays all the results on the web.

It runs on ShiningPanda, which abstracts away a lot of the technical work involved in running a Jenkins server. ShiningPanda is still in beta, but it’s already useful for me; this has been a lot easier than my previous attempt at running Jenkins on EC2. I also love the fact that ShiningPanda is focused on Python and Python-related tools; to contrast, I remember how frustrated I was when I was trying to use CloudBees for the same task and discovered that they don’t even have nose installed…

Anyway, enjoy the new CI server!

New Wing 4 script: `instantiate`

Tue, 26 Apr 2011 15:47:57 +0300

Hey, I made a new script for Wing IDE 4. Users of Wing may want to check it out.

It’s called instantiate.

What it does is simple. Often you want to instantiate an object from a class like this:

cat_nip = CatNip()

The annoying thing about typing this is that you don’t get autocompletion for cat_nip because it doesn’t exist yet. But you do usually get autocompletion for CatNip. Hence this script: You type CatNip, execute the script, and then automatically get this line:

cat_nip = CatNip()

With the cursor already placed inside the parentheses for you to type the arguments.

The source is on GitHub:

https://github.com/cool-RR/cute-wing-stuff

Enjoy!

2 out of 3 GitHub forks are completely empty

Sat, 09 Apr 2011 21:25:00 +0300

About 2 years ago I started the GarlicSim project on GitHub. GitHub’s most famous feature is “forking”: When another person forks your code, they create their own copy which they may work on and change the code, and later show you their changes (a.k.a. “pull request”) so you could merge those changes into your own repo. True open-source spirit.

GarlicSim was forked 4 times in the 2 years of its existence. I remember the first time someone forked GarlicSim. I was very excited. I thought someone was going to help me work on it. I asked that person what he was thinking of adding. He didn’t respond to my messages. He didn’t add any code to his fork. Neither did the 3 other forks of GarlicSim that followed.

Then it became clear to me that GitHub forks don’t mean much. I guess most people just click “fork” because they think it’s cool, or because they would like to imagine themselves working on the project. But they don’t actually do any development on their forks.

I wanted to check whether this phenomenon happens on other peoples’ repos too; maybe I was just unlucky with GarlicSim? Apparently not. My small research showed that 2 out of 3 GitHub forks are completely empty.

My modest research

I checked 12 random GitHub projects which have at least one fork, not including the original repo. I used GitHub’s own “Random repository” link to find them, and pruned through all the fork-less repos (more than a hundred) until I got my 12. Then I counted how many forks each of these repos had; the smallest had just one fork, the biggest had 8 forks. (Again, not including the original repo.) The median number of forks was 1. The total number of forks across the 12 projects was 27.

Then I used Github’s excellent “Network” view to see how many of these repositories actually had any commits not present in the original repo. Only 8 out of the 27 forks had any content in them.

To put it simply: 2 out of every 3 forks in GitHub are completely empty. Their owners did not add even a single trivial line of code to the project, not even just for playing around in their own fork.

And of course, regarding those ~33% of forks which actually contains code, I did not check how many of these have trivial code and how many have substantial code. It’s quite possible that a big share of them just add a few trivial lines that never get merged into the original project.

So keep that in mind next time you browse Github and see a project with 20 forks. There aren’t really 20 active forks, there are perhaps 7 active forks, and even that is being optimistic.

(And I’d like to note: I love GitHub with all my heart. It’s a great tool and I use it both in my OSS work and my job. I just want people to have a realistic view of what a GitHub fork means.)

GarlicSim 0.6.3 released, becomes fully open-source!

Fri, 08 Apr 2011 01:27:49 +0300

I have big news. I’ve decided to release the entire code of GarlicSim as open-source under the LGPL license. Previously, only garlicsim and garlicsim_lib were released under the LGPL, and the garlicsim_wx GUI was closed-source. Now I’ve released garlicsim_wx under the LGPL as well, so the GarlicSim project is completely open source. Yay!

I’ve made a new release, GarlicSim 0.6.3, which clearly states that garlicsim_wx is now released under the LGPL. I also made a few minor additions in this release:

What’s new in GarlicSim 0.6.3

garlicsim_wx gets released as open-source under the LGPL license.
Most of the sample simpacks in garlicsim_lib have been rewritten to be more elegant and well-documented.
NumPy is no longer needed in order to run the garlicsim_lib.simpacks.queue simpack.
A class garlicsim.general_misc.identities.HasIdentity was added, to be used as a base class for every object that needs to maintain its identity across simulation states.
A bunch of fixes, more tests, and more documentation.

Go on and install the new version!

The next revolution after Git

Tue, 05 Apr 2011 13:29:00 +0300

(Note: I talk a lot about Git in this essay, but the same thing can probably be said about Mercurial, Bazaar, et al. I simply use Git because that’s the system that I’m personally familiar with.)

I’ve been a Git user for a few years now. I love it. Being able to branch-n-merge freely is a very powerful tool for anyone who needs to manage a codebase. But I find that there is one area where Git falls short; in this essay I will make a humble first attempt at designing a system that tries to solve this problem. I hope that someone will pick up the glove and create this project. If anyone implements this well, I would personally consider it to be the next revolution after Git; i.e., The difference in productivity between this system and Git would be on the same scale as the difference between Git and SVN.

The missing feature is: Templating.

The problem

In a codebase you often have code snippets that repeat themselves a few times in different places. For example you have a code file chair.py that defines a class Chair, and another file bar_stool.py that defines a class BarStool, and they share a lot of code, but they have a few small differences between them. You don’t want your source-control to store two copies of the same code; not because you’re cheap on hard-disk space, but because every time you’ll want to change the shared code, you’ll have to change it in two (or more) places. This is called the DRY principle, “Don’t repeat yourself”, and it’s a very good principle.

I know what you’re thinking. “You should refactor the Chair and BarStool classes so that the shared code will be put in a base class!” Sure, if you can. I’m personally a refactoring fanatic. I refactor everything that moves. But sometimes you just can’t refactor. Sometimes you’re completely unable to refactor, and sometimes you are able to refactor but it’s just not practical, because it will make your code too dynamic and introduce too much indirection.[1]

GarlicSim has several examples of situations in which you have repeating code that would be impractical to refactor. A good one is the setup.py files. GarlicSim is comprised of three Python packages: garlicsim, garlicsim_lib and garlicsim_wx. Like all Python packages, they each have a setup.py file: garlicsim/setup.py, garlicsim_lib/setup.py and garlicsim_wx/setup.py. As you can see, these files have many lines in common with each other. But there is no practical way to refactor those identical parts away.

The solution, in the abstract

The solution is templating. Instead of manipulating the code files directly, you manipulate succinct descriptions of the code files. So instead of maintaining three setup.py files, I would have to maintain one generic setup.py template, and then describe each of the three actual setup.py files as deviations from that template.

Any programmer worth his salt would feel in his guts that templating is “the correct solution,” that it’s the way things should be. But then, almost no open-source project uses such a templating scheme to maintain their code. Why?

The solution needs to be comfortable to work with

There’s a big difference between solving a problem in the abstract and solving it in practice. A templating system is indeed “the correct solution,” but that’s not enough to make it a solution worthy of being used.

Imagine if I tried to use a templating system, say Jinja, to produce my setup.py files. I would have a general template, and then three descriptions of how each setup.py file deviates from the templates. And then I would have to generate the actual setup.py files from the templates.

That would be annoying. One of the big issues is that you can’t edit the generated code. I’ve often seen messages like:

# This file was created automatically by the templating system.
# Don't modify this file, modify the templating interface instead.

That really sucks. It sucks to be forced to edit your template files instead of the code files. Your IDE just can’t grok the template as a source file, so it loses all intelligence features. The same thing can be said about the developer himself, actually… You always want to be able to view and edit your source files directly.

A design that might work

Here’s a design that might work. Let’s codename it “Leapfrog”, just for the sake of easy reference.

This is a design that works on top of Git, or on top of any other source-control system. Now, when you have a Git repo, Git saves all its data in a hidden .git folder in your project’s root folder. We are going to do something similar; we’ll save all of our data in a .leapfrog folder that will sit in the root folder alongside the .git folder.

The .leapfrog folder will contain all the templates and all the data needed to generate your project’s source files. The .leapfrog folder will be the only thing tracked by Git; your actual source files will be git-ignored.

So the contents of your .leapfrog folder may look like this:

.leapfrog/
    setup.py.abstract
	garlicsim/
	    setup.py
	garlicsim_lib/
	    setup.py
	garlicsim_wx/
	    setup.py

Where setup.py.abstract is a template, and each setup.py file extends that template. This is all the information that is needed to generate the actual setup.py files in the so-called “working directory.” (The template format can be in Jinja or whatever.)

Now, in order to synchronize the working directory with the .leapfrog directory, you’ll have an interface quite similar to Git. A leapfrog checkout action would generate the code files from the templates. But the powerful feature would be that you could go the other way too; you could edit the code files and then “stage” that to the templates.

For example, you could edit garlicim_lib/setup.py and add a bunch of lines. Then you would run leapfrog add . to stage, just like you do in Git. The changes will be added to the .leapfrog/garlicim_lib/setup.py template. Then you could use git add . and git commit to stage and commit the changes to the template to Git.

Pre-rebuttal

Probably some people will say, “there are many existing templating systems out there that you can use!” To which I can give the Drew Houston reply:

You: There are a million of templating systems out there!
Me: Do you use any of them to manage your codebase?
You: No.
Me: …

Final words

There are probably mistakes in my design. There are probably problems I didn’t think of. Good chances are that the eventual implementation will look nothing like the design I described. That’s okay, this is how these things work. I just hope to get the ball rolling for people to start thinking about this problem and come up with solutions.

I really hope that some open-source developer out there will read this essay and implement it, or implement his own take on it, so we could all benefit from this tool.

————————————————

Notes

[1] I personally remember one contract Django job in which I had a few repeating code segments in a few classes. I refactored them into one base class. But that architecture was just not meant to be. It was too dynamic; I was creating class attributes dynamically and doing other nasty stuff. Maybe in LISP one would be able to come up with an elegant refactoring for that specific problem, but not in Python. That refactoring was a mistake on my part that cost my client a bit of money.

Thanks to Amir Rachum for helping me brainstorm on this idea.