Tuesday, July 24, 2012

Gigabit Hack Weekend

I went to a gigabit hacking event this past weekend hosted by Mozilla Ignite at the Internet Archive's building. I got to meet some interesting people and had a great time.

The general theme of the event was to try and demonstrate what kind of applications we can build when people's internet connections get really fast (e.g. gigabit). Since I work on federated 3D repositories with both the Sirikata project's Open3DHub, and also OurBricks, I wanted to  showcase applications that can be built with online 3D repositories and fast connections.

3D models can be quite big. Games usually ship a big DVD full of content or make you download several gigabytes worth of content before you can start playing. In contrast, putting 3D applications on the web demands low-latency start times. To showcase online 3D, I took the awesome ThreeFab project and added the ability to load files from an external repository.

For a demo:

  1. Visit http://blackjk3.github.com/threefab/.
  2. Click the "Import" button.
  3. Pick a 3D model from http://open3dhub.com/ and copy the "Direct Download" link box.
  4. Paste the link into the Import box.
  5. The model should load into the editor. (most models work)
The ThreeFab editor lets you export the scene you make. Here's a demo of a scene I created this weekend running on jsfiddle:

What's important to remember there is that all of the 3D content is being loaded asynchronously in the background directly from Open3DHub. The editor supports URLs from any site as long as they support CORS headers. As a result of the weekend, the Internet Archive is working on enabling these headers for its content, so hopefully we will be able to load 3D content directly from the archive soon.

Thanks to Mozilla and the Internet Archive for putting on a fun event!

Thursday, July 19, 2012

Automatic Debug Shell in Python

I've been working on a few long-running Python scripts lately. I find debugging to be difficult when a script takes a long time to run, since you don't get any feedback until execution hits the point in the code you're working on.

Python's awesome pdb module lets you drop into an interactive shell by calling the set_trace function. This is really useful because you can inspect local variables from the interactive shell. I found it tedious, though, to keep inserting these statements into my code, hoping it was in the right place, and wasting time when an exception was thrown somewhere other than what I was expecting.

I found a nice recipe that lets you drop into interactive mode as soon as an unhandled exception occurs. I created a gist that incorporates some changes from the comments. It only enters the interactive shell if the script has an appropriate TTY session, and if the exception is not a SyntaxError.

Here's an example script showing how to use it:
import debug

def what():
   x = 3
   raise NotImplementedError()

if __name__ == '__main__':

And an example of using it:
$ python test.py 
Traceback (most recent call last):
  File "test.py", line 8, in <module>
  File "test.py", line 5, in what
    raise NotImplementedError()

> /home/jterrace/python-exception-debug/test.py(5)what()
-> raise NotImplementedError()
(Pdb) x

This is really nice for debugging, because it drops you to a shell that has all the local variables when the exception occurred.

Yang pointed out the ipdb module. It looks awesome as an alternative to pdb and it comes with a method for doing this automatically.

Seamless Sharing

I uploaded an album yesterday of my recent trip to the ICME 2012 conference in Melbourne, Australia to Google+. I wanted to share a link to the album to Facebook, so I posted a link to the album. I was surprised that nothing popped up on Facebook for the preview, so I ran some experiments.

I created a public album on Google+, a public album on Facebook, and posted a public tweet with an image attached. I then shared each one on the other two social networks. I was surprised that the only one that actually showed content was sharing a tweet to Google+ and Facebook. Here's a grid showing all the results:

Comparison of sharing a photo/album across different social network. The rows are the source and the columns are the destination.
What's interesting is that other things do work. For example, I shared a link to a Flickr album, and it shows the content across all three networks. I'm guessing that maybe Facebook and Google+ don't have the proper Open Graph tags for the other networks to parse, but this really should be a seamless experience across networks.

Wednesday, July 11, 2012

Xvfb Memory Leak Workaround

I've been using Xvfb, the X virtual frame buffer, for a few projects. Xvfb allows you to run applications that require a display, without actually having a graphics card or a screen. I started noticing that the resident memory used by the Xvfb process would continually go up. I could literally run processes that connect to the display in a loop and watch the memory grow. Clearly, there was some kind of memory leak going on.

I finally found a workaround to preventing this memory issue. If you add -noreset to the argument of Xvfb, the memory issue vanishes. By default, when the last client to Xvfb disconnects, the server resets itself. My guess is that this works fine when video memory is backed by a hardware device, but Xvfb has a bug where it doesn't free the memory it allocated for the buffer. By setting the noreset option, the server no longer restarts and the memory isn't lost.

Here's my Xvfb command line:
Xvfb :1 -screen 0 1024x768x24 -ac +extension GLX +render -noreset

An explanation of the other arguments:

  • :1 - Runs the server on "display 1". Most X servers run on display 0 by default.
  • -screen 0 1024x768x24 - Sets screen 0 (the default screen) to a resolution of 1024x768 with 24 bits per pixel.
  • -ac - Disables access control because X access control is incredibly painful to get right, and since the server is usually only accessible from localhost, it's not a big deal.
  • +extension GLX - Enables the OpenGL extension, allowing graphics programs that use OpenGL to work inside the virtual display.
  • +render - Enables the X Rendering extension, enabling advanced image compositing features that most applications will need.
I also use a simple Xvfb start script. It's available as a gist.