tag:blogger.com,1999:blog-23067335092093146812024-03-13T14:15:59.949-04:00Programming SanityMy random notes on programming and the internets.Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-2306733509209314681.post-4311243734030139112023-12-10T16:04:00.002-05:002023-12-10T16:46:09.167-05:00Google Calendar E Ink Display<div class="separator" style="clear: both;"><span style="font-family: arial;">I've always wanted to display our family calendar in a central location, like in the kitchen. Various options exist, but a powered display really limits where you can place something, and I never liked the way it would look.</span></div><div class="separator" style="clear: both;"><span style="font-family: arial;"><br /></span></div><div class="separator" style="clear: both;"><span style="font-family: arial;">When I saw the <a href="https://openepaperlink.de/" target="_blank">OpenEPaperLink</a> project, I had to try it out. Here's the end result:</span></div><div class="separator" style="clear: both;"><span style="font-family: arial;"><br /></span></div><div class="separator" style="clear: both;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIbcD8gsC5wECThvDFyEHroVs3Qpvxg-XA8IMZyNxUcOGWCJvM1YTlLYMBsk9eFrKlOcH3eeBYGqZEgGK6_fdKMnFqtLNEuCrnMrMIvFWiYsQmn4aqAdCk08CXMA_3zXFj3SjeL9LdNxJRR0qMItuXo5IgYuVQKTBfobvtkvxNI1Qu2lWQp1VLDNsJ6NwB/s4366/PXL_20231210_200857684.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3472" data-original-width="4366" height="254" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIbcD8gsC5wECThvDFyEHroVs3Qpvxg-XA8IMZyNxUcOGWCJvM1YTlLYMBsk9eFrKlOcH3eeBYGqZEgGK6_fdKMnFqtLNEuCrnMrMIvFWiYsQmn4aqAdCk08CXMA_3zXFj3SjeL9LdNxJRR0qMItuXo5IgYuVQKTBfobvtkvxNI1Qu2lWQp1VLDNsJ6NwB/s320/PXL_20231210_200857684.jpg" width="320" /></a></div><br /><span style="font-family: arial;">The hardware I purchased for this:</span></div><div class="separator" style="clear: both;"><ul style="text-align: left;"><li><span style="font-family: arial;"><a href="https://www.tindie.com/products/electronics-by-nic/openepaperlink-mini-ap-v3-zigbee-wifi-gateway/" target="_blank">OpenEPaperLink Mini-AP v3</a> as a base station</span></li><li><span style="font-family: arial;"><a href="https://www.tindie.com/products/electronics-by-nic/42-epaper-tag-for-openepaperlink/" target="_blank">4.2" epaper tag</a> for the screen</span></li></ul><div><span style="font-family: arial;">These things are surprisingly affordable for what they offer. The rest of the project is all software, primarily <a href="https://www.home-assistant.io/" target="_blank">Home Assistant</a>. The more I've used Home Assistant, the more I'm impressed with the amazing community surrounding it.</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">The things I had to install in Home Assistant:</span></div><div><ul style="text-align: left;"><li><span style="font-family: arial;"><a href="https://www.home-assistant.io/integrations/google" target="_blank">Google Calendar home assistant integration</a>: this allows you to pull information from your Google Calendar, which imports it to home assistant as <a href="https://developers.home-assistant.io/docs/core/entity/calendar/" target="_blank">Calendar entities</a>.</span></li><li><span style="font-family: arial;"><a href="https://github.com/jonasniesner/open_epaper_link_homeassistant" target="_blank">OpenEPaperLink home assistant integration</a>: this detects the OpenEPaperLink hub on your network and adds devices for each of the displays. It then exposes a service call you can use to send data.</span></li><li><span style="font-family: arial;"><a href="https://github.com/hassio-addons/addon-node-red" target="_blank">Home Assistant Node-RED</a>: this integrates <a href="https://nodered.org/" target="_blank">Node-RED</a> into home assistant, which is a flow-based programming interface. It's much more flexible than Home Assistant's built-in yaml-based routines. In particular, it has the ability to execute JavaScript code during a flow, which is necessary to transform the data (more on this later).</span></li></ul><div><span style="font-family: arial;">Once this was all set up and configured, I could create the Node-RED flow. Here's what the overall flow looks like:</span></div><div><span style="font-family: arial;"><br /></span></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYbGAzrRegzZsPRM0IqGKfry7sPqKq2GMjiF88Ag2656VyYCJhhMdvP9MsnfkrASYPHIUwM1VU60mukQ7uI5yfpyZ3RKUj7fn7fSBhc-JU294x_aM-lWdYPGh_dTrYsMem9erXfvOcRj5OYk2_Z6POIpxBLs0HHvuKr03uZV3vjBpmV-czOc3M2KE-jvic/s639/Screenshot%202023-12-10%20164415.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="517" data-original-width="639" height="259" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYbGAzrRegzZsPRM0IqGKfry7sPqKq2GMjiF88Ag2656VyYCJhhMdvP9MsnfkrASYPHIUwM1VU60mukQ7uI5yfpyZ3RKUj7fn7fSBhc-JU294x_aM-lWdYPGh_dTrYsMem9erXfvOcRj5OYk2_Z6POIpxBLs0HHvuKr03uZV3vjBpmV-czOc3M2KE-jvic/s320/Screenshot%202023-12-10%20164415.png" width="320" /></a></div><br /><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">The first step in the flow is an <a href="https://nodered.org/docs/user-guide/nodes#inject" target="_blank">Inject node</a>, which is configured to trigger the flow once an hour.</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">The next step is a <a href="https://zachowj.github.io/node-red-contrib-home-assistant-websocket/node/call-service.html" target="_blank">call service node</a> which calls </span><span style="font-family: courier;">calendar.get_events</span><span style="font-family: arial;">. This is a new method, which replaces </span><a href="https://www.home-assistant.io/integrations/calendar/#service-calendarlist_events" target="_blank"><span style="font-family: courier;">calendar.list_events</span></a><span style="font-family: arial;">, which doesn't seem to be documented yet. The parameters I pass get all calendar events in the next 4 days, starting from 3 hours ago:</span></div></div><div><span><div style="font-family: arial;"></div><blockquote><div><span style="font-family: courier;">{</span></div><div><span style="font-family: courier;"> /* 3 hours ago */</span></div><div><span style="font-family: courier;"> "start_date_time": $fromMillis(</span></div></blockquote><blockquote><div><span style="font-family: courier;"> $millis() - 1000 * 60 * 60 * 3),</span></div></blockquote><blockquote><div><span style="font-family: courier;"> "duration": {"days": 4}</span></div><div><span style="font-family: courier;">}</span></div></blockquote><div style="font-family: arial;"></div><div style="font-family: arial;">The output gets piped into a <a href="https://nodered.org/docs/user-guide/writing-functions" target="_blank">function node</a>. This is really the only code I had to write for this. Instead of inlining it here, I placed it in a <a href="https://gist.github.com/jterrace/f412fdfac90687719cf3016ff968cc0b" target="_blank">gist here</a>.</div><div style="font-family: arial;"><br /></div><div><span style="font-family: arial;">The script takes the message payload output from the calendar service call, parses the dates and times, has some special handling for multi-day events, formats the dates and events in a custom format, and then transforms it into the output payload format expected by the </span><a href="https://github.com/jonasniesner/open_epaper_link_homeassistant/blob/main/docs/drawcustom/supported_types.md"><span style="font-family: courier;">open_epaper_link.drawcustom</span></a><span style="font-family: arial;"> service call.</span></div><div style="font-family: arial;"><br /></div><div><span style="font-family: arial;">Next, I wanted to prevent the display from refreshing unnecessarily. I piped the output of my function into a <a href="https://flowfuse.com/node-red/core-nodes/rbe/" target="_blank">Filter node</a>. This conveniently has a mode where you can stop the flow if the input doesn't change. E Ink devices only consume a lot of power when the display changes. However, the way the </span><span style="font-family: courier;">open_epaper_link.drawcustom</span><span style="font-family: arial;"> service call works is it renders into an image, which then gets sent to the display. If you send the same image twice, it still consumes power. To prevent refreshing the display when not necessary, the whole flow gets stopped by this filter node if what's being displayed doesn't change.</span></div><div><span style="font-family: arial;"><br /></span></div><div><span style="font-family: arial;">The next step is a call service node again, this time with </span><span style="font-family: courier;">open_epaper_link.drawcustom</span><span style="font-family: arial;"> as the target. The parameter this time is very simple, </span><span style="font-family: courier;">{"payload": payload}</span><span style="font-family: arial;">, since my function outputs in the payload format expected by the service call.</span></div><div style="font-family: arial;"><br /></div><div style="font-family: arial;">That's it! Now I have an auto updating E Ink display that shows my calendar. I bought a few more of these E Ink displays to play around with, so I'll hopefully be adding more!</div><div style="font-family: arial;"><br /></div></span></div></div>Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-23770771706741302032013-03-11T12:50:00.000-04:002018-03-26T23:59:09.559-04:00Vaurien: The Chaos ProxyI had a need to test out a program's behavior when its backend web server returned errors. Unit testing would have been difficult in this case, because I specifically wanted to test the program's macro result when encountering, say, 10% server response errors.<br />
<br />
After searching around, I came across <a href="http://vaurien.readthedocs.org/en/1.7/">Vaurien, the Chaos TCP Proxy</a>. It's seriously cool. As its name suggests, it's a TCP proxy server that you can route a local connection through to a backend server. In TCP mode, it can delay packets, insert bad data, or drop packets, testing the socks (<a href="http://en.wikipedia.org/wiki/SOCKS">pun intended</a>) off your application.<br />
<br />
It also supports the application-level protocols: HTTP, Memcache, MySQL, Redis, and SMTP, and its architecture is very modular, so it's easy to plug in new protocols.<br />
<br />
For example, here's how I can run it in HTTP mode:<br />
<pre class="brush:plain; gutter:false;">$ vaurien --protocol http --proxy 127.0.0.1:8888 \
--backend www.google.com:80 --behavior 50:error</pre>
This says to run an HTTP proxy that connects to google.com on the backend and return a 5xx HTTP error code 50% of the time. Testing it out with curl:
<pre class="brush:plain; gutter:false;">$ curl --head -H "Host: www.google.com" http://127.0.0.1:8888/
HTTP/1.1 200 OK
...
$ curl --head -H "Host: www.google.com" http://127.0.0.1:8888/
HTTP/1.1 500 Internal Server Error
Content-Type: text/html; charset=UTF-8
$ curl --head -H "Host: www.google.com" http://127.0.0.1:8888/
HTTP/1.1 502 Bad Gateway
Content-Type: text/html; charset=UTF-8
</pre>
<br />
Filed this away as a super useful tool.Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-55523459517442081652013-01-17T15:29:00.000-05:002013-01-17T16:31:39.928-05:00Git Branches by DateI tend to create a huge number of branches in my git repositories, but I have a bad habit of not cleaning them up once I'm finished with them.<br />
<br />
I found a nice command from an <a href="http://stackoverflow.com/a/5972362/624900">answer on StackOverflow</a> that allows you to sort branches by date. I modified it slightly to also show the date when printing the branch information:<br />
<br />
<pre class="brush:plain; gutter:false;">$ git for-each-ref --sort=-committerdate refs/heads/ --format='%(committerdate) %09 %(refname:short)'
Mon Jan 14 23:46:15 2013 +0000 keyfile-seek-rebase
Sat Jan 12 02:09:04 2013 +0000 perfdiag-mbit-fix
Fri Jan 11 17:38:13 2013 -0800 keyfile-seek
Thu Jan 10 01:05:43 2013 +0000 master
</pre>
<br />
You can also set the date format to be relative (or other possibilities, see <span style="font-family: "Courier New",Courier,monospace;">man git-for-each-ref</span>):<br />
<br />
<pre class="brush:plain; gutter:false;">$ git for-each-ref --sort=-committerdate refs/heads/ --format='%(committerdate:relative) %09 %(refname:short)'
3 days ago keyfile-seek-rebase
6 days ago perfdiag-mbit-fix
6 days ago keyfile-seek
8 days ago master
</pre>
<br />
I added it to my <span style="font-family: "Courier New",Courier,monospace;">.gitconfig</span> file as an alias:<br />
<pre class="brush:plain; gutter:false;">[alias]
branchdates = for-each-ref --sort=-committerdate refs/heads/ --format='%(committerdate:relative) %09 %(refname:short)'
</pre>
<br />
That allows me to just type <span style="font-family: "Courier New",Courier,monospace;">git branchdates</span> and get a nice listing of my local branches by date.Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-31712333895508436042013-01-14T15:39:00.001-05:002013-01-14T15:39:47.106-05:00Google Cloud Storage Signed URLsGoogle Cloud Storage has a feature called <a href="https://developers.google.com/storage/docs/accesscontrol#Signed-URLs">Signed URLs</a>, which allows you to use your private key file to authorize access to a specific operation to a third-party.<br />
<br />
Putting all the bits together to create a properly signed URL can be a bit tricky, so I wrote a Python example that we just open-sourced in a repository called <a href="https://github.com/GoogleCloudPlatform/storage-signedurls-python">storage-signedurls-python</a>. It demonstrates signing a PUT, GET, and DELETE request to Cloud Storage.<br />
<br />
The example uses the awesome <a href="http://docs.python-requests.org/en/latest/">requests</a> module for its HTTP operations and <a href="https://www.dlitz.net/software/pycrypto/">PyCrypto</a> for its RSA signing methods.Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-51226182102261517812012-12-17T13:04:00.000-05:002012-12-17T13:04:08.720-05:00Google Cloud Storage JSON ExampleI recently started working at Google on the <a href="https://cloud.google.com/products/cloud-storage">Google Cloud Storage</a> platform.<br />
<br />
I wanted to try out the experimental <a href="https://developers.google.com/storage/docs/json_api/">Cloud Storage JSON API</a>, and I was extremely excited to see that they support <a href="http://en.wikipedia.org/wiki/Cross-origin_resource_sharing">CORS</a> for all of their methods, and you can even <a href="https://developers.google.com/storage/docs/cross-origin">enable CORS on your own buckets and files</a>.<br />
<br />
To show the power of Cloud Storage + CORS, I released an <a href="https://github.com/GoogleCloudPlatform/storage-metabucket-javascript">example on GitHub</a> that uses HTML, JavaScript and CSS to display the contents of a bucket.<br />
<br />
<a href="http://jterrace-jstree.storage.googleapis.com/index.html">The demo</a> uses <a href="http://twitter.github.com/bootstrap/">Bootstrap</a>, <a href="http://jquery.com/">jQuery</a>, <a href="http://www.jstree.com/">jsTree</a>, and only about 200 lines of JavaScript that I wrote myself. The project is called Metabucket, because it's very "meta" - the demo is hosted in the same bucket that it displays!Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-51404635460579352882012-09-13T15:25:00.000-04:002012-09-13T15:25:14.289-04:00Bash Completion for Mac OS XI've been using my Mac more often lately, and I realized I was really missing the fancy bash completion features that Ubuntu has. Turns out, it's pretty easy to enable on Mac with Homebrew:<br />
<br />
<pre class="brush:bash">brew install bash-completion
</pre>
<br />
Then, as usual, homebrew tells you exactly what you have to do. Just add this to .bashrc:<br />
<br />
<pre class="brush:bash">if [ -f $(brew --prefix)/etc/bash_completion ]; then
. $(brew --prefix)/etc/bash_completion
fi</pre>
<br />
Now you get fancy completion. Here's me hitting tab twice with a git command:<br />
<br />
<pre class="brush:bash">$ git checkout rdiankov
rdiankov-master rdiankov/animation rdiankov/animation2
rdiankov/collada15spec rdiankov/master rdiankov/pypy</pre>
<br />
Awesome.<br />
<br />Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-83731124118512535422012-09-02T16:26:00.001-04:002012-09-02T16:26:57.669-04:00Comparing Image Comparison AlgorithmsAs part of my research into optimizing 3D content delivery for dynamic virtual worlds, I needed to compare two screenshots of a rendering of a scene and come up with an <i>objective</i> measure for how different the images are. For example, here are two images that I want to compare:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table><tbody>
<tr><td><a href="http://4.bp.blogspot.com/-U7w3iZ3hJ6M/UEO4KrYNSwI/AAAAAAAADIQ/KUYOujCPyIY/s1600/0003.03.tiff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="150" src="http://4.bp.blogspot.com/-U7w3iZ3hJ6M/UEO4KrYNSwI/AAAAAAAADIQ/KUYOujCPyIY/s200/0003.03.tiff.png" width="200" /></a>
</td>
<td><a href="http://4.bp.blogspot.com/-E51SPcToQAc/UEO4M2M62WI/AAAAAAAADIY/Qf6tkyObSE8/s1600/0003.03.tiff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="150" src="http://4.bp.blogspot.com/-E51SPcToQAc/UEO4M2M62WI/AAAAAAAADIY/Qf6tkyObSE8/s200/0003.03.tiff.png" width="200" /></a>
</td></tr>
</tbody></table>
<br />
As you can clearly see, the image on the left is missing some objects, and the texture for the terrain is a lower resolution. A human could look at this and notice the difference, but I needed a numerical value to measure <i>how</i> different the two images are.<br />
<br />
This is a problem that has been well-studied in the vision and graphics research communities. The most common algorithm is the <a href="http://en.wikipedia.org/wiki/Structural_similarity">Structural Similarity Image Metric</a> (SSIM), outlined in the 2004 paper <i><a href="https://ece.uwaterloo.ca/~z70wang/publications/ssim.pdf">Image Quality Assessment: From Error Visibility to Structural Similarity</a></i> by Wang, et al. The most common previous methods were <a href="http://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio">PNSR</a> and <a href="http://en.wikipedia.org/wiki/Root_mean_square_deviation">RMSE</a>, but those metrics didn't take into account the <i>perceptual</i> difference between the two images. Another metric that takes into account perception is the 2001 paper <i><a href="http://dl.acm.org/citation.cfm?id=383748">Spatiotemporal Sensitivity and Visual Attention for Efficient Rendering of Dynamic Environments</a></i> by Yee, et al.<br />
<br />
I wanted to compare a few of these metrics to try and see how they differ. Zhou Wang from the 2004 SSIM paper maintains a <a href="https://ece.uwaterloo.ca/~z70wang/research/ssim/">nice site with some information about SSIM and comparisons to other metrics</a>. I decided to start by taking the images of Einstein he created, labeled as Meanshift, Contrast, Impulse, Blur, and JPG:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table><tbody>
<tr><td><a href="http://1.bp.blogspot.com/-E65X0zUbE_4/UEO79YDZg7I/AAAAAAAADIo/J8hBgoneSa0/s1600/meanshift.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="100" src="http://1.bp.blogspot.com/-E65X0zUbE_4/UEO79YDZg7I/AAAAAAAADIo/J8hBgoneSa0/s200/meanshift.gif" width="100" /></a></td>
<td><a href="http://3.bp.blogspot.com/-kNqenQZChxc/UEO7-YaHFaI/AAAAAAAADIw/2auEtPDRQhE/s1600/contrast.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="100" src="http://3.bp.blogspot.com/-kNqenQZChxc/UEO7-YaHFaI/AAAAAAAADIw/2auEtPDRQhE/s200/contrast.gif" width="100" /></a>
</td>
<td><a href="http://1.bp.blogspot.com/-DzS2bKl0Qf8/UEO7_bta6CI/AAAAAAAADI4/vk9edhPE4OQ/s1600/impulse.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="100" src="http://1.bp.blogspot.com/-DzS2bKl0Qf8/UEO7_bta6CI/AAAAAAAADI4/vk9edhPE4OQ/s200/impulse.gif" width="100" /></a>
</td>
</tr>
<tr>
<td><a href="http://2.bp.blogspot.com/-HbRFpoFLA3k/UEO8AE1nfQI/AAAAAAAADJA/xTjVkEhbGJQ/s1600/blur.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="100" src="http://2.bp.blogspot.com/-HbRFpoFLA3k/UEO8AE1nfQI/AAAAAAAADJA/xTjVkEhbGJQ/s200/blur.gif" width="100" /></a>
</td>
<td><a href="http://4.bp.blogspot.com/--VEYMobXmog/UEO8Azwle2I/AAAAAAAADJI/uobYRRoIJdU/s1600/jpg.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="100" src="http://4.bp.blogspot.com/--VEYMobXmog/UEO8Azwle2I/AAAAAAAADJI/uobYRRoIJdU/s200/jpg.gif" width="100" /></a>
</td><td></td></tr>
</tbody></table>
<br />
The comparison programs I chose were <a href="https://github.com/jterrace/pyssim">pyssim</a> for SSIM, <a href="https://github.com/myint/perceptualdiff">perceptualdiff</a> for the 2001 paper, and the <a href="http://www.imagemagick.org/script/compare.php">compare command from ImageMagick</a> for RMSE and PNSR. The results I came up with for those five images of Einstein:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-4MarzhvzJ1E/UEO98QVEdBI/AAAAAAAADJQ/K2cqeHXYmNY/s1600/algo-comparison.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="276" src="http://3.bp.blogspot.com/-4MarzhvzJ1E/UEO98QVEdBI/AAAAAAAADJQ/K2cqeHXYmNY/s400/algo-comparison.png" width="400" /></a></div>
<br />
As you can see, PNSR and RMSE are pretty much useless at comparing these images. It treats them all equally. Note that the SSIM values are actually inverted so that lower values mean a lower error rate for both the SSIM and perceptualdiff graphs. The SSIM and perceptualdiff metrics seem to agree on Meanshift and JPG, but have a reverse ordering of the error rates for Contrast, Impulse, and Blur.<br />
<br />
To get a better idea of how these algorithms would work for <i>my</i> dataset, I took an example run of my scene rendering. This is a series of screenshots over time, with both the ground truth, correct image, and an image I want to compare to it. I plotted the error values for all of the four metrics over time:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-u2WGsIdIDps/UEO_Jr_84bI/AAAAAAAADJY/J6UFQ2tIs_E/s1600/run-comparison.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="277" src="http://3.bp.blogspot.com/-u2WGsIdIDps/UEO_Jr_84bI/AAAAAAAADJY/J6UFQ2tIs_E/s400/run-comparison.png" width="400" /></a></div>
<br />
Here we see a different story. RMSE and SSIM seem to be almost identical (with respect to their scale) and PNSR (inverted) and perceptualdiff also show similar shapes. All of the metrics seem to compare about equally in this case. You can find the code for generating these two graphs in my repository <a href="https://github.com/jterrace/image-comparison-comparison">image-comparison-comparison</a> on GitHub.<br />
<br />
I think the moral of the story is make sure you explore multiple options for your dataset. Your domain might be different from other areas where the relevant algorithms were applied. I actually ended up using perceptualdiff because it gives a nice, intuitive output value: the number of pixels perceptually different in the two images, and because it supports MPI, so it runs <i>really fast </i>on machines with 16 cores.Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-44254606824751748952012-08-22T12:34:00.000-04:002012-08-22T12:34:45.261-04:00How I Write LaTeXI've been really happy with using the <a href="http://texlipse.sourceforge.net/">TeXlipse</a> plugin for Eclipse to write my LaTeX documents. There are some really awesome features, so I thought I'd highlight some:<br />
<br />
<h4>
LaTeX Context</h4>
TeXlipse knows all the LaTeX commands, so you get auto-completion when you start typing:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-a431TiCqAc4/UDUGFpdhUHI/AAAAAAAADHg/fbkqT0AsCC8/s1600/latex-context.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="141" src="http://3.bp.blogspot.com/-a431TiCqAc4/UDUGFpdhUHI/AAAAAAAADHg/fbkqT0AsCC8/s400/latex-context.png" width="400" /></a></div>
<span id="goog_1041903641"></span><span id="goog_1041903642"></span><br />
<h4>
Document Context</h4>
TeXlipse keeps an index of labels and references, so if you try and reference or cite something that doesn't exist, you get a warning:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-3oC_68O8QOY/UDUGlZVdlfI/AAAAAAAADHo/GfeDlexpZfk/s1600/error-detection.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="75" src="http://1.bp.blogspot.com/-3oC_68O8QOY/UDUGlZVdlfI/AAAAAAAADHo/GfeDlexpZfk/s400/error-detection.png" width="400" /></a></div>
<br />
It even has auto-completion for these labels too:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-wE1OFz_dK04/UDUGtDqn_OI/AAAAAAAADHw/sAlHgM-VxEg/s1600/doc-context.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="106" src="http://2.bp.blogspot.com/-wE1OFz_dK04/UDUGtDqn_OI/AAAAAAAADHw/sAlHgM-VxEg/s320/doc-context.png" width="320" /></a></div>
<br />
<h4>
Bibliography Folding</h4>
The editor is full of little things that make your life easier. For example, you can configure bibliography files to be folded by default when you open them. It makes it much easier to navigate the document:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-yiu3SjmDzJE/UDUHMY-s3SI/AAAAAAAADH4/6Sk2lJ10vnE/s1600/bib-folding.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="128" src="http://2.bp.blogspot.com/-yiu3SjmDzJE/UDUHMY-s3SI/AAAAAAAADH4/6Sk2lJ10vnE/s320/bib-folding.png" width="320" /></a></div>
<br />
<h4>
Spell Checking</h4>
TeXclipse can use the <a href="http://aspell.net/">aspell</a> command, or you can <a href="http://sourceforge.net/projects/texlipse/files/dictionaries/">download a dictionary file</a> and get instant feedback as you type:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-794nj2dFcpE/UDUHqd9HsWI/AAAAAAAADIA/bzPmsiV7LZU/s1600/spell-check.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="71" src="http://3.bp.blogspot.com/-794nj2dFcpE/UDUHqd9HsWI/AAAAAAAADIA/bzPmsiV7LZU/s320/spell-check.png" width="320" /></a></div>
<br />
<h4>
Continuous Compiling</h4>
TeXclipse runs pdflatex (or you can use latex+dvipdf, pslatex+ps2pdf, etc.) in the background after you save a file. It knows how many times to invoke the compiler so the document gets built properly. If you have the PDF document open, most viewers will automatically reload the file when it gets changed, so you get instant feedback on how the final document looks. It also has the option of using <a href="http://borisvl.github.com/Pdf4Eclipse/">Pdf4Eclipse</a>, a PDF viewer that runs inside Eclipse. What's really nice about this is that it uses SyncTeX to connect the PDF with the original source files. You can double click on something in the PDF and it jumps directly to the source that generated that part of the document. It makes it <i>really</i> easy to make quick edits while reading the PDF.<br />
<br />
<h4>
Makefile</h4>
Since I usually collaborate with other people that don't use Eclipse, and in case I want to build the project outside of Eclipse, I also use <a href="http://code.google.com/p/latex-makefile/">latex-makefile</a> project. This Makefile is amazing. You don't have to configure anything. You just drop it into the source directory and everything magically works. I can't recommend this enough.Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-84508196445903309722012-07-24T11:17:00.001-04:002012-07-24T11:17:31.421-04:00Gigabit Hack WeekendI went to a <a href="http://sfgighackdays.eventbrite.com/">gigabit hacking event</a> this past weekend hosted by <a href="https://mozillaignite.org/">Mozilla Ignite</a> at the <a href="http://archive.org/">Internet Archive</a>'s building. I got to meet some interesting people and had a great time.<br />
<br />
The general theme of the event was to try and demonstrate what kind of applications we can build when people's internet connections get really fast (e.g. gigabit). Since I work on federated 3D repositories with both the <a href="http://sirikata.com/">Sirikata</a> project's <a href="http://open3dhub.com/">Open3DHub</a>, and also <a href="http://ourbricks.com/">OurBricks</a>, I wanted to showcase applications that can be built with online 3D repositories and fast connections.<br />
<br />
3D models can be quite big. Games usually ship a big DVD full of content or make you download several gigabytes worth of content before you can start playing. In contrast, putting 3D applications on the web demands low-latency start times. To showcase online 3D, I took the awesome <a href="https://github.com/blackjk3/threefab/">ThreeFab</a> project and added the ability to load files from an external repository.<br />
<br />
For a demo:<br />
<br />
<ol>
<li><span style="background-color: white;">Visit </span><a href="http://blackjk3.github.com/threefab/" style="background-color: white;">http://blackjk3.github.com/threefab/</a><span style="background-color: white;">.</span></li>
<li>Click the "Import" button.</li>
<li>Pick a 3D model from <a href="http://open3dhub.com/">http://open3dhub.com/</a> and copy the "Direct Download" link box.</li>
<li>Paste the link into the Import box.</li>
<li>The model should load into the editor. (most models work)</li>
</ol>
<div>
The ThreeFab editor lets you export the scene you make. Here's a demo of a scene I created this weekend running on jsfiddle:</div>
<div>
<a href="http://jsfiddle.net/ssDqf/">http://jsfiddle.net/ssDqf/</a></div>
<div>
<br /></div>
<div>
What's important to remember there is that all of the 3D content is being loaded asynchronously in the background directly from Open3DHub. The editor supports URLs from any site as long as they support <a href="http://enable-cors.org/">CORS headers</a>. As a result of the weekend, the Internet Archive is working on enabling these headers for its content, so hopefully we will be able to load 3D content directly from the archive soon.</div>
<div>
<br /></div>
<div>
Thanks to Mozilla and the Internet Archive for putting on a fun event!</div>Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-17010803283656928842012-07-19T13:20:00.002-04:002012-07-19T15:12:55.735-04:00Automatic Debug Shell in PythonI've been working on a few long-running Python scripts lately. I find debugging to be difficult when a script takes a long time to run, since you don't get any feedback until execution hits the point in the code you're working on.<br />
<br />
Python's awesome <a href="http://docs.python.org/library/pdb.html">pdb</a> module lets you drop into an interactive shell by calling the <a href="http://docs.python.org/library/pdb.html#pdb.set_trace">set_trace</a> function. This is really useful because you can inspect local variables from the interactive shell. I found it tedious, though, to keep inserting these statements into my code, hoping it was in the right place, and wasting time when an exception was thrown somewhere other than what I was expecting.<br />
<br />
I found a nice <a href="http://code.activestate.com/recipes/65287/">recipe</a> that lets you drop into interactive mode as soon as an unhandled exception occurs. I created <a href="https://gist.github.com/3145174">a gist</a> that incorporates some changes from the comments. It only enters the interactive shell if the script has an appropriate TTY session, and if the exception is not a SyntaxError.<br />
<br />
Here's an example script showing how to use it:<br />
<pre class="brush:python">#test.py
import debug
def what():
x = 3
raise NotImplementedError()
if __name__ == '__main__':
what()</pre>
<div>
<br />
And an example of using it:<br />
<pre class="brush:bash">$ python test.py
Traceback (most recent call last):
File "test.py", line 8, in <module>
what()
File "test.py", line 5, in what
raise NotImplementedError()
NotImplementedError
> /home/jterrace/python-exception-debug/test.py(5)what()
-> raise NotImplementedError()
(Pdb) x
3
</pre>
<br />
This is really nice for debugging, because it drops you to a shell that has all the local variables when the exception occurred.<br />
<br />
<b>Update</b><br />
<a href="http://yz.mit.edu/">Yang</a> <a href="https://twitter.com/yaaang/status/226029926627831808">pointed out</a> the <a href="https://github.com/gotcha/ipdb">ipdb</a> module. It looks awesome as an alternative to pdb and it comes with a method for doing this automatically.</div>Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-69368673764701722722012-07-19T11:05:00.000-04:002012-07-19T11:05:42.281-04:00Seamless SharingI uploaded an album yesterday of my recent trip to the ICME 2012 conference in Melbourne, Australia to Google+. I wanted to share a link to the album to Facebook, so I posted a link to the album. I was surprised that nothing popped up on Facebook for the preview, so I ran some experiments.<div>
<br /></div>
<div>
I created a public album on Google+, a public album on Facebook, and posted a public tweet with an image attached. I then shared each one on the other two social networks. I was surprised that the only one that actually showed content was sharing a tweet to Google+ and Facebook. Here's a grid showing all the results:</div>
<div>
<br /><div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-ZPs1Y3OZ7Ck/UAgg97IhAHI/AAAAAAAADFw/g8p6Ul4FLPw/s1600/all.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="145" src="http://3.bp.blogspot.com/-ZPs1Y3OZ7Ck/UAgg97IhAHI/AAAAAAAADFw/g8p6Ul4FLPw/s320/all.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of sharing a photo/album across different social network. The rows are the source and the columns are the destination.</td></tr>
</tbody></table>
What's interesting is that other things do work. For example, I shared a link to a Flickr album, and it shows the content across all three networks. I'm guessing that maybe Facebook and Google+ don't have the proper Open Graph tags for the other networks to parse, but this really should be a seamless experience across networks.</div>
</div>Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.comtag:blogger.com,1999:blog-2306733509209314681.post-7531205861641159182012-07-11T01:54:00.000-04:002012-07-11T01:54:05.296-04:00Xvfb Memory Leak WorkaroundI've been using Xvfb, the X virtual frame buffer, for a few projects. Xvfb allows you to run applications that require a display, without actually having a graphics card or a screen. I started noticing that the resident memory used by the Xvfb process would continually go up. I could literally run processes that connect to the display in a loop and watch the memory grow. Clearly, there was some kind of memory leak going on.<br />
<br />
I finally found a workaround to preventing this memory issue. If you add <span style="font-family: 'Courier New', Courier, monospace;">-noreset</span> to the argument of Xvfb, the memory issue vanishes. By default, when the last client to Xvfb disconnects, the server resets itself. My guess is that this works fine when video memory is backed by a hardware device, but Xvfb has a bug where it doesn't free the memory it allocated for the buffer. By setting the noreset option, the server no longer restarts and the memory isn't lost.<br />
<br />
Here's my Xvfb command line:<br />
<span style="font-size: 12px; line-height: 16px; text-align: left; white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;">Xvfb :1 -screen 0 1024x768x24 -ac +extension GLX +render -noreset</span></span><br />
<br />
An explanation of the other arguments:<br />
<br />
<ul>
<li><span style="font-family: 'Courier New', Courier, monospace;">:1</span> - Runs the server on "display 1". Most X servers run on display 0 by default.</li>
<li><span style="font-family: 'Courier New', Courier, monospace;">-screen 0 1024x768x24</span> - Sets screen 0 (the default screen) to a resolution of 1024x768 with 24 bits per pixel.</li>
<li><span style="font-family: 'Courier New', Courier, monospace;">-ac</span> - Disables access control because X access control is incredibly painful to get right, and since the server is usually only accessible from localhost, it's not a big deal.</li>
<li><span style="font-family: 'Courier New', Courier, monospace;">+extension GLX</span> - Enables the OpenGL extension, allowing graphics programs that use OpenGL to work inside the virtual display.</li>
<li><span style="font-family: 'Courier New', Courier, monospace;">+render</span> - Enables the X Rendering extension, enabling advanced image compositing features that most applications will need.</li>
</ul>
<div>
I also use a simple Xvfb start script. It's available <a href="https://gist.github.com/2911875">as a gist</a>.</div>Jeff Terracehttp://www.blogger.com/profile/05494402793921207666noreply@blogger.com