HTML5 goes hand in hand with the cloud. And while the cloud has its share of both patrons and detractors, HTML5 is now reaching levels of performance which make it attractive as a local, native platform. And it can easily become a local, native platform as soon as you equip a computer with a (secured) web server.

And this was the foundation for my decision to reboot Silverscreen, my HTPC solution. Silverscreen before was a C# app written to utilize GStreamer and OpenGL to deliver video in gorgeous fashion to the television and film enthusiasts among us. C# as a platform has serious issues in terms of ease-of-deployment, and is very philosophically limited in it’s use on non-Microsoft platforms. C# also has less support out of the box from content vendors. The most ironic situation was my near inability to interface directly with Netflix, despite the fact that Netflix is delivered using Microsoft Silverlight, which is a platform based on C# and .NET.

Yes, an HTPC application built in HTML5, but also deliverable as an application just like Boxee or XBMC. This means Silverscreen will work without installation from the web, or it can be downloaded and installed and enjoyed when no network connection is around. 

I have replaced C# with the Web, and it opens new possibilities which will change the course of entertainment history forever.

It is interesting to note that 2 out of the three big players in the open source HTPC domain are based on the same code base. Building an application of this scale is a difficult process. It’s taken quite a lot of time and effort to get as far as I have, and there is still a lot of work left to do.

One of the first big technical challenges was welding my chosen media framework (GStreamer) with OpenGL efficiently. Since I am using Mono and C# to write the application, I at first aimed at separating Mono’s performance profile from the potential performance of the video system by passing off the video frame conversion / OpenGL texture uploading to the experimental Gstreamer-OpenGL plugins, which are as of yet not shipped with Ubuntu out of the box. While I knew this would add a dependency, I assumed (incorrectly as you will see) that the performance boost would more than make up for the additional installation complexity.

The mechanism that facilitates texture sharing between the Gstreamer GL plugins and the larger OpenGL application is GL context sharing. Basically, the application process is allocated multiple OpenGL “contexts” which can be switched between using a platform-specific API. On Linux it is GLX context sharing and on Windows it is WGL. Mac OS X has a similar context sharing feature in their AGL APIs. Basically my plan was to use the “glupload” element to convert video/x-raw-yuv video frames from GStreamer to video/x-opengl frames, which consist merely of a 4 byte texture identifier. “glupload” launches a new thread that reads a YUV buffer, converts it to RGB (using GLSL shaders if possible), reuses/creates a new texture to hold the data, and uploads it. Then it sends this identifier down the line to the next element. GLUpload (and apparently the rest of the GL elements) are additionally able to drop the video onto a private video window, though this was of little importance for my purposes.

It didn’t take a long time to come up with the initial code to implement this scheme. And it didn’t take long before I could watch garbled texture memory racing across the otherwise pristinely rendered scene. In between studying at university I tried to determine what I was doing wrong… why wasn’t I getting a picture!? After some time I began to suspect that the problem was not my doing. No matter how much synchronization I did between the glupload thread and the renderer thread, I could not get a clear picture. At about this time I began to look around for other people making use of texture sharing, particularly on the Intel drivers I was using for my laptop. It turns out that since context sharing is almost never used these days, the Intel drivers have severe synchronization bugs of their own which prevent the feature from working at all. The same code that gave me problems on Intel worked fine on a system with an NVIDIA GPU. Unfortunately I did not, and do not today, have a system at this time which has a discrete GPU. I would suspect that many users of my software won’t have such hardware as well.

I decided that my application would require an alternative method of converting video frames to useful OpenGL textures. Since I knew that this code would have to become a branched implementation that could use different techniques on different hardware, I decided to challenge my assumption that the straightforward path would be too slow to be useful. After all, if it WAS too slow we could instead use the threaded GLupload code path for hardware that supports and call it technical limitation.

With my newly found extra time over winter break I have managed to write the new code path. It is both capable of using GStreamer’s autoconvert element or using a YUV-to-RGB GLSL shader for hardware conversion inside the rendering pipeline itself.  The playbin feeds its video frames to an appsink, and my app then passes the frame buffer directly to OpenGL to bring the video data into a texture. This process can have a bit of CPU grinding if the GLSL shader or autoconvert operation might not be fully hardware accelerated.

Once all of this was fully implemented I could assess the level of performance that the app could deliver on my mid-range 2010 laptop (Thinkpad SL410 Core 2 Duo @ 2.0ghz with 3GB DDR2 and Intel Mobile 4 graphics). SD video played adequately, but 720p and up plays jaggy compared to Totem and other video players which can easily dole HD with swagger.

I set out to optimize this code path to get HD into the fold. I could not offer an HTPC application that uses OpenGL for fancy flashy graphics and then deliver substandard video performance compared to less flashy, more vanilla media players. After playing with the details a bit I discovered that my use of mipmaps on the video frames was both wholly unnecessary and tripling (or more) the time needed to upload a frame to OpenGL. Mipmaps are miniaturized copies of the image data which are used to speed up textured rendering when the texture needs to be scaled down significantly. This is most useful in 3D games (where OpenGL came from) where the constantly changing camera angle and distance can easily lead to rendering a large amount of textures in a single frame. Mipmapping attempts to rectify this by imitating a property of real life:  We can’t see the same detail from far away as we see up close.

The video frames in my application are either drawn at full size (or somewhere very near it) or in a small minitiaturized form in the corner of the window. Additionally, the texture will be drawn no more than 2-3 times for every frame, unlike many uses of mipmapping where a texture is tiled or drawn many many times in a single frame. Once gluBuild2DMipmaps() was replaced with glTexImage2D(), HD video in my application reached almost identical performance levels as Totem and the rest of the optimized, stable video player applications. Performance isn’t _perfect_ on this little laptop, but it’s clear that this code path will scale nicely up to the stronger CPU/GPU combos found in HTPC and enthusiast/gaming rigs which my app is most likely to be deployed on.

So I’ve now got several pluggable implementations for creating OpenGL Gstreamer video sinks, which should allow for faster video for capable drivers using GST-GL while falling back to the regulo ole manual pump-based solution. Additionally, this will also allow us to make the gstreamer-opengl package optional, since as mentioned earlier, it is not included in all the Linux distributions yet.

One of the first things I noticed was how much I liked the environment. Those who know will tell you that Smalltalk was where most of the fancy development tools we use today originated. Things like refactoring, IDE code intelligence, class browsing and builtin debugging were firsts when Smalltalk introduced them, and they are in excellent form.

Since Smalltalk is an image-based system, all code and objects live literally forever, until you lose or delete the image which contains them. You can use a workspace to assign a variable an object, save the image and exit, and when you start it up later, the object will be waiting for you, just as you left it. With a system like this, the environment can let you do strange things like modify the structure of a user interface on the fly. Also, since code is just another object in the system, the source code is carried around with it, and each image could have vastly different implementations of the system APIs or indeed any code in the system. This is very, very useful for molding the system libraries and frameworks to do exactly what you need, without having to maintain the housekeeping details like you would with other languages (ie, a branch in DVCS or the directory structure where you keep your modified libraries).

So the image-based nature of Smalltalk does great things for the system as a development environment, as these inclinations tend to allow the programmer to organize his/her environment in just the way they want. But the disparity between “the image” and the “the executable” has become, in my opinion, one of the biggest stumbling blocks for teaching the virtues of message-oriented objects to programmers trained with Java/C# and C++.

Don’t get me wrong KDE 4, you are an awesome beast.

But I have some big gripes with you, and they are very sad gripes indeed. I don’t know who’s responsible for them and I don’t really care. I just hope they get fixed by the time 4.4 is official.

For one thing, KDE wallet needs to go or be seriously rehauled. It is not uncommon when you first log in and your session is restored from your previous boot that you will be baraged with 7 or so KDE wallet password windows. KNetworkManager asks for my wallet password presumably to store a security key even when we’re connecting to an unsecured network. Then, even if I entered my password on the first window, Kopete asks for my wallet password because the window was lurking behind the first one, and if you don’t want to type it and didn’t enter the password on the first window (because say, you’re trying to just get something done without answering the stupid barrage of password windows in the center of your screen) it will proceed to pop up password requests for all the IM accounts you have configured. Personally I have six of them. I also use some pretty fancy window effects which make clicking through those password windows really painful.

At one point when reconnecting after losing my wireless connection, I literally was hit with about 14 password windows for no apparent reason. I never want to see that stupid window modal and center again. If you must have it put it in the damn corner where it belongs.

Naturally these are just two applications (KNetworkManager, Kopete) which have managed to throw 7 password windows at me. I just want them to go away at this point, I don’t care if I can’t log in to my IM services.

Second point, the folder view in Plasma is just broken out of all hell. The whole point of having folder view (afaict) is to allow you to have multiple views of your filesystem, not just a single limited Desktop folder. However, folder views are notoriously unable to remember where they are supposed to be, nor what size they are. So you come to a situation like this when you log in:

No matter how much you try, they never stay in place.

What’s worse, oftentimes you will be sizing your folder views and they will suddenly bounce into the wrong position and become stuck in their place. No plasma dongle on the side of the folder view at all. You can still interact with it but you cannot move it around.

KRunner is so great. I got rid of my KDE launcher button in the corner because I just don’t use it and both launchers suck pretty bad (ESPECIALLY Lancelot: possibly the worst possible design evar). Problem though is that the autocomplete functionality is broken. If you type, say ‘kcha’ and you see in gray ‘rselect’, then press enter, nothing happens. You must first press arrow left to complete the text. This is so stupid it’s mind-numbing, and makes a lot of the utility of KRunner just *disappear*.

Also, though probably not KDE’s fault at all, kde4-window-decorator for compiz has the most vicious bug where hundreds of fake square windows are launched for no reason continuously until there’s 336 windows open, all grouped on your taskbar (thankfully!!). Each time the system launches one of these windows it steals focus from what I’m typing in, so for awhile every time I logged in I had to fight it for control of the KRunner window to fervently type “emerald –replace” and press enter. Luckily I found out what setting compiz was using to launch the kde4 decorator and switched it to Emerald permanently.

On a different note, good luck with the stupid KDE SC branding crap. If you want to run a marketing campaign at least come up with a good one.

In conclusion, I hope you get your act together KDE (and to an equally large extent Canonical for Kubuntu). KDE 3.5 was fantastic for it’s era, and I just want a version of KDE 4 with that same rock solid stability. Please!?

Some performance testing
I’ve done some preliminary performance testing with results you can seethe at. It takes approximately 7 seconds to start iPhone Python apps. This is excruciatingly slow, especially for some of the apps I’m writing which replace builtin apps that load damn-near instantly. I’ve observed that almost 100% of this time is spent loading PyObjC and performing bundle/function loading (objc.loadBundle/obj.loadBundleFunctions)

There’s virtually nothing that can be done about this except to somehow make PyObjC faster. On the bright side, defining a large amount of classes (like enumerations) adds almost no startup time, which is good for the toolkit I’m writing (see below).

Performance within the apps is pretty decent, but here it is obvious that there is some overhead in the translation between Objective C and Python. I created a simple custom dialer control using CoreGraphics, implemented with a single image displayed nine times (one for each digit) and 9 seperate renderings of a single character each. When you tap each button, the whole view is repainted with the selected button highlighted, and then repainted again when the finger is lifted. There is a noticeable lag when redrawing. This is a naive way to implement a control (drawRect gives you a clipping rectangle for a reason, use it!), but the performance of such simple operations should presumably be better. I’m well aware that CoreGraphics is not the fastest public graphics API available and have not drafted a similar test in Objective C for lack of a toolchain to do so, so please let me know if this operation in CoreGraphics just sucks.

Memory usage is actually quite reasonable, which is very good considering the paltry 50-60M of RAM apps have to work with when you subtract the amount used by the iPhoneOS and Springboard. A simple app I wrote to display debug logs weighs in at an RSIZE of about 10M (VSIZE 67M). Comparatively, MobileSafari in it’s native backgrounded state with two tabs open weighs 16M RSIZE, 95M VSIZE. MobileAddressBook (a small, simple application similiar in scope to my debug logs app) weighs 12M RSIZE, 68M VSIZE when in the foreground. These are not scientifically gathered numbers, but it’s not unreasonable to assume this is pretty typical. Even with the additional weight of the Python runtime, memory usage is not significantly increased.

Springboard App Launching Restrictions

Perhaps these are documented elsewhere but it’s worthwhile to get another copy out onto the web. Springboard exercises a number of restrictions on the apps that it starts. These restrictions do not apply to normal processes in so much as that matters considering you cannot run UIKit apps from the command line anyway.

• The process which calls UIApplicationMain must lie within the application’s bundle. This is why the example Python app bundles contain a symbolic link to /usr/bin/python. This means you cannot directly start a Python app using the interpreter directive (“#!/usr/bin/python”)
• The process ID must be the same as the one spawned by Springboard. This is why the BASH script does ‘exec’ and it must be this way. You cannot spawn a new process to take care of the app’s UI, though it seems to be perfectly acceptable to start processes to do other things.
• When Springboard launches an app which uses a script as it’s executable, it checks that the interpreter directive points to an executable within ‘/bin’ (there may be other allowed applications but ‘/bin’ is the only one I’ve confirmed). Scripts using /usr/bin interpreters (or any other directory) will not be executed.

So why does this matter? Well currently Python apps on the iPhone are a bit clunky. You’ve got a BASH script which execs your symbolic link to Python with your app’s main python script as an argument, and optionally does some logging redirection. I discovered most of these limitations by trying to take BASH out of the equation. I failed at this for these reasons, though it seems entirely possible to create a simple C app which does the setup much quicker than loading BASH. For now though I’ve found a clever solution, which at least cleans up an app’s directory structure. Toss this header onto your app’s main Python script, make it executable, and set it in your Info.plist’s CFBundleExecutable key:


#!/bin/bash
"""":
# This section is a BASH initialization script which sets up logging of stderr
# to /var/mobile/.err.log to catch Python errors
exec "$(dirname "$0")"/Python -u $0 2>/var/mobile/`basename $0 .py`.err.log
# -----------------------------------------------------------------------#
# End BASH, Begin Python """
import sys
...


As you might be able to see, the script is first executed by BASH. The line “”"”: is both a no-op in BASH and the beginning of a
multi-line comment in Python (the last double quote and colon are ignored as part of the comment).
Since we do an ‘exec’ within BASH on ourself, nothing past the long dash line is executed. When Python takes over, it starts interpreting from the ‘import sys’ line. This approach is clean and simple, allows you to free up slots in your editor, and makes editing your logging directives easier. As in my previous BASH app launcher examples, this will create a log in /var/mobile with the name MyCoolApp.err.log, where MyCoolApp is the name of your app (at least, according to the bundle’s directory name).

An iPhone Toolkit for Python

As I briefly mentioned in the last installment, I am working on a Python toolkit for iPhone app development. The toolkit will do all the PyObjC wrapping for you, and will provide a large number of convenience classes which ease the process of writing your apps. The toolkit will also include a set of scripts which make testing and debugging your code much easier.

A surprising number of iPhone APIs are C-oriented, probably owing to the limited storage and memory of the devices which employ iPhoneOS. These APIs will be offered in their awkward C glory in addition to Python-specific APIs which are much easier to use. The toolkit already contains a Python class for CoreGraphics contexts, and (more or less) most of the API is already covered. The toolkit will strive to create bindings for private frameworks as well (to the extent possible, given the near complete lack of community documentation on them). Why would we not provide them considering Apple has already banned your Python apps from being listed in the app store?

Having log files of your app’s standard output and python errors is great, but having to type a command to print the log or in some way refresh your view of it is tedious. It would be better if we had a view of the output log as it happened.

Initially I tried to create a cradle of sorts for launching Python apps in a “debugging mode”. This mode would detect exceptions and Python crashes and pop into the debug log viewer application I wrote. This MIGHT still be possible, but has so far thwarted my discovery due to the Springboard restrictions listed above. If anyone has more information on how to facilitate this I would love to know.

It is possible however to have a live view of Python’s output. Like any decent UNIX implementation, iPhoneOS supports named pipes, also known as FIFOs. Named pipes are special files which allow two processes to send data to each other. All we need to do is modify our logging code to write to a FIFO, and at the same time read it from another process.

So one of the scripts included in my new toolkit does just this. Once launched, it clears the screen and starts reading the FIFO. This blocks until your app opens the FIFO for writing. The data is then written to the screen, and when your app quits and closes the file descriptor, the script stops reading the data. Stick it in an infinite while loop and you have an incredibly useful error console. No more printing log files!

Setting up your development environment every time you boot can be tedious. I’ve created a script to do the lifting for you automatically. All tools provided with the toolkit use a unified method of determining if your iPhoneOS device is connected, and allows you to mount it on your own if you please without errors.

All this and more will be released as alpha software in my next installment, so stay tuned!

Through the course of this, I’ve noticed a lot of caveats and gotchas that are not well described in the existing iPhone Python information available on the web. Hopefully this info will be useful for the next iPhone Python dev who is desperately googling for the information they need!

Firstly, Apple is very bad at reorganizing data without thoroughly breaking URLs. The Objective C documentation specifies a compact encoding for representing method signatures, which is used by PyObjC as a guide when marshalling data between Python and the C world. Normally the user of an ObjC class need not worry about these encoded signatures, as Objective-C and Apple’s frameworks provide enough metadata to figure them out automagically. However, some frameworks (such as CoreGraphics and UIKit) export functions which are not part of any Objective-C class, but are instead normal C accessible functions. In the case of CoreGraphics, I presume this is to allow C applications to easily leverage the same API as Objective-C ones.

PyObjC is unable to infer the signatures of these methods, because they are not covered by any Objective-C metadata. This makes sense, since they aren’t Objective-C signals anyway. So in these situations, the functions need to be imported from the Python side using a special method of the ‘objc’ object like so:


objc.loadBundleFunctions(BundleObject, globals(), [("Method1", "Signature1"), ...])


The format for the ‘Signature1′ part is documented by Apple. Several blogs about this subject linked to the information, but Apple reorganized the whole document and left a broad redirect from the old locations to the index of the new version. It took me a very long time to find the information in the new structure, because there was no obvious name for the feature! For the record, they are “encoded signatures” and the process is “signature encoding”. I’m putting the information here to avoid relying on Apple for it.

Signature Encoding Format

A signature consists of a set of characters, which each represent an element of the method’s signature. The elements include the return type, implicit arguments (like the instance pointer and selector for Objective-C messages), and explicit arguments. A number of modifiers are supported as well.

@   - object
i,I - int, unsigned int
:   - method selector (SEL)
@:  - instance and selector, means function is an
instance method (use after return type, before args)
V,v - void
B   - C++ bool (0 = false, other = true)
f   - float
d   - double
*   - charptr
s,S - short, unsigned short
l,L - long, unsigned long
q,Q - long long, unsigned long long
bn  - bit field of n bits
?   - unknown type, void*, function pointers
c   - signed char
C   - unsigned char
^t  - pointer to type t
#   - objc Class
[nt]- array of length n containing type t
{name=t..} - structure, where 't..' is the order and type of fields
(name=t..) - union, where 't..' is the types of union contents

Marshalling Strings
PyObjC accepts normal python strings for char* elements (‘*’). NSString is exposed to Python as a subclass of the standard unicode type, which allows them to be immediately usable. You can also provide any Python string in place of an actual NSString. Getting an *actual* NSString object is hairy however, so it seems the best calling method to use for NSString methods is NSString.methodName_(stringObject, arg1, …)

Subclassing
In Python, you instantiate Objective-C objects the same way you would in Objective-C. You use alloc() and an init() method or a combined create() method depending on the provided API. This doesn’t change when inheriting Objective-C classes in Python. You need only create an init() method for your subclass which calls the appropriate init() of the superclass, and don’t forget to return self at the end! PyObjC takes care of creating an appropriate alloc() which returns an object of your new type.

So an example constructor might look like:


def MyClass:
def initWithFrame_(self, frame):
self = super(MyClass, self).initWithFrame_(frame)
return self


Note that the call to the superclass method initWithFrame_ will differ depending on what class you are inheriting from.

CoreGraphics
CoreGraphics is an ugly API. To work well in all languages, the API is function-oriented– which is all well and good until you add Apple’s ridiculously bloated naming conventions (“CGContextGetUserSpaceToDeviceSpaceTransform” anyone?) Since PyObjC can’t discover normal
functions on it’s own, you have to import them individually using objc.loadBundleFunctions (see above). PyObjC provides a wrapper module for Quartz but unfortunately it is not provided in the pyobjc distribution I have installed on my iPod, and it uses Objective-C helper methods which need to be compiled (I still do not have a working toolchain), so I cannot yet use it.

To make CoreGraphics integrate better with Python I am building an object-oriented wrapper API which resembles cairomm. It strips operation names of unnecessary words and makes the details easier to remember. PyObjC by default marshals CGRect, CGPoint, CGSize etc into tuples. CGRect is actually a tuple of 2 tuples which each have 2 floats (ie, ((x, y), (w, h))). I’ve made Python types which can take a tuple representation and make it more accessible with named fields:


rect = CGRect(window.frame())
print "the x coord: " + str(rect.x)
print "the whole thing: " + rect.toString()
window.setFrame_(rect.toTuple())


Enums are also not imported automatically by PyObjC, so to retain the sanity of the code, I am building subclasses of int to pin the proper values to their names.

I plan to clean up the code and release it as a reusable iPhone Python toolkit, but that’s for the next part of this ongoing series. I’ll also talk about performance and Springboard integration, among probably many other things.

Happy Hacking!

Luckily, thanks to saurik and crew, we have a Python port available from Cydia/Rock. Very little work has gone into discovering what it takes to write Python applications for the iPhone, and thus very little information is available. I hope to remedy that with this new series of posts. I will be adding new posts as soon as I have new information so more people can begin prototyping their apps in Python (and thus avoid the immediate need for a working Objective-C toolchain)

Getting Started with iPhone Python
To begin you will need the iPhone Python port which can be found in the iphone-python APT package, or just search for iPhone Python from Cydia, Rock, or Icy.

Once you’ve installed it and resprung, you will notice a new application called HelloPython. This application, like all jailbreak apps (and in fact all of Apple’s stock apps) are found in the /Applications directory in the root filesystem of your device. This directory is actually a symlink to /var/stash/Applications.XXX, so if you are interacting with your filesystem using Linux’s sshfs like I am, you will need to use the full name since the symlink will be broken in this situation.

All iPhone apps are OSX bundles. The bundle names end with ‘.app’, ie MobileSafari.app, MobileNotes.app etc. Each application bundle contains an Info.plist file which defines a number of properties of the application including the title shown in the Springboard, the bundle identifier, the filename of the executable to start when the application is launched etc.

HelloPython simply displays your contacts. There is another more advanced example Python application which was released to the iphone-python mailing list at telesphoreo called pyExamp. You can find it here (http://www.telesphoreo.org/pipermail/iphone-python/2009-February/000324.html).

Neither application is even close to useful, nor are they typical of a real application’s structure. They do however illustrate the PyObjC support which allows you to interface with the iPhone APIs. For more information on the iPhone APIs, visit http://developer.apple.com and sign up as an iPhone app developer if you haven’t already.

The first step in writing your app is to create a viable template in which to write and test the code. There is no iPhone simulator that can test your Python apps, the only way to do it is to put your app on your (jailbroken) iPhone and run it. The iPhone OS does not directly give you error messages, nor is there any way to view the output of your application directly when it is run from the Springboard. It is also not possible to start iPhone apps via the console. Erica Sadun’s “Erica Utilities” package features a ‘launch’ application which unfortunately does not work for OS 3.1.2. Running your app from SSH or MobileTerminal.app still has advantages: the app should launch completely normally, but it will not display anything (headless).

When an iPhone-Python app launches, typically the Springboard will run a BASH script (see the ‘HelloPython’ and ‘pyExamp’ scripts for examples), which launches Python and points it at your main application script (HelloPython.py or pyExamp.py). A UNIX developer’s first instinct for reading output from an app run would be to use ‘tee’ so that output is visible when run from the command line as well as logged to a file when run from Springboard. Unfortunately this will not work!! If you use ‘tee’ your applications will not launch. You will likely see the error ‘Failed to register com.company.appName with bootstrap server: Unknown error’ in your log. This is because of the redirection BASH does to send the app stdout/stderr to ‘tee’.

Normal redirection like ’1>&2 2>/var/mobile/myApp.log’ will work just fine. Unfortunately this means you will have to display the log file to get the output, even when you run the app from the command line. My apps use a generic script which does not need to be modified when used in a new application, feel free to use and improve it:
#!/bin/bash
exec “$(dirname “$0″)”/Python -u “$(dirname “$0″)”/`basename $0`.py 2>&1 1>/var/mobile/`basename $0`.err.log

Important: Notice the ‘-u’ option to python. This causes Python to run in unbuffered mode. If you don’t do this, your app output may never get written to the log file (or even screen) when Python is killed or terminated abruptly. So if the script is called ‘MyApp’, then the log will be /var/mobile/MyApp.err.log.

This setup is sufficient for getting your program output. Some people prefer redirecting Python’s stdout from within Python itself like this:

sys.stdout = open(‘/var/mobile/myApp.log’, ‘w’)
sys.stderr = sys.stdout

However you will not see Python parse errors using this technique (obviously), so you would still need to use the first mechanism to capture parse/global errors which occur before the above code is even executed.

Before running your new application you will need to change the bundle identifier (and whatever else you want to change) in Info.plist. This is only needed to prevent the iPhone OS from thinking your app is already running (of course, not a problem unless you are backgrounding HelloPython for some odd reason).

With this info in mind, you should be able to make a copy of HelloPython.app or pyExamp.app and get it to run. Next you’ll have to actually write the application!

My next post will talk about the awkwardness of PyObjC and how to make iPhone-Python development smoother.

I understand that Wifi location services is the only method the iPod Touch can use to determine it’s location. I also understand that mapping the location of every access point in the US is a difficult, if untenable goal. In this age, this sort of huge effort is much improved by using crowd-sourcing. As a geek who loves organizing information, I would be honored to help Apple and Skyhook improve their access point maps, but this is very difficult because of the lack of effort on the part of Skyhook, and to a much lesser extent, yourselves.

I live in the upper peninsula of Michigan, and it is very clear that Skyhook has never sent their trucks through my area. My home access points, those I use at my university, the public APs found at grocery stores and coffee shops all provide no location information on my iPod Touch. What’s worse, is the location check actually spins for about 40 seconds before just giving up, instead of quickly recognizing that the AP is not mapped. Location-aware applications ask to check my location multiple times, near constantly, and some applications simply do not work if location detection fails.

To begin to remedy this situation I have begun submitting MAC addresses with location information directly to Skyhook but the process is very rough. Their web page for doing so is very inconvenient as it lacks any sort of location search, and instead requires you to zone into your location from a global view, which is error prone and time consuming. None of my submissions thus far (the first sent in 2 weeks ago) have been added to your database yet (and thus location-based apps DO NOT work at all for anyone in my area). There is no iPod app for submitting access points, and the iPod does not reveal the AP’s MAC address to me. Therefore, I must boot my laptop to find the MAC address of my current AP, which means using the Skyhook submission page on my iPod is pretty useless.

On top of that they have totally botched the Linux support for their Loki toolbar, meaning I must use ‘iwconfig’ to get my AP MAC and plug it into that terrible web page each and every time I want to submit a new AP. Mac OS X is a UNIX system like Linux, do you really want to work with a partner who offers a UNIX “installation script” that looks like this?

(the file is called “LokiPlugin_Installer.sh”):

– beginning of file –
Self Extracting Installer

installer_actions.sh
npplugin.tar

Running Installer

libnploki.so
loki_logo_dialog.png
Installation complete
– end of file –
Figure 1: This is what happens when you let a moron manage your Linux support

My university has one of the most advanced WiFi networks in the central United States. Intel frequently does WiFi-related internships here and just this year they launched a WiMax testing program. As you can imagine, there are literally hundreds of APs spread across our fairly large campus, which would be a massive amount of wonderful data for Skyhook and yourselves, and could offer very (VERY) fine grained location data to our students and staff.

I posit to you Apple, that you replace Skyhook Wireless as the location data provider for the iPod Touch for one which is more capable of managing the ends of the business that affect your customers. I know it is very possible for Apple to change location providers or completely drop the features with an OS update, and I suggest you do so. With the exception of the location services system, I have been incredibly impressed and happy with my new iPod Touch 3G, which makes the utter failure of such services even more in sharp contrast to Apple’s usual reputation for ridiculously good quality (I have *never* gotten a proper location).

Even if there are no viable databases you could adopt that would provide the scope that Skyhook already does, there is no reason you could not improve upon their reporting by integrating location submission into the iPod Maps application. There is no reason the dialog which says “Could not determine your location” couldn’t have a button which says “Map my current location”, which opens Maps and allows you to stick the pin on your current location, which is submitted to you, and eventually to Skyhook. Please AT LEAST make it possible to use Location-aware apps even when Skyhook inevitably fails to determine your location.

Thank you for your time. I hope you address this problem as soon as possible.

Sincerely, your devoted iTunes and iPod Touch customer,

William Lahti

(This letter was sent to Apple Feedback and Support shortly after posting this.)

If this sounds a lot like what Boxee does, that’s because it does do this. Unfortunately though, Boxee analysis is rudimentary, and requires particular ordering and formatting of the file name elements (title, year, episode number, etc) to achieve decent reliability of listings retrieved by the client.

Can’t find a listing? Those Scene Tags Are In The Way.

MediaExpert’s approach is different: it filters the “title string” (initially, the file name without the extension, and all non-alphanumeric characters replaced with spaces) until it has something it believes is very likely the title of the content. It does this by recognizing common scene tags and removing them from the title string which will be searched by metadata services. The tags are stored in the hint information, allowing the application or other filter plugins to make use of the information. It already recognizes a great deal of tags which are formatting or source related (ex. 480p, DVDRip). It also recognizes some of the most popular scene release groups. Finally, it treats any words found within square brackets ([, ]) as scene tags. It looks for a four digit number which starts with either 1 or 2 and if found, considers it the year the content was released.

The system supports pluggable “scraper” filters as well, with one builtin one: TelevisionScraper, which looks for season/episode information in a variety of formats including sNeN, eN, NxN, and more. Unlike Boxee, a minimum of 1 digit is allowed with a maximum of 3 for seasons, and 3 for episodes. This information is stored within the metadata hint.

Now the metadata services query their respective providers and return the results. The system is then able to narrow down the possibilities via merit, that is, a best-score-wins heuristic based on things like how close the actual year is to the one provided in the file name, whether the content type (episode, movie, etc) matches the metadata provided in the hint (like season/episode numbers), how close the expected title is to the real title (using Levenschtein distance).

Altogether this makes for a powerhouse of media detection capability, without any compelling need to compulsively rename your media collection (hey, feel free if you want to).

The system will also work with many other media types like music, news, adult content, etc. The media-expert tool will also be capable of exporting the metadata in XML format for caching and distribution.

I read about how this whole Free Software thing got started. I learned about Richard Stallman, and his early work developing a ridiculously impressive text editor. He decided to steward it’s source code differently than anyone else at the time. Instead of charging for the program binaries or merely freely distributing the source, he would instead codify a new set of principles which would ensure the software remained free and accessible for all users.

Merely open source would not do, no one would be allowed to shackle his code. In this act, a community was born. A user could enjoy the use of a huge library of software, a user-developer could work to improve it, and everyone benefits. Oftentimes the user-developers would be so enamored by the grace of the gesture and the sweat of hard work invested by the developers, and they would begin their own separate projects to expand the coverage of what would quickly become a veritable ecosystem of code. Stallman made no assertions as to the sale of the software, only that all modified source code would be available to all users. It did not take long for the economically-minded to craft business models which resonated with the principles of Free Software.

Though I learned of the power of free software long after the movement was long underway, my commitment to the community and tenants of free and open source software is very strong. A great many others feel this way, most even more than I with a resume of community participation and contribution which greatly outweighs my own. I myself have contributed free code to such edge projects as Slicker and Y, and have from time to time submitted patches to a few of the larger desktop projects such as Xorg.

But the work which I am most dearly fond and proud of was my work on the SharpOS kernel. We did something incredibly novel and intriguing, and we managed to meet and exceed our initial goals. The project was truly open source in it’s creation, evolving from a shared discussion of a number of open source C# developers participating in the Mono Project at the time. During my tenure there I also introduced the foundations for a new windowing system (SharpWS), BASH-like command shell (Nash), and a set of UNIX base utilities to complement the SharpOS base system.

For those who weren’t in-the-know during it’s development (and I commend you, then, for reading my blog anyway), SharpOS became dormant because of a clash between the commitments to free software of some of the developers, and the pull away toward an open source model. Mircea Cristian Racasan (Chriss) led the majority of the core developers in the notion that our considerable work should remain protected under the terms of the General Public License, but outside influences wanted to see the code become open source instead. I did offer to switch licenses for the secondary projects I had contributed (SharpWS, Nash, and SharpOS-CoreUtils), but could not advocate the same for the SharpOS kernel, nor the AOT which Chriss had mostly developed.

As a result of the unavoidable schism, more projects with similar goals to SharpOS appeared, first with a fellow Chad Hower, much-aligned to Microsoft (and thus non-copyleft open source) and then with one Scott Balmos, with whom we merely had design disagreements (licensing was still an issue, but to a lesser extent). Considerable effort was applied to these initiatives, and the scene fragmented. Harsh words were exchanged, a lot of difficult effort was duplicated, wheels were reinvented. I advocated for friendship and respect between the factions, and I among others helped push for a joint forum where code could be shared, topics discussed openly, and mutual respect could eventually lead to more progress for all of the groups involved.

And here is where my conflict begins, o friends, for despite my dedication to the ways of Free Software, the church of Richard Stallman has made my fellow C# developers and I to be extant. I am a principled man, once a radically anti-Microsoft boy. My principles include that of tolerance and community. Another of my principles is recognizing the accomplishments of a particular technical/engineering solution. Microsoft made an excellent development platform with their .NET efforts which far surpasses Java in design, coherence, usability, and features. The CLI is easier to extend and to implement. The behavior of Microsoft towards CLR licensing and standardization has been more than adequate, and any of Microsoft’s patents which cover the standardized portion of the CLI have been waived under a royalty-free[sic]* license required by the ECMA. This leaves the nonstandard portions like System.Windows.Forms, System.Data, and others which are distinct and easily removed if Microsoft decides to return to it’s dirty bastard ways. Listen, Microsoft is not trustworthy, they are not our friend, and they are competition for the Linux desktop. But for the most part Microsoft is reaching out in a positive way, and this needs to be acknowledged.

The truth of the matter is that the code that we write in C# cannot be taken away from us. Microsoft cannot sue anyone for writing software in C#. They cannot sue anyone for including a Common Language Runtime, and they cannot sue anybody for using software written in C#. The FSF knows this enough to implement the CLI in their DotGNU project. Yet still Stallman preaches not using C# software and not developing C# software. To me, this means some people who might find my software useful will fail to try it out because Richard Stallman told them to avoid C#. What do you have against me, RMS? This is incredibly insulting to True Believers of FOSS who code C#, and I am left hurt.

Microsoft can sue people for using patents via unofficial clones of their proprietary APIs, which the community acknowledges. Why isn’t this where Richard focuses? He should be advocating the blacklisting of Microsoft’s extra APIs which do offer an actual *threat* to the Linux companies which are distributing them right now. It seems he has grown out of touch with what is relevant, opting instead to merely draw a black line between himself and anyone vaguely associated with Microsoft, taking no exception to hurling insults and generally ignoring the voices of the Free Software fellows who see C# and the CLR for what they are: an excellent programming language on top of an excellent development platform.

With his recent comments about Miguel de Icaza and his continued effort to make pariahs of C# supporters in the FOSS community, Stallman is alienating the entirety of the Mono, MOSA, Axiom, Grammatica, Tomboy, and Banshee communities among many others.

Look us all in the eyes, Stallman, and tell us that our contribution means nothing.

* UPDATE: Having run into a bit more reading around forums and stuff I realized that I was wrong about ECMA requiring royalty-free patent licenses for patents related to the standards. While this certainly produces some doubts in my mind, I still have to stand by the CLR, as the wonderful platform that it is. Microsoft does have the Community Promise, which basically says we won’t sue you if you implement enough of the specs, but I was left with a sense of ambiguity after reading into it. The solution though is pressing Microsoft to follow the tried and true open source patent licenses like those used by the Open Invention Network and IBM. The solution is not to just push Microsoft away and ignore the progress, alienating the developers who code with Microsoft’s frameworks.

I finally settled on a new portable hard drive. My last portable was a 1TB Hammer moreSpace, a big, clunky drive which required a hefty amount of additional external power to function. It lasted all of 2 weeks, topping out at about 45GB of data before it broke in a terrible coffee table -> floor transfer which I have dubbed “The End Of The Beginning” of my data storage renaissance. Here I am six months later getting another one! But the difference is huge. For one, this new drive (the WD My Passport) does not require external power. Yup, it runs on less than 5V from a USB cable. Second, this drive is very small. It fits nicely in my pocket or backpack and is easy to transport. Finally, its 320G, not 1TB. This is good! Why? How do you even begin to back up a terabyte-grade storage device?? Unless you have the cash to pick up another one, the answer is: you don’t. But a 320GB drive is much more manageable (plus it fit into the amount of money I wanted to spend).

One of the cool things I was looking forward to doing was installing Ubuntu on it. This would enable me to run multiple OSes without dealing with the University tech crew to get a dual boot partition set up (or go it alone and install my own Windows, the only copy of which I currently own is installed on my desktop box). With this in mind, I split up the hard drive into a number of partitions.

Here’s where the Ubuntu Jaunty review part of this post comes in. As always, Ubuntu booted into a graphical desktop from the LiveCD perfectly, with a few caveats. For one, it didn’t bother recognizing the highest resolution when showing the ubuntu boot screen, so it was a little blurry. But X itself did, and all was good. But when Ubuntu played it’s well-known startup sound, I was horrified to hear that Ubuntu overstepped the nominal PCM levels on my Intel HDA card, meaning I heard a terribly-distorted mess of a startup sound.

Then I opened the Ubuntu installer. I made it to where the partition/disk setup page should be and the Ubuntu installer froze. Everything else worked, and the busy cursor was shown fine, but nothing happened. Rebooted and tried again with the same result. Then I rebooted and selected Install from the boot menu. It still took awhile to get to the disk setup screen but it worked this time. Ubuntu told me I had no operating systems (there was, obviously, there was Windows on my laptop HDD), but underneath the “Take over the disk” option it said “Warning, Windows XP Professional will be deleted”. Haha. Anyway I’ll chock that up to an Ubuntu dev forgetting that people have multiple hard drives at all. In any case the install completed and I was able to boot Ubuntu from the drive with no problems. So: only a *very* minor audio problem to get Jaunty installed on this Thinkpad R61.

Now, prior to this I have been having issues with the disk performance on my laptop HDD. Speed has been gradually slowing down- I figured this was a lack of defrag runs. Trying to run the defragmenter yielded “No, there is a disk check scheduled for next boot, do that first.” But rebooting did not make chkdsk run. I ran chkdsk /f from the command line, both in normal and safe mode. “Chkdsk needs exclusive access, would you like to schedule for reboot?” That did nothing as well.

So, with this back story in mind, I went to reboot into Windows. Unfortunately though, I immediately received about 25-30 alert notifications upon logging in that “Such and such file is corrupted. Please run chkdsk”. Riiight. I fumbled with awhile before I found out about chkdsk /x which tries to force unmount a volume, but I didn’t get a chance to use it. I went to bed.

Now this morning I attempted to boot, only to find that Windows would not exit it’s startup screen, both in safe and normal mode. I brought the box to the University help desk so I could use a Windows disk to get at the Recovery Console, so I could run chkdsk. The first one (normal chkdsk) froze at 25%. The second one (chkdsk /r which checks for bad blocks) froze at 25%. I was running out of time and didn’t want to use up the help desk guy any longer so I quickly booted up Ubuntu from my portable and did a quick emergency backup of whatever I could before doing a reimage of the hard drive, which finished without problems.

It seems whenever I upgrade my storage capabilities I am downgraded by fate at about the same time. In this case I didn’t wind up losing any capacity but it was still a hassle, and I probably lost some minor data. Maybe this round was a return on the bad karma I accumulated when I broke my 1TB drive :-\. If so, I hope I’ve paid that all off now because the storage renaissance is far overdue.

Well, I bet he would have liked the SharpOS AOT under the MIT license, because he could’ve integrated that right into MonoTouch and then sold the basic version for $400 a pop. I don’t think Chriss (the main dev of our AOT engine) would’ve seen a penny of that.

I’m glad we never took that advice, and I think it’s just cold that something so attractive to the community and so heavily based on the community’s work would be quite this closed. This is why I don’t use the MIT license.

I can only hope that once Novell feels they’ve recouped their losses and made a nice chunk of change that they will consider letting us FOSS peons take a crack with it.

I’ve been a fan of the show since about half way through the first season, and had I known the utter mind fuck I was in for, I would’ve jumped on the bandwagon from the pilot. The only reason I didn’t is because of my preconceptions of Eliza Dushku and my lack of understanding of the premise.

So to do my part in educating the masses on the question, “What is Dollhouse about?”, I will tell you. Dollhouse is about a secret organization which develops technology which can replace someone’s persona with a different one. They figure out how to store personas, modify them, combine them, and load them back into people. They deploy this technology at several top-secret “Dollhouses” around the world. Each dollhouse has a number of “actives”, who are people who volunteered (and I use the word loosely) to loan their bodies to the organization in exchange for forgiveness of a crime, reduced prison sentences, or just plain money. The motives of the organization are suspect, though they claim the overall goal of this is to “help improve people’s lives”. The actives (dolls) go on engagements (read: missions), where they are prepared by imprinting them with an appropriate persona which can achieve the desired goal. Sometimes the actives might be ninjas. Sometimes they become negotiators. Sometimes they are lovers.

A good portion of the engagements see the actives work as courtesans for the client in an elaborate fantasy, though the series doesn’t focus too heavily on this. This is just how the Dollhouse pays the bills. It’s clear that the organization behind them has ulterior motives.

When the actives are not engaged with a client, they are wiped clear of persona, and they become innocent and child-like. The hook is that one active (played by Eliza Dushku) starts to become resistant to this process, and begins to remember things she was supposed to have forgotten.

Omega

After watching the finale episode “Omega” a few months ago I anxiously awaited the second season, ready to find out how Echo’s new “state of mind” would influence the direction the show moved in. However, as many people know, the 12th episode finale is not really the last episode. Fox commissioned 13 episodes for Dollhouse’s first run. Whedon scrapped the original Dollhouse pilot and cannibalized it for the rest of the season, meaning to fulfill contracts and such, they needed another episode. Whedon’s solution was to write a new episode with a smaller budget to include on the DVDs.

When I heard that there was more Dollhouse to be had before the second season airs, I was ecstatic. Unfortunately, the DVD release wouldn’t be until the 2009 Comic-con, so I had a wait. But that time has come, and I have watched the episode, and I must say that it is a mind fuck on *top* of a mind fuck.

Epitaph One

Epitaph One breaks from the current story line to reveal some of the future repercussions of the technology developed by the Rossum corporation. Given the fact that this episode had a smaller budget then the others, I think it’s odd that Whedon decided to go with the futuristic apocalypse plot, but it worked perfectly. Much of the episode takes place in 2019, after the Dollhouse technology has gone mainstream and landed into the hands of China’s military (probably other nations too, but China is specifically mentioned). That being said, to the Dollhouse fans who haven’t seen it yet: it will be a little confusing at first. Hell, I wasn’t sure I was actually watching Dollhouse at first. But it does make sense. You just might have to watch it a couple times- as a lot of the plot is by inference.

It seems the performance of Hulu Desktop is degrading very badly for my system. It doesn’t have the best graphics hardware but I’ve been watching Hulu desktop happily since it came out and it has never been this bad. My system is a Thinkpad R61 which I received from my university one year ago. It has Intel 965 Express graphics and a Core 2 Duo @ 1.8ghz with a gig of RAM. Running Windows XP SP3. All nonessential programs and services including antivirus disabled has no effect, and my application performance monitor, which is the excellent Process Explorer from the Sysinternals group at Microsoft indicates that Hulu desktop is the only process which is consuming any sort of CPU over the entire runtime of videos on Hulu desktop.

This is not network saturation, the buffering is typically full. This is not an audio issue either, as it continues to be as good as ever. Just the video is choppy, noticeably below 30fps.

In sharp contrast to the bad performance of the 480p content being served to me via your software, my current system performs very well on all Flash video sites via my browser. Watching 720p HD on Youtube is flawless, watching an upscaled 480p video from IGN is on par with what I’d expect from this system, and watching the HD gallery or any of the 480p content at the Hulu website runs perfectly. The primary difference between my browser and Hulu Desktop is that the former uses Flash 10 and the latter uses Flash 9. A much smaller difference is the intensity of the Hulu Desktop application compared to Youtube, IGN, Hulu’s own web player, or any other Flash video player, especially for a computer which is quite capable of playing comparatively graphically intense video games like Spore and any of the Prince of Persia Sands of Time games at high resolution.

Flash 10 delivered impressive performance enhancements for online video viewers who had any sort of decent graphics chipset, because it improved seriously upon the ability for Flash to accelerate video playback, scaling, and processing.

How would Hulu make it better? They would move to Flash 10, as soon as humanly possible. How could they make this better for us in the mean time? They could return the FLASH QUALITY controls until the Hulu Desktop team is ready to move to Flash 10.

How would this help, you ask? The Flash “Quality” setting in the context menu of all standard Flash movies changes the *Rendering* quality, not the video quality. Setting it to low turns off *antialiasing* on fonts, vectors, and anything else. It still does basic compositing, so you can still use video overlays and see translucent items properly, but you get the once infamous jaggies at the edge of stuff.

Personally, I would prefer lack of antialiasing if I could avoid missing yet *another* critical second of the amazing dramas and other fine television shows this service offers me. At the very least, turning the Flash Quality menu back on is nothing more than the flip of a switch and an update. Whatever the reasons are for being shackled to Flash 9, if you insist on bundling an outdated version of your application platform, at least give us the ability to squeeze some sort of reasonable experience out it!

Despite this protest, I still thank you greatly for making a very enjoyable service and for launching Hulu Desktop as an open beta as early as you did, even though the whole range of Hulu promotions has not yet been integrated. I use Hulu for almost all of my television consumption and have been overall blown away by the excellent content, good video quality, perhaps too reasonable of an advertising experience, and the firm embrace of Web 2.0 functionality. I hope that you’ll take these points into consideration.

Sincerely, your humble user & alien food, who was HBB (here before Baldwin),

user account rezonant
0 friends (no one seems to have friends yet anyway)
380 videos watched
72 subscriptions
2 reviews
490 ratings
33 discussions

- Sign both these petitions to tell NBC we want it renewed

http://www.petitionspot.com/petitions/renewnbcchuck

http://www.youchoose.net/campaign/save_chuck_nbc_show

- Send a box of Nerds to NBC at:
NBC’s “Chuck”
Attn: Ben Silverman
3000 W Alameda Ave., Admin Bldg
Burbank, CA 91523

NBC’s “Chuck”
Attn: Angela Bromstad
100 Universal City Plaza, Bldg 1320E 4th Floor
Universal City, CA 91608

- Head over to Hulu and watch some episodes of Chuck even if you’ve already seen them all. Write a review for the show and make sure to rate the episodes (come on, five stars!)

- Stop by the NBC messageboards and post some messages showing your support for the show:

http://boards.nbc.com/nbc/index.php?showforum=90

- Here’s a form where you can send your own message to NBC:

http://effusiondesign.com/save-chuck.html

- Go to Digg and digg up all the Save Chuck articles you can find with the search feature. Search for things like ‘chuck renew’, ‘save chuck’, etc

- Buy the DVDs and merchandise from NBC’s site (the store works too but the decision will be made by Friday, the site gets the numbers to NBC quicker)

- Send letters to NBC (Chuck himself gives you some addresses in this post)

The problem, of course, is that observing the user’s interactions requires Javascript. Javascript requires a browser, and spammers aren’t using browsers to fill out forms on the web, are they? Sure, if interaction-based verification systems were varied enough, it would easily hinder the average spammer’s ability to spam, but it would be quite trivial to overcome this.

No matter how you arrange it, in the end the spammer not only can read the Javascript code to see what’s going on, but he can mimick it’s responses to the server.

Let’s say this hypothetical spambot detector told the server it’s result in a hidden HTML field. Easy, spammer fills the field with the expected value. OK, let’s say the detector sent an AJAX call to the server once it was sure this wasn’t a spammer, but oh yeah the spammer can send AJAX calls too.

The reason CAPTCHAs are effective is because the server withholds a bit of information that the user must figure out from the image. With any sort of behavior-based (and thus user-side script-based) CAPTCHA, there really isn’t any information you can hold back. Worse yet, how would the spambot detector vary the situation for each unique page access? It couldn’t.

There’s a reason why the brightest minds in the CS field have been only tweaking the current model, because it’s the most effective (and possibly only) way to stop spam. Even if it’s only temporary. So I don’t think we’ll be able to get rid of “human checks” of some form or another any time soon.

That being said, the field has been shifting substantially toward knowledge based CAPTCHAs like the ones mentioned earlier in the article. Case in point is the forums for the open source game Cube, where you are asked a question like “what color is the sky?” Without being prepared for this particular question, a computer would not know the answer was “blue” unless it was capable of understanding the concept of a sky, as well as being familiar with the sky itself. With a sufficient number of questions chosen in a uniformly random way, this is a good way to stop spam. Also, the efficiency of this model continues to increase linearly with the amount of questions your system can ask: if you have 10,000 questions (and thus 10,000 correct answers), the spammer will need to prepare his bot to answer a great deal of these questions.

Furthermore, unlike letter-based CAPTCHAs, you can assume that a user is a bot with much less evidence. Honestly, assuming the user can read the language of the questions and that they are truly common knowledge questions, a user shouldn’t need to request a new question more than twice, right? So why not limit it to two. This would mean that out of those 10,000 questions in your system, the spammer would have to prepare 5,000 responses to 5,000 known questions to have a reasonable chance at getting through.

Naturally, this model depends on the secrecy of the questions. Any open source CAPTCHA system could be easily cracked since the spammer could just grep through the source to pull the questions and corresponding answers out of the code. This really is the downfall of all set-based (or non-random) CAPTCHAs.

The article also mentions something I hadn’t heard about before: using a hidden field to trick spambots into filling it, considering no human could fill a box they couldn’t actually get to. The author dismisses this too quickly. It would be trivial to come up with a system where the server randomizes the name and order of the hidden field and the genuine message field. The server would not indicate inside the web page which was the correct one, but would instead keep that information in session storage for each user. The main CSS file for the page would be dynamic, and would provide a rule matching the field indicated in the session information, such that the hidden style is only applied to the field which is supposed to be hidden per any given request.

This can be circumvented, but not without quite a bit of work. The spambot would have to parse and apply the CSS to the input fields to determine which ones are hidden. Go ahead and add some more layers of obfuscation, such as providing that rule in one of 3-4 different CSS files, then make the system capable of doing this with _any_ CSS file, and perhaps have multiple hidden fields and you’ve got a pretty tough system.

I think the *real* key to reducing illicit CAPTCHA solutions is to make the process as varied as possible among sites. Just as operating system monoculture promotes the spread of computer virii, CAPTCHA monoculture promotes the spread of illicit solutions.

http://ubuntuforums.org/showthread.php?t=966939

Ubuntu Hardy plus all the extra packages I’ve installed weighs in at just under 6 gigs, leaving too little headroom to go and download the Intrepid Ibex upgrade packages without some tricks.

I went ahead and moved my /var/cache/apt folder onto my storage drive and symlinked it into place. Apt was fine with this of course so I went ahead and launched the Kubuntu upgrade mechanism (opening Adept, fetch updates, “Version Upgrade”).

Now, I’ve never *ever* had one of these automated installers work properly, but it’s always been easy to continue the install process after it inevitably decides to abort because it failed once downloading a single package. It usually leaves the changed /etc/apt/sources.list and all the downloaded packages, so I just pop open a term and type ‘sudo apt-get dist-upgrade’. Upgrade process continues flawlessly until it’s done.

This time was a little more nerve-wracking. Apparently they “improved” the upgrade tool to immediately start reverting the system (after you click “OK” to the package download failure dialog) to the Hardy configuration (though they keep the downloaded packages for you). This happens without asking you whether you’d like to continue the process on your own.

Naturally, it would make sense for them to put a tiny itty bit of fucking sanity by retrying failed packages a few times, but instead they put more work into having it fail smoothly.

So it first reverted the Apt changes, THEN asked me if I was sure I wanted to abort the upgrade, you know, considering I never wanted to stop it in the first place. I did the smart thing and killed that fucker manually without clicking “No” to continue the upgrade so it wouldn’t revert anything else (maybe it would go back on it’s promise and delete the gigabyte of packages I just waited for 2 hours to download!!– not my connection, btw, Ubuntu’s archive servers start serving at full speed (500K/s) but after awhile it wobbles between 50K/s and about 1K/s).

So I had to go and manually fix my /etc/apt/sources.list, replacing ‘hardy’s with ‘intrepid’s all over. Finally, an apt-get update and dist-upgrade got the process rolling again.

This will be the last attempt to use these piece of shit auto-upgrade tools. When I upgrade from Ibex to Jaunty, it will be manual.

I’ll tell you the story of me trying to upgrade Mono to version 2.4 and KDE to 4.2.2 on my Ubuntu 8.04 LTS box. Well there actually isn’t a story unless you want to hear about me searching for answers that don’t exist. I could compile Mono if I really wanted to, but I really don’t. KDE, though, is a royal pain in the ass, just barely easier than building the new modularized X server packaging. No way.

It’s rather stupid that there don’t seem to be ANY packages available for a LONG TERM RELEASE, not even from third party or backports. These are major pieces of software which have very fast release cycles- they deserve to be supported on Canonical’s current flagship Ubuntu.

In the case of Mono only version 2.0 is supported in Ibex iirc, because Mono has been throwing out releases almost as if they were trying to catch up with something.

sigh Fuck it I guess I have to upgrade. Grr.

Unfortunately it is, from what I gather, impossible to disable these dialogs. Microsoft provides an API for adding an application to an exclusion list, though. But this only removes the Send button!! And of course that API is Win32-specific and thus must be used via PInvoke. I implemented this, though it’s not a solution, just to avoid having any stupid reports sent to MS.

There is another solution which works for all non-CLR exceptions, involving setting an exception handler for unhandled exceptions (a mouthful for sure), but this requires added code inside the test unit application (compiled by the Fera compiler), so this is not really a doable solution.

So for now, I must keep my enter key finger ready to dismiss the mess every time I run the test suite.

The model chosen is similar to AspectJ and other AOP languages. You define member matching patterns (called ‘pointcuts’) and advice in the form of join methods or error conditions within special types called ‘aspects’. Aspect classes can inherit normal classes or aspects (but classes cannot inherit aspects). Aspects cannot be instantiated directly, as they are singleton classes. All aspects have an Instance property which retrieves this singleton instance.

All pointcuts and advice are named like any other CIL member. Pointcuts can match methods, nested join points in methods, fields, indexers, properties, events and other pointcuts. Join methods reference these pointcuts (or create them anonymously), and attach a join target (one of ‘in’, ‘before’ or ‘after’).

Methods can also define nested join points with the ‘join’ keyword, which can be matched by pointcuts like anything else. These join points have signatures just like methods, and data is passed from the method to the join point for consumption by join advice. Example:

public void Blah ():
    join Inner (int)
{
   join Inner (3);
}

A pointcut which matches the Inner nested point looks like this:

pointcut BlahInner {
   method void %.Blah().Inner (int);
}

The % character is used as a wildcard within pointcut match clauses. When targetting properties/fields/events with a pointcut, you can match individual events for those types. Field and property accepts ‘get’ and ‘set’, events accept add/remove. The parameters specified in method/indexer matches have special properties. You can use the ‘…’ symbol as a parameter to specify that there is zero or more arbitrary parameters. You can only have one ‘…’ in a pointcut match definition (this isn’t an arbitrary limitation, parameter matching becomes very ambiguous when this is not the case). The same pattern matching is available for the types of the parameters, but parameters using type pattern matching cannot be captured within join advice.

A join advice definition looks like this:

join (int i):
   BlahInner (int i)
{
   // do something here
}

This particular join matches all items of the BlahInner pointcut, adding a more specific version of the parameter list which overrides the one described within the BlahInner pointcut. I say “more specific” intentionally, you cannot attach an overridden parameter list which would not have been matched by the original pointcut. As you see, a name is attached to the int parameter specified in the BlahInner pointcut. This is called parameter capturing. The name (which is the same as the real parameter ‘int i’) is linked so that calls to the join from items matching BlahInner are provided with the method’s first int argument.

You can also define error/warning advice which causes compile-time errors/warnings respectively when a pointcut matches something. It is also possible to filter these errors based on the caller information.

All types/members created for aspect-related stuff is marked with attributes from the Fera.Runtime.Aspects namespace. Aspect classes are marked with AspectAttribute, Pointcuts are generated as constant null CIL fields with PointcutAttribute and a set of PointcutMatchAttributes. Join methods become normal methods with JoinAdviceAttribute and a set of PointcutMatchAttributes.

I’ve got the syntactic elements defined within the Fera grammar, and the IR representation contains the proper structures to support them. Next I need to work on the resolution of name patterns as well as resolving non-pattern builtin types into their fully qualified equivalents. Then I must create the aspect application code, which, to do efficiently, will probably require having a list of the aspects available for when each method is compiled. (that way you don’t start with an join() method and then search the entire type system to find items which match it, the opposite is much more efficient). More info later.

First, CIL represents “protected” blocks via a special metadata table associated with each method body. Each protected block directive is an entry in this table. It’s first worth noting that each entry can only specify a single handler, and that handler must be one of catch, finally, filter, or fault. C# only exposes catch and finally, though it’s possible they may make use of the other two in some manner. Each try block is defined by a range of instructions, specified by their offsets. The ‘start’ part of the try block is the first protected instruction in the block, while the ‘end’ part of the try block is the instruction immediately proceeding the last instruction in the block. This is important to note, as setting the end instruction of a try block to the tailing ‘leave’ instruction which routes control flow to where it should go will result in verification errors from peverify/ilverify, and may even cause the runtime to emit an InvalidProgramException.

In addition, it is illegal to have more than one type of handler per try block, meaning something like:

try {
   // x x x
} catch (Exception) {
   // x x x
} finally {
   // x x x
}

must be split into a pair of try blocks like this:

try {
   try {
      // x x x
   } catch (Exception) {
      // x x x
   }
} finally {
   // x x x
}

Not doing this results in an InvalidProgramException when the compiled program is run.

Another quirk is the behavior of the ‘ret’ instruction within a protected block. If used, the finally handlers will not run (and more than likely you’ll get runtime complaints, in addition to verification errors). When returning inside a protected block, the ‘leave’ instruction should be used, with a target instruction which is at the tail of the method. When ‘leave’ executes, control is transferred to the finally block first.

If the method you’re compiling has a return value, the value to return must be saved in an intermediate variable, because transfer to the finally block will clear/change or otherwise mess up the stack. So to put these ‘return’ requirements together, it is necessary for the code controlling the method compilation to be aware of whether the method contains any try blocks which contain ‘return’ statements, unless you don’t mind generating some useless instructions that are unreachable when there aren’t any ‘return’ statements that use it.

On another note, I implemented a new part of the code flow analysis which checks for paths that don’t return when the method return type is not void. This doesn’t require dominators or anything crazy, just a basic code flow graph. I just winged it really but the algorithm’s time efficiency is O(nb) where n = the maximum depth of the code flow graph (ignoring recursions) and b = the average number of branches in a boundary block. The algorithm employs batch processing instead of recursion, so the memory requirements are very slightly higher, namely O(2n + b) worst case. It might be more efficient to use recursion, but managing to avoid following loops would be more difficult, and it would be harder to track for debugging. I’m happy with the one I’ve got, it required very little work to get working properly (causing no regressions even!)

I’ve managed to get more than 100 of fera’s testcases to pass, now! The exact number is 125 / 127 testcases (the remaining two are generics1 and doexits1, both features are not implemented yet). I finished implementing variable parameters and a lot of constructor / object-oriented stuff like base/this constructor calls.

Since it was inherently easy thanks to the way I built the compiler, it is currently possible to call the base version of a virtual method via the constructor-like base() syntax:

public override method void Write (string s):
   base (s)
{
   // x x x
}

This is less to type than using ‘base.Write (s)’ but is limited to situations where the base version should be called first. It only works for methods that return void right now, the compiler emits an error if this is not the case. Also you can only use ‘base’, the compiler doesn’t let you do ‘this’. I’m still on the fence about whether to keep this.

By the way, this isn’t new but I haven’t mentioned it in any other posts: Fera defines constructors and destructors like so:

public construct ()
{
}

public destruct ()
{
}

This is in sharp contrast to almost every other C-family language which make you type the class name of the class for the method name when defining constructors. Not only does this make typing out a new constructor take less time in most scenarios, but no distraction is necessary if you aren’t sure exactly what the class name is your looking at or you forgot the nuances of it’s naming.

The first is the concept of a loop with a condition somewhere in the middle of it’s body. This can already be done in C# (and indeed any language with a while loop) with:

while (true) {
   // x x x
   if (condition) break;
   // x x x
}

In Fera, you can do the same thing with a little less typing and a bit more style:

do {
   // x x x
   break when (condition);
   // x x x
}

This also allows for the programmer to perform infinite loops, though the compiler will emit a warning, letting you know that you’ve forgot to put in some kind of condition for your loop. You can also use the ‘when’ construct with ‘next’ and ‘continue’ where ever they can be used. The compiler simply reverses the construct, so ‘next when (x);’ turns into ‘if (x) next;’.

Originally I went with ‘break if (x);’. Some languages allow if() tails on any instruction, but they are usually not whitespace-agnostic. I looked at this case to decide against it:

if (x)
   doSomething ();
else
   doSomethingElse ()      // OOPS! forgot my semicolon!

if (x)
   doSomethingTotallyDifferent ();

Forgetting semicolons happens all the time, and this error wouldn’t even be acknowledged by the compiler, making this bug very hard to find. Adding the ‘when’ keyword avoids this ambiguity, and fits in nicely with the do{} loop syntax. I’ll be further investigating whether it’s a good idea to allow this for all instructions (or most anyway).

There are some specific error conditions for the infinite ‘do’ loop; for one, the construct:

do;

causes a compile-time error, not because we disfavor pure infinite loops, but rather because it looks retarded. Also, only certain instructions are allowed to be put as the action of an infinite do loop. Curly brace groups, standalone expressions which have known/unknown side effects, conditionals, or loops are allowed. So this construct ‘do echo “annoying!”;’ is thankfully not allowed.

If you want a pure infinite loop, just use ‘loop;’. This will always emit a warning, telling you it should never be used in production code (unless you explicitly ignore it, or elevate the warning to be treated like an error).

Another type of loop (which is not implemented yet by my compiler) is do/exits.

var int x = 0;
var int y = 3;
do {
   // x x x
} exits {
   case x > 10:
      // x x x
   case y < 0:
      // x x x
}

This is syntactically very different from similar constructs in other languages, as it reuses the switch() syntax and logic and tosses it in with the do loop. The result is a pretty unobscured syntax that is easy to remember once you’ve learned the nuances of both.

Using ‘break’ in this loop will cause the exits to be evaluated, before control returns outside the loop. Using ‘next’ will also cause the exits to be evaluated, except that a lack of any exits will result in a new loop iteration. Within the ‘exits’ block, the ‘continue’ and ‘break’ work exactly like they do in switch/branch.

Owing to the fact that we use ‘next’ for loops and ‘continue’ for switches, you can use ‘next’ within an exit block to have the loop return to iteration. This is a unique property among the loop types that Fera offers, something which must be done with if() in C#.

Naturally the test casing marathon continues. Currently there are a total of 90 tests within the test suite, and 71 of them pass. 10 are expected fails, and the other 9 are regressions. I tested many of the earliest work on this project by hand, so now that I’ve added testcases for them, I discovered many bugs in the Fera grammar and of course the compiler itself.

One of the most perplexing problems was something I thought was happening in the compiler. The problem was happening in the do/while testcases which used a compound condition like so:

var int x = 0;

do {
   // x x x
   ++x;
} while (x > 0 && x < 5);

This particular formulation ensures that the compiler isn’t creating the FWhileLoop instruction which represents the loop with PostCondition = false. Doing so would result in the loop behaving like a while() loop, in which case it would never do the loop action.

The problem was that it was running infinitely, and there is a Console.Write(x) within it, which resulted in an infinite sea of digits across the console window.

I quickly figured out it was the compound nature of the condition, so I turned my attention to test ‘if3′, which tested the if() statement with a similar condition. It happened to be ‘x == 3 && y == 4′. The testcase failed because the action of the if() never runs though it should.

Inspecting the CIL showed the comparison code to be something like this:

0  ldloc.0     ; load x to the stack
1  ldc.i4.3    ; load constant zero to the stack
2  dup         ; save the top value
3  brfalse 8   ; skip the second condition if false
4  pop         ; pop the unused save value
5  ldloc.1     ; load y
6  ldc.i4.4    ; load 4
7  ceq         ; compare equality
8  nop         ; label

Naturally, there should be a ceq inserted at #2. The way it is now, the first condition would always end up being true. It looked like the FComparison expression operand was not outputting ‘ceq’. However upon inspecting that code it was fine and I was able to verify that it in fact was running the branch that emitted the instruction.

Only after much head scratching did I realize that the ‘ceq’ was there, but was far later in the body then it should be. At a hunch I checked the token listing for the file being compiled and sure enough, it was treating the expression like this:

x == (3 && y == 4)

Which would cause a more complete compiler to yelp but not here. So the tree of expression operands looked like:

FComparison
 | FVariableAccess
 | FBinaryBoolean
 |  | FConstant
 |  | FComparison
 |  |  | FVariableAccess
 |  |  | FConstant

I had to add a new level in the expression grammar called ‘CValue’ which is comprised of a ‘Value’ term and an optional set of ComparisonTails. Binary logic operations are handled in the level above it, causing comparisons to be treated as children of the binary logic operators. The tree thus became:

FBinaryBoolean
 | FComparison
 |  | FVariableAccess
 |  | FConstant
 | FComparison
 |  | FVariableAccess
 |  | FConstant

Pretty.

I wanted to expand this to support more than just the basic latin alphabet, hoping it would be as easy as reformulating this expression into character classes, now that Grammatica 1.5 apparently supports unicode regular expressions. First I tried something like:

IDENTIFIER = <<[[:alpha:]][[:alnum:]]*>>

However it appears that Grammatica entirely ignores this type of structure, instead treating it like a character set composed of :s and the letters a, l, p, h, a, etc. So I found out about the \p{Class} formulation for property classes…

IDENTIFIER = <<[\p{L&}][\p{L&}]*>>

The \p formulations like \p{L&} for these weren’t working until I consulted Java’s own set of property classes, which list Alpha and Alnum, so it turned into:

IDENTIFIER = <<[\p{Alpha}][\p{Alnum}]*>>

This made it past the grammar build, targetting .NET for the tokenizer code. But when the compiler runs, from all appearances I gather that it is using .NET’s regular expression library, which does not accept {Alpha} and {Alnum} but prefers the Unicode block names instead like {L&}.

At a prime moment of head scratching I came up with a solution (though it’s not as pretty as the formulations above:

IDENTIFIER = <<[\p{Ll}\p{Lu}\p{Lt}][\p{Ll}\p{Lu}\p{Lt}\p{Nd}]*>>

This makes it through grammar compilation and parsing, matching the correct input. It matches any alphabetic letter (upper, lower, title case) as the first character, and any alphanumeric character for the rest.

The inability to use the Java style classes looks like a bug/oversight in the C# port of Grammatica, unless I miss a more elegant way to pull this off?

In any case it works, so that’s what I’m using. Grammatica has a very long release cycle and they *just* released version 1.5. It’s unlikely there will be a new version for quite awhile. So for anyone who needs Unicode-level identifier tokens, your welcome!

(pass)    decl-field1 (Declarations: Field (test 1))
(pass)    decl-field2 (Declarations: Field (test 2, verified result))
(regress) decl-property1 (Declarations: Property (test 1))
          Caught exception 'Object reference not set to an instance of an object.'
(regress) decl-property2 (Declarations: Property (test 2, verified result))
          Caught exception 'Object reference not set to an instance of an object.'
(pass)    helloworld (Hello, world!)
(pass)    echo1 (Echo (test 1))
(regress) echo2 (Echo (test 2: conversion to string))
          Unexpected program console output (padded to align to [U]nexp...):
          {no output}
(pass)    partial-types1 (Partial types)
(pass)    variables1 (Variable support (simple definition))
(pass)    variables2 (Variable support (initialization))
(pass)    variables3 (Variable support (assignment))
(pass)    variables4 (Variable support (variable as instance))
(fail)    field-get1 (Field access (test 1))
          Caught exception 'OpCode ldsfld have null operand'
(fail)    field-set1 (Field assignment (test 1))
          Compile was not successful (1 errors)
(fail)    field-set2 (Field assignment (test 2, verified result))
          Compile was not successful (1 errors)
(fail)    property-get1 (Property access (test 1))
          Compile was not successful (2 errors)
(fail)    property-set1 (Property assignment (test 1))
          Compile was not successful (1 errors)
(regress) mutate1 (Mutation (++x))
          Caught exception 'Operation is not valid due to the current state of the object.'
(pass)    methodcall1 (Method call (no parameters, void return))
(pass)    methodcall2 (Method call (1 parameter, void return))
(pass)    methodcall3 (Method call (1 parameter, throw-away return))
(pass)    methodcall4 (Method call (1 parameter, used return))
(pass)    methodcall5 (Method call (incorrect parameters, missing 1))
(pass)    methodcall6 (Method call (incorrect parameters, 1 extra))
(pass)    bool-constants1 (Boolean constants (true))
(pass)    bool-constants2 (Boolean constants (check))
(pass)    bool-not (Boolean unary operator (!))
(pass)    bool-and (Boolean binary operator (&&))
(pass)    bool-or (Boolean binary operator (||, test 1))
(pass)    bool-or2 (Boolean binary operator (||, test 2))
(pass)    bool-or3 (Boolean binary operator (||, test 3))
(pass)    invalid-standalone-expr1 (Invalid standalone expressions (x + 3))
(regress) invalid-standalone-expr2 (Invalid standalone expressions (MethodWithoutSideEffects()))
          Unexpected program console output (padded to align to [U]nexp...):
          success

(pass)    imports-structure1 (Imported Namespaces (unqualified type for method return))
(pass)    imports-structure1 (Imported Namespaces (unqualified type for method return))
(pass)    imports-imperative1 (Imported Namespaces (unqualified type for variable definition))
(pass)    switch1 (Switch test 1 (case variable scope))
(pass)    switch2 (Switch test 2 (dynamic cases))
(pass)    invalid-type-variable (Unknown type (variable))
(pass)    missing-variable-initializer (Missing variable initializer (var x;))
(pass)    valuetype-instance1 (Passing by-value types as instance parameter (int x; x.ToString ()))
(pass)    valuetype-instance2 (Passing by-value types as instance parameter (addressof parameter))
(pass)    valuetype-constant-instance1 (Passing by-value constants as instance parameter (38.ToString ()))
(pass)    arrays1 (Arrays (array def/construction))
(pass)    arrays2 (Arrays (element assignment))
(pass)    arrays3 (Arrays (element access))
(pass)    arrays4 (Arrays (use of Length))
(fail)    variable-params1 (Variable parameters (basic test 1))
          Compile was not successful (1 errors)
(fail)    generics1 (Generics (test 1))
          Compile was not successful (1 errors)

I also implemented translation support for all errors and messages in the Fera compiler (Fc). Here’s a sample run of a testcase in my (probably incorrect) Google Translate-based Spanish translation:

C:\fera>fc invalid-standalone-expr2.f /lang:es

invalid-standalone-expr2.f(13,4): Advertencia: 'FeraLanguageTest.Program.MethodWithoutSideEffects' El método no tiene efectos secundarios y devuelve un valor. Este uso del método e
s, probablemente, no.

Compilación éxito (invalid-standalone-expr2.exe): 1 advertencias

Mmm fun. Of course I also implemented a pseudo translation which makes Fera’s messages super brief and concise, useful for people who know what they’re doing and want the compiler to yap as little as possible.

So I did some of that the last couple of days, fixing a critical scoping bug for resolving imported types on declaration entries and implementing arrays (which went so smoothly that I had to pat myself on the back with this blog entry).

I also implemented inspection of method side effects, culminating in this warning which appears whenever you use a side-effect-free method as a standalone expression:

invalid-standalone-expr2.f(13,4): Warning: The method ‘FeraLanguageTest.Program.MethodWithoutSideEffects’ does not have side effects and returns a value. This usage of the method is probably not intended.

I’m developing the code in a very agile/test-driven way. Here is the Fera testcase suite results for today:

(pass) helloworld (Hello, world!)
(pass) echo1 (Echo (test 1))
(pass) partial-types1 (Partial types)
(pass) variables1 (Variable support (simple definition))
(pass) variables2 (Variable support (initialization))
(pass) variables3 (Variable support (assignment))
(pass) variables4 (Variable support (variable as instance))
(fail) mutate1 (Mutation (++x))
Caught exception ‘Operation is not valid due to the current state of the object.’
(pass) bool-constants1 (Boolean constants (true))
(pass) bool-constants2 (Boolean constants (check))
(pass) bool-not (Boolean unary operator (!))
(pass) bool-and (Boolean binary operator (&&))
(pass) bool-or (Boolean binary operator (||, test 1))
(pass) bool-or2 (Boolean binary operator (||, test 2))
(pass) bool-or3 (Boolean binary operator (||, test 3))
(pass) methodcall1 (Method call (no parameters, void return))
(pass) methodcall2 (Method call (1 parameter, void return))
(pass) methodcall3 (Method call (1 parameter, throw-away return))
(pass) methodcall4 (Method call (1 parameter, used return))
(pass) methodcall5 (Method call (incorrect parameters, missing 1))
(pass) methodcall6 (Method call (incorrect parameters, 1 extra))
(pass) invalid-standalone-expr1 (Invalid standalone expressions (x + 3))
(pass) invalid-standalone-expr2 (Invalid standalone expressions (MethodWithoutSideEffects()))
(pass) imports-structure1 (Imported Namespaces (unqualified type for method return))
(pass) imports-structure1 (Imported Namespaces (unqualified type for method return))
(pass) imports-imperative1 (Imported Namespaces (unqualified type for variable definition))
(pass) switch1 (Switch test 1 (case variable scope))
(pass) switch2 (Switch test 2 (dynamic cases))
(pass) invalid-type-variable (Unknown type (variable))
(pass) missing-variable-initializer (Missing variable initializer (var x;))
(pass) valuetype-instance1 (Passing by-value types as instance parameter (int x; x.ToString ()))
(pass) valuetype-instance2 (Passing by-value types as instance parameter (addressof parameter))
(pass) valuetype-constant-instance1 (Passing by-value constants as instance parameter (38.ToString ()))
(pass) arrays1 (Arrays (array def/construction))
(pass) arrays2 (Arrays (element assignment))
(pass) arrays3 (Arrays (element access))
(fail) arrays4 (Arrays (use of Length))
Caught exception ‘Unknown error’
(fail) variable-params1 (Variable parameters (basic test 1))
Compile was not successful (1 errors)
(fail) generics1 (Generics (test 1))
Compile was not successful (1 errors)
Press any key to continue . . .

As you see, there’s still some bugs in arrays and variable parameters / generics are totally broken. As a nod to test driven development: the mutate testcase fails because of the changes to the Assignable expression model, I wouldn’t have known without this great unit test framework!

I’m doing it purely for educational purposes right now, but if it turns out good maybe I’ll release it. It’s very similar to C#, but it just outright drops most of the historical baggage that it brought along from C.

The best example of this is the ‘switch’ construct in the language, which is something that pisses me off about C#. In C# switch statements you are required to place a ‘break’ instruction at the end of each case, with the exception of cases with no instructions, which fallthrough into the next case.

Example:

switch (x) {
 case 0:
 case 1:
   int y = 4;
   Console.WriteLine ("x is either 0 or 1");
   break; // this MUST be here
 case 2:
   int y = 6;   // compile-time error, 'y' already exists in the current scope
   Console.WriteLine ("x is 2");
   break; // this TOO must be here
}

Yes, even that last break has to be there, supposedly to make sure a developer never adds a case without adding the ‘break’ between the old last case and the new case. If you think about the logic of that it’s pretty retarded, and really is only there to make it look like Microsoft has improved it. As you see with the multiple definitions of ‘y’, the whole inside of a switch statement is within the scope it is placed in, meaning in order to define similarly-named variables within multiple case statements, you must place {}s around the cases like so:

switch (x) {
 case 0:
 case 1: {
   int y = 4;
   Console.WriteLine ("x is either 0 or 1");
   break; // this MUST be here
 } case 2: {
   int y = 6;   // compile-time error, 'y' already exists in the current scope
   Console.WriteLine ("x is 2");
   break; // this TOO must be here
 }
}

And boy, doesn’t that look retarded. All the current IDEs get confused too when you do this, changing the indent on lots of stuff while you hammer out the curly braces.

So with that said, here’s Fera’s interpretation of the switch statement.

switch (x) {
 case 0:
 case 1:
   int y = 4;
   Console.WriteLine ("x is either 0 or 1");
 case 2:
   int y = 6;
   Console.WriteLine ("x is 2");
}

As you see, each case statement is given it’s own variable scope, and a break at the end of each case is implicit. To fallthrough to the next statement, you use the ‘continue’ construct like so:

switch (x) {
 case 0:
 case 1:
   int y = 4;
   Console.WriteLine ("x is either 0 or 1");
   continue;
 case 2:
   int y = 6;
   Console.WriteLine ("x is 0, 1, or 2");
}

The ‘break’ construct works like it does in C# and other languages, using it causes the instruction pointer to exit the whole switch statement, allowing the next instruction to execute.

Another interesting thing about Fera’s switch statement is that it’s totally dynamic. The compiler is (well, isn’t yet but will be) capable of determining the static nature of the case statements for a given switch statement, so using static cases will allow the compiler to turn it into a constant time branch (CTB). But if it can’t do that it will just compile it as a simple cascading set of if/else branches.

int x = 3;
int y = 2;
switch (x) {
 case y + 1:
 case y + 2:
   int y = 4;
   Console.WriteLine ("x is 1 or 2 more than y");
 case 2:
   Console.WriteLine ("x is 2");
   break; // this TOO must be here
}

Naturally there are times when you want to ensure that the compiler can use static analysis to improve compilation of your switch. For this purpose, Fera provides the ‘branch’ construct:

branch (x) {
 case 0:
 case 1:
   Console.WriteLine ("x is either 0 or 1");
 case y+1:  // this causes a compile-time error
   Console.WriteLine ("x is 1 more than y");
}

This doesn’t guarantee that the construct will compile to a constant time branch any more than C#’s does, but at least it gives the compiler a chance while making sure the programmer doesn’t get any fancy ideas and use dynamic expressions for it’s cases.

So there’s tons more stuff I’m working to implement, but this is a particularly interesting bit which is already mostly implemented. I think I’ll post a few more blogs with info about some of the other features, like contract-based design, aspect-oriented programming, advanced language-level features and more.

http://www.newsweek.com/id/169192?digg=1

But I have a crazy theory of my own. What is Obama isn’t the Antichrist– what if the battle WAS the election, and Jesus Christ has already won. Yes that’s right, what if Obama was the resurrection of your lord? Is it so hard to believe because he’s black? Do you think that God sees the white man as his son when he made black first? Africa is the cradle of human life.

So, assuming Obama is actually Christ, how do we define the armies? Perhaps the Republicans are the armies of the “Antichrist”, being the intention of ill will toward the world. After eight years of degrading human rights and terrorizing the world under the guise of it’s protection, I think it fits. Maybe the end of your world of suffering is already coming to pass. Maybe Obama will go on to unite us all in preparation for this next thousand years of peace and prosperity.

I’m not religious, but that’s a good thing. I’m not clouded by idealogy, only principles. And tell me that hoarding our prosperity in the US is RIGHT and I’ll show you a greedy son of a bitch. People in hundreds of countries starve to death everyday, their children die from easily curable diseases, and the best the conservative Christian group will do is send a dollar a day to them.

Jesus had a saying for this: “Give a man a fish and he’ll eat for a day, teach a man to fish and he’ll eat for a lifetime”. Sending money isn’t the solution. We have to honestly and seriously help our neighbors as we would like to be helped if it were us in that situation.