Disproving the ‘embarrassingly easy’ tribe was, well, embarrassingly easy. It needed a lone IBM engineer, working three months to build a Jeopardy playing computer that would fail to reach the champions quadrant. In this quadrant, champions win the buzzer sixty percent of the time. Disproving the ‘too ambitious’ tribe would need several years of work that culminated in two nationally televised games that are subject of this book. The author also delved into the technical wizardry employed by IBM. Here is the coveted “intellectual property” for aspiring wizards:

• Data sources. Lots of them.
• Data mining algorithms. Highly optimized.
• Neural Network to apply confidence measures to the data mining algorithms.
• CPU cores. Preferably at least two thousand.
• 1 mechanical finger for buzzer. This is very important!
• Cutesy branding. For pandering to the eyeballs masses viewers at home.
• Non-threatening name. Note: Avoid science fiction references.

After you’ve got all these pieces stitched together, you have a computer with the intelligence of a two year old. Optimize, optimize, and optimize some more. Now you’ve graduated to a three year old, which as any parent will tell you, is worse. Keep optimizing and eventually the machine will reach the champions quadrant.

So now that IBM has proven that it can build a formidable trivia game playing computer, what’s next? The author thinks it may be re-purposed as an expert system for use by legal, medical or business “professionals”. Apparently, computers can be used for other stuff besides web browsing, games, and twittering.

Whatevs.

Checkout the book here. Caution: requires reading and may prove to be useful and entertaining.

Ultimately, the bets Corzine placed wound up better than the firm itself. The European debt trades were profitable, though too late for MF Global.

Before Congress on Thursday, Mr. Corzine continued to emphasize how well his trades held up. “As of today, none of the foreign debt securities that MF Global used,” he said, “has defaulted or been restructured.”

“There actually were no losses.”

Source: NYTimes.com

The slot machine rang the jackpot after he was done playing it.

Speaking of gambling….

Chris Fargis thought his big job interview was over. But when the partners at Wall Street upstart Toro Trading finished with their questions, they broke out a deck of cards and a green-felt card table. Mind playing a few hands of poker?

Source: LA Times

I wonder if they have $5 buffets specials on Wall Street to entice the gamblers traders.

Its decision to chop over thirty thousand jobs is laced with irony because for the past decade, hundreds, if not thousands of high-priced consultants have have been milking the agency with the promise of helping it reduce its operating costs. Leave it to the suits to come up with the most obvious, uninnovative, non-solutions. Instead, what the USPS needs is a visionary leader, not a career civil servant, to take charge and lead a business process turn around.

Currently, the biggest revenue generator for the USPS is its junk bulk mail service. Were it not for Super Saver coupon books and other marketing trash, most US residences could go days or even weeks without receiving a letter. If the USPS were to eliminate last-mile delivery for all but the most important mail and have individuals collect their mail from central post office boxes, it could result in a huge saving in fuel costs. By eliminating bulk mail, it could help save the trees and greatly improve the customer experience. Sure, it would lose the easy money but that’s not so bad if it finds alternative revenue sources. Revenue sources that don’t rely on bombarding their customers with unsolicited advertisements. For example:

• On-site business centers ala Kinko’s.
• Mail digitization service (i.e. scan and e-mail correspondence to the recipients).
• Door-step delivery as a premium service.
• Ancillary or third-party on-site services, eg: electronic recycling deposits.
• Electronic Fund Transfers, ala Moneygram or Western Union.
• Co-located retail vendors, eg: Mobile phone vendors, H&R Block, etc
• Fully automated self-service machines (including mailing supply vending).
• Drop-ship warehousing.
• …and many more…

I think the USPS does a great job. It just needs new revenue models to inject a new lease of life into itself. Its massive workforce and prime real estate holdings should be viewed as assets with huge monitzation potential. This could be the time for a historic new start for the agency.

Hookbox (PPT)

For some pension recipients, the deals seem like the way out of a financial crisis. Joseph Serina, a metal-fabrication worker who spent 21 years in the Navy, received $57,450 three years ago from a group of investors in return for promising them $125,280 in pension payments over eight years.

[snip]

The biggest risk for investors is that pension recipients renege on their promises to turn over monthly checks to the investor.

If more people knew how to do basic math and understand interest calculations up front, would fewer of them seek pay day loans??

Though enterprise users with heavy demand and big budgets might be willing to pay for a commercial system and all the tools, support, and service agreements those systems provide, users with more modest budgets or requirements find open source systems more attractive.

– Stonebraker et al, MapReduce and Parallel DBMSs: Friends or Foes? Communications of the ACM Vol. 53 No. 1, Pages 64-71 10.1145/1629175.1629197

In other words:

Open source + easy learning curve + good enough > closed source + sophisticated + performant.

2!!7!4

Yes, that’s a valid expression from the proprietary K programming language ^[1].

From Wikipedia:

Reading from right to left the first ! is modulo division that is performed on 7 and 4 resulting in 3. The next ! is enumeration and lists the integers less than 3, resulting in the list 0 1 2. The final ! is rotation where the list on the right is rotated two times to the left producing the final result of 2 0 1.

Depending on your application, there are at least three ways this can be done in Python:

# Using the collections module
from collections import deque
d = deque(range(7%4))
d.rotate(-2) # this is an in-place rotation

# Using the numpy module
from numpy import roll
roll(range(7%4), -2) # this returns a numpy.ndarray

# Using list slicing and concatenation
r = range(7%4)
r= r[2:] + r[:2]

So which is faster? Rolling, rotating, or slicing?

def method_rotate (x, y=500):
    x.rotate(y)

def method_roll (x, y=500):
    roll(x, y)

def method_slice (x, y=500):
    r = x[y:] + x[:y]

x = range(1e7)
xa = array(x)
xd = deque(x)

%timeit method_rotate(x)
100000 loops, best of 3: 7.83 us per loop

%timeit method_roll(xa)
10 loops, best of 3: 95 ms per loop

%timeit method_slice(x)
1 loops, best of 3: 229 ms per loop

So deque rotation is blazingly fast which is not surprising given that rotation is equivalent to repeated pop-append which is exactly what deques are optimized for ^[2]. Roll and slice-concat are at least three orders of magnitude slower which is probably a testament to the design of deques.

But here is the equivalent slicing and concatenation operation in R ^[3]:

r = 1:1e7
system.time(c(r[500:1e7],r[1:500]) )
    user  system elapsed
  0.184   0.060   0.243

So it would seem that R can rotate an array faster than numpy.roll. Now I know it’s not that much faster and in practice Python would probably come out ahead. But still. R?

Right now I think it would be really nice to go back to the roots of Erlang and add Prolog-like predicate logic to the language, a kind of new mixture of predicate logic plus message passing.

— Joe Armstrong, Creator of Erlang (link)

Software frameworks come into existence when general purpose languages are adapted for a very specific purpose. Like Python has been adapted for web application development via django. Eventually the frameworks reach a critical level of complexity which takes on a life of its own. Consider Java’s Hadoop eco-system as a prime example. Eventually, the status quo gets disrupted by a new (renewed?) language/paradigm. I hope that’s what Joe was talking about.

Green Card Game

I arrived in the summer of 1999 as a seventeen year old college student in the United States. In 2004 I started working full time which paved the way for me to receive an employment based (EB) green card. While the green card application was pending, I got married, settled down, bought a house, had a kid, and paid income and social security taxes.

Finally, in the summer of 2011 we received our green cards in the mail along with a congratulatory letter from the USCIS titled “Welcome to the United States.” After having lived in the US for eleven years and anxiously awaiting the green card for the last seven of those years, I would say the congratulatory letter is a tad belated. But as they say, better late than never.

Currently there are several thousand applications for EB green cards pending a decision. Many of them have been pending for more than the seven years it took for my application to be ajudicated and approved. There is no way of accurately determining which stage an application is in or when it will be adjudicated. It’s a painfully slow and opaque bureaucratic process that can feel like a game of chance with diminishing returns as time wears on.

Those who are still playing the game, know that if you play long enough you will eventually reach the finish line. The end however will feel like it took forever and the satisfaction of reaching it won’t be as intense as the anticipation of reaching it. Probably best to think of it as a new beginning and chug along…

One of the practical differences between Python and Ruby is that the latter supports ‘open classes’. That is to say, a class can be defined several times in different files and the interpreter extends the class each time. Here is a simple snippet that demonstrates this in action:

# define class A with one method
class A
    def foo
        puts "in foo"
    end
end

# open class A and add one more method
class A
    def bar
        puts "in bar"
    end
end

a = A.new()
a.foo
in foo
a.bar
in bar

This behavior allows a Ruby programmer to extend even a built-in type like Fixnum or String. This is a second practical difference between the two languages. In Python, built-in types are immutable. If a programmer wants to extend a built-in, they have to sub-class it.

And that brings us to a third practical difference: mixins. Python supports mixins via multiple inheritance. Ruby supports mixins via modules which is just a namespace for a logically related set of constants, classes, and functions. A module can be included in a class to enhance it. Let’s continue the previous example:

# this is a simple module that encompasses one function
module Baz
    def baz
        puts "in baz"
    end
end

# now we extend class A by including the module
class A
    include Baz
end

# create a new instance of 'A' that has three methods: foo, bar, baz
a2 = A.new
a2.baz # this prints "in baz"!

If you are a pythonista encountering Ruby for the first time then these differences might give you an uneasy feeling. That’s because in Python, this behavior of Ruby is referred to as ‘monkey patching’ and is generally considered unpythonic. But it lends Ruby its trademark flexibility.

These and some other practical differences aside (¹), Ruby and Python are both dynamic, strongly typed, object-oriented languages so they can accommodate the same kind of programming model.

Now, for an object-oriented language to be able to support meta programming, there typically has to be a way to programatically create a new class during runtime. In Python, I’ve previously demonstrated this as the purpose of type callable. Its Ruby equivalent is Class.new:

# create a new class, B, that inherits from A
B = Class.new(A)

So Class.new affords us a way to create anonymous classes at runtime. It is a powerful technique that is used heavily to generate complex classes for the programmer without having to declare them explicitly.

Ruby also has a very handy class, Struct, that allows us to create anonymous classes with user-specified attributes. It is used throughout the excellent PacketFu library (²) which is Ruby’s equivalent of Python’s dpkt (³) from which we cited code snippets in the previous post.

Here is a snippet from PacketFu that demonstrates how Struct is used:

       # EthHeader is being declared by inheriting from an anonymous class.
       # The anonymous class is being created by calling Struct.new and
       # passing it several symbols as arguments.
       # The anonymous class will be generated so that the aforementioned arguments
       # become instance attributes of the EthHeader. In this sense, it is a meta class
       # of EthHeader.
       class EthHeader < Struct.new(:eth_dst, :eth_src, :eth_proto, :body)
                # StructFu is a module in the PacketFu library. It is being used as a mixin here.
                include StructFu

                # initialize the instance attributes here.
                # EthMac and Int16 are classes defined above EthHeader.
                def initialize(args={})
                        super(
                                EthMac.new.read(args[:eth_dst]),
                                EthMac.new.read(args[:eth_src]),
                                Int16.new(args[:eth_proto] || 0x0800),
                                StructFu::String.new.read(args[:body])
                        )
                end
               # ... more stuff follows here ...

Now, the Struct class is implemented in C as part of Ruby's core (⁴).
But let's see how it could be implemented in Ruby itself:

# Let's name our implementation 'Structure'
class Structure

	# The reference implementation includes the Enumerable module
	# so we do it for consistency.
	include Enumerable

	# override the 'new' method of the Structure metaclass.
	# create a new anonymous class with specified
	# symbols as instance variables.
	def self.new (*symbols)

		# the class is created by passing a block to Class.new
		klass = Class.new do

			# iterate through the symbols passed as arguments
			# and create instance attributes from them.
			symbols.each do |sym|
				attr_accessor sym
			end

			# create a class instance attribute 'members' and assign
			# the list of symbols. this attribute is useful for
			# inspecting the class and is also used to initialize
			# an instance. we do this by opening the meta class of klass:
			class << self
				attr_accessor :members
			end
			@members = symbols.clone

			# initialize the instance variables by assigning
			# (optional) program supplied values to them.
			def initialize (*vals)
				if not vals.empty?
					self.class.members.each_with_index do |mem, i|
						instance_variable_set("@"+mem.to_s, vals[i])
					end
				end
			end
		end
	end
end

N.B: This is a working implementation but it is not in full conformity with the reference implementation. Among the things that have been left out for brevity:
* Support for supplying a name for the generated class.
* 'include'ing the generated class in the Structure module itself.
* ... and probably a few other things.

The meta programming begins with def self.new. The self keyword in Ruby is used in a very different sense than in Python. Generally it refers to the meta class of a class. This gives the Ruby programmer the ability to access and modify the meta class inline! In fact meta programming in Ruby is all about managing self (5).

Astute Python programmers will have picked up on two more practical differences between Ruby and Python meta programming by now, namely that Ruby meta classes are anonymous while in Python they are explicit, and secondly, Ruby allows meta class manipulation inline. The latter difference is probably what provoked the troll to dismiss Python as making meta programming hard by forcing them to define the meta class apriori.

Now lets see how this would actually be used:

# define EthHeader by inheriting from the class generated by 'Structure' instead of 'Struct'
class EthHeader < Structure.new(:eth_dst, :eth_src, :eth_proto, :eth_body)
	# implement EthHeader exactly as in PacketFu
end

# instantiate an EthHeader object
header = EthHeader.new
header.eth_proto = 0x0800
# ... and so on ...

So there you have it... our own working re-implementation of Struct written in Ruby.

There are several tools in Ruby meta programming toolbox that I did not touch upon in this post (6). For Python programmers, this serves as a quick overview of meta programming in Ruby. While for Ruby programmers who might already be familiar with the tricks employed here but they may have taken Struct for granted, this hopefully gave them a new appreciation for their favorite factory class.

u can now haz the meta codez. enjoy!

(1) For a more comprehensive treatment of the practical differences between Ruby and Python, see Ruby Python "Power" by Ian Bicking.

(2) PacketFu repository on Google Code

(3) dpkt repository on Google Code.

(4) Implementation of the Struct class in the Ruby source code repository on github.com

(5) Metaprogramming in Ruby: It’s All About the Self

(6) Ruby's Meta Programming Toolbox