by Keven Lin

Clojure + Mathematica

Tue, 03 Nov 2009 16:40:12 -0800

A few people asked how I wrote a webapp using Clojure with Mathematica. The solution is actually quiet simple - Clojure runs on JVM and Java interop (ability to call any Java libraries) works great. Mathematica provides a J/Link Toolkit that allows you to control the Math kernel from Java.

Why Clojure?

You could also use Java and link to Mathematica, but Clojure has the advantages of a functional and dynamic language. I won’t go into the details of Java vs. Clojure. Paul Graham made a good argument for Lisp in Revenge of the Nerds that’s definitely worth reading. Mathematica also includes a Python library for the MathLink library which does not support the newer versions of Python. You could also power Mathematica websites using webMathematica.

2+2 with Clojure and Mathematica

An equivalent of “hello world” program, following is the code performing a simple arithmetic using Mathematica. Note that to run the script, I need to force run JVM in 32-bit mode (using -d32 option) in Snow Leopard because J/Link package does not provide the 64-bit library. To run the script, all you have to do is call java and make sure to include both J/Link and clojure jars.

A Mathematica Webapp

Compojure is a micro web framework that is similar to Sinatra. It is an ideal choice to demo a stock quote webapp get latest stock price using the Mathematica’s FinancialData function.

Result:

Twitter Analytics using Mathematica

Thu, 28 May 2009 20:28:00 -0700

Twitter has been just about everywhere lately.  As its popularity continues to grow, how do you keep track of the expanding list of friends and followers?  As more companies are using Twitter to promote their products, good tools are necessary to manage hundreds or thousands of contacts and analyze their Twitter social graph .

Twitter has an open api, and Mathematica is the best analytical tool out there.  Mathematica’s image manipulation capability also makes it a very nice data visualizer.  So let us plug Twitter in Mathematica, and play with the data.

In the examples, let’s use Wolfram_Alpha (my personal twitter data is not very interesting).  WolframAlpha has just recently launched, and they have been actively posting news and updates with Twitter.  Let’s first look at W|A’s recent tweets.

In[329]:=

In[331]:=

So here we import the user’s timeline, and transform the list of XMLObjects to a list of rules from attribute to value.

Now we can visualize the time and frequency of the user’s recent tweets in a timeline, where x-axis is the date and y-axis is the number of seconds during the day (eg. 30000s = 8:20 am).

In[334]:=

Out[336]=

After seeing Mathematica’s visualizations, we want to do analysis and play around with the friendship data. Let us load up the users Wolfram Alpha is following,

In[338]:=

Out[338]=

In[339]:=

Out[339]=

All friends’ profile data are loaded, and the size is consistent with the number on the website. To apply a filter, say look for the friends with screen name begins with “a”,

In[340]:=

Out[341]=

And put the name to the profile picture.

In[342]:=

Out[342]=

One of most important features on Wolfram Alpha is performing arithmetic and comparisons on objects. For example, try comparing between three stocks GOOG, AAPL and MSFT, or add up your nutrition value of a meal at McDonald’s. I also want to compare between a list of friends, which can be easily done by:

In[343]:=

Out[343]=

Now I’m interested to find out which friends have most followers, by sorting the friends with follower counts, and take top 20 results.

In[344]:=

Visualize of the results with a directed graph. An arrow from user A to user B indicates that user A is following user B.

In[345]:=

Out[345]=

We can also visualize the same list of users in a bubble plot. X - axis is number of followings, y - axis is number of followers, and the size of the image is determined by the ratio of followers / friends (ie, image is bigger if number of followers / number of friends is higher).

In[346]:=

Out[346]=

Because of Wolfram Alpha, most programmers probably have seen what Mathematica can do. Its applications are endless: I have been using it to analyze stock arbitrage opportunities, and we are also interested in analyzing the way people read comics on iPhone.

The examples above were over spare hours (or minutes) over last few nights just for fun. If anyone has any suggestions about Twitter / Mathematica, let me know (@kevenlin). This blog post from WolframBlog will also show you other cool things such as tweet from Mathematica.

Newest addition to the family: Damian Lin function...

Sat, 25 Apr 2009 11:57:00 -0700

Newest addition to the family: Damian Lin

Pair Trading using Mathematica

Fri, 24 Apr 2009 13:09:00 -0700

Pair Trading is one of the ideas that I have been very interested in, but could not find right tool for analysis. Sure, most traders use Excel, but that’s simply too many copying and pasting cells. Patrick showed me the cool things he could do with Mathematica, and I thought that might be the perfect tool for stock analysis.

So what is Pair Trading? In short, it is a market-neutral strategy used by hedge funds and investment bank. Trader first identifies two stocks which prices have moved together in past. When the spread between the stocks widen for any reasons, buy the underperforming stock and short the outperforming one. When the spread eventually converge and the trader will close the positions and profit.

Mathematica has lots of impressive features for stock analysis. For example,

One line of code will grab GOOG prices since 2006 and plot on a chart. Note that I don’t have to write parsers for stock data, because financial data is part of larger set of curated data that comes with Mathematica.

Now let’s use Mathematica on Coca-Cola(KO) and Pepsi(PEP), a classic example of a correlated pair.

As expected, the pair has similar movements in prices over lasts year. Now we plot the price of KO divided by PEP and get the following chart. (Note: Blue Curve is the KO / PEP pair ratio, Red Line in middle is mean, Blue and Yellow Lines are 1 standard deviation, and Red and Green Lines are 2 standard deviation).

The price ratio seems to oscillate around a mean. But we do see when the ratio appears to go below and above 2 standard deviation from the mean. The strategy is to execute a paired trade when the pair ratio is over +/- 2 standard deviation from mean. When the ratio reverts to mean, we will close the positions.

As shown on the chart, the price ratio is currently around +2 standard deviation, now might be a good time to SHORT KO / LONG PEP. For options trading, you would buy a KO put and buy a PEP call that expire on same time.

Trader should be aware of risk of drifting. This happens when the two correlated stock prices start to drift apart. The risk can be controlled if trader closes the pair (and take loss) if their prices do not converge within a time interval (eg. 6 months) or when pair ratio is above certain tolerable line (say +/- 3 standard deviations).

Mathematica has simplified the analysis with stats and charting library and curated finance data. Only 5 lines of code are needed to produce this analysis. It is also worth mentioning Mathematica also has awesome documentation to help you get started. If you have Mathematica, you can download the notebook.

Two reasons why most entrepreneurs fail

Thu, 22 Jan 2009 13:59:00 -0800

“by doing nothing, and by doing the wrong things.” - pg via Hacker News

Family Portrait

Thu, 11 Dec 2008 14:28:29 -0800

Family Portrait

Ingredients for great beer. Taken from Granville Island...

Fri, 21 Nov 2008 15:14:30 -0800

Ingredients for great beer. Taken from Granville Island Brewery.

A classic metaphor from Taleb : A turkey is fed for a 1000 days...

Thu, 23 Oct 2008 21:09:53 -0700

A classic metaphor from Taleb :

A turkey is fed for a 1000 days - every day confirms to its statistical department that the human race cares about its welfare “with increased statistical significance”. On the 1001 day, the turkey has a surprise.

"that code should be seen not as a static thing, like the answer to a math problem, but as an..."

Thu, 23 Oct 2008 09:50:07 -0700

“that code should be seen not as a static thing, like the answer to a math problem, but as an evolving effort to figure out the right question to be the answer to; and that it should thus be written to be easy to change.”

- Paul Graham, on the one thing every software engineer should know

Update: Years are automatically updated. var montharray=new...

Tue, 07 Oct 2008 23:28:00 -0700

Update: Years are automatically updated.

Ack bundle in TextMate - Faster Find in Project

Tue, 07 Oct 2008 21:34:46 -0700

One thing that bothered me with TextMate was the Find in Project function. In searching for replacement, Olivier pointed me to Ack!, which is an excellent tool to replace find+grep.

While doing so, I came across a nifty Ack bundle for Textmate. It’s REALLY FAST. Enjoy!.

Open source library for quantitative finance

Fri, 19 Sep 2008 10:40:42 -0700

Open source library for quantitative finance:

Open source framework for modeling, trading and risk management in finance. Includes Python binding. Despite the market turmoil, anyone wants to start a hedge fund?

(found via http://news.ycombinator.com/item?id=309198)

research done by mechanical turk

Mon, 15 Sep 2008 13:31:00 -0700

research done by mechanical turk:

brilliant! casual research by surveying random population using mechanical turk.

MemCache Flush All (for memcache_client )

Thu, 11 Sep 2008 22:31:13 -0700

Some times we might want to clear memcache without restarting memcache server.

When calling Cache.get / set, we are really using a wrapper module Cache that simplify access to memcache client.

To flush memcache, call CACHE.flush_all

Contact form for iTune support

Tue, 09 Sep 2008 17:00:50 -0700

Contact form for iTune support:

This link is hidden so deeply, it always takes me few minutes to dig it up again.

Ruby include, extend, require and load explained

Wed, 03 Sep 2008 15:07:01 -0700

The Ruby ‘include’ statement references to a named module; it appends features of a module into classes or other modules.

‘extend’ adds the features of a module to one instance (object) at run-time

‘require’ is used for loading nonmodule Ruby sources and binaries. ’require’ is similar to a ‘load’ ,but it will not load a file if it has already been loaded.

‘include’ and ‘require’ are essentially unrelated. You may need to do a require follow by an include to use some externally stored module.

Git - revert / undo local modifications

Tue, 12 Aug 2008 15:35:00 -0700

To undo local modification to file

git checkout -f /path/to/file

To revert a recent commit
git reset HEAD~

Good reference for Git material: Kernel Hackers’ Guide to git

Tai Chi Demonstration by Master Yuan Zhen Wu, 32nd generation...

Tue, 05 Aug 2008 15:59:00 -0700

Tai Chi Demonstration by Master Yuan Zhen Wu, 32nd generation warrior monk from Shaolin Temple. http://shaolintemple.ca

approach on product design

Sun, 03 Aug 2008 17:12:04 -0700

approach on product design:

Paul Buchheit:

What is the humble approach to product design? Pay attention. Notice which things are working and which aren’t. Experiment and iterate. Question your assumptions. Remember that you are wrong about a lot of things. Watch for the signals. Lose your technical and design snobbery. Whatever works, works.

Restart Apache w/ mod_rails on Leopard

Fri, 18 Jul 2008 09:01:41 -0700

sudo apachectl restart