There is still another month of chill to get through, but signs of spring are now all around our house in Canberra! Top row: magnolia tree and almond tree; bottom row: liquidambar tree. The birds are starting to come back too. (Photos taken on 27 August 2016.)
Everything that is old is new again. That’s the feeling I get when I look at Spark, which I learned is one of the fastest growing Apache projects in the big data space. There is remarkable similarity in the underlying architecture between Spark and that of a Massively Parallel Processing (MPP) Database like Greenplum or … More Apache Spark vs MPP Databases
The Kalman Filter is one of the more useful tools in data science, but while there are a lot of well-written descriptions of the Bayesian tracking technique available online and in technical books/articles, for some reason it’s hard to find a simple derivation of the Kalman Filter from first principles. In this short note, I show how … More A Derivation of the Kalman Filter
The NSW and ACT governments have just announced a ban of greyhound racing starting in 2017 following a Four Corners investigation into animal cruelty in the industry. Taking a long-term view, there can be no doubt that this is a good decision. However, to make sure the decision doesn’t turn into a disaster in the short-term, we … More Life with a Greyhound
Who would have thought a garbage bin can look so interesting? Ice on bin, 16 June 2016, Arndell.
I have never warmed up to the use of “black swan” to signify an improbable but impactful event. The very first swan I saw was a resident at the Australian National University. It was naturally a black swan. For years I saw only black swans — there are many of them living all around Canberra. I saw … More Black Swans, White Swans
The life of a sea squirt has an important lesson for data science. For those who haven’t heard, sea squirts come to life as larvae that swim freely around. In that state, however, they are not capable of feeding so they will soon settle to the bottom of the ocean and cement themselves headfirst to … More Agile Data Science: Start with An Action
Whenever two organisations come together to share data, we have a data integration problem. Mapping of datasets is typically done manually and that can be a labour-intensive and error-prone process. Importantly, the manual data-mapping process doesn’t scale and that is a problem when you want to build an information-sharing network where arbitrary organisations can sign … More Automatic Data Integration using Normalised Compression Distance
I had a chance to visit Westpac in early May as part of our agency’s Know Your Industry initiative. In that two days, we met with leaders from across the Westpac Group and it’s quite amazing to see how the Agile mindset has penetrated every corner of the banking group and become a part of … More Agile Retail Banking
Cinemas everywhere have little hesitation in making movie-goers sit through 15-20 minutes of advertising before each movie. It struck me recently that the amount of eye-rolling one experiences in those 15-20 minutes is actually a pretty good index of a country’s freedom and liberty. In India, one is made to stand and sing the national … More A Cinema-goer’s Guide to Freedom and Liberty
