I woke up on New Year’s Eve to the sight of an owl perching on top of my air-conditioner condenser unit outside. Now, I’m used to seeing all kinds of wild lives in the backyard of my Canberra home, but a visit from the Wise in my Singapore apartment, which is a stone’s throw away … More An Owl in the House
Most modern big data platforms support parallel execution of (non-native) code written in languages like Python, Perl, R, and Java. On Greenplum and HAWQ, two massively parallel relational database systems, these facilities come in the form of PL/Python, PL/Perl, PL/R, and PL/Java, which are inherited from PostgreSQL. These programming facilities are useful for a range … More PL/Fortran and PL/C++ on PostgreSQL and Greenplum
Most mechanical watches operate at an accuracy of +/- 12 seconds per day. A COSC-certified chronometer, which is what you can expect to get from quality Swiss watch makers like Omega, operates at -4/+6 seconds per day. For most practical uses, these are perfectly acceptable accuracy measures. To get anything more accurate from a mechanical … More Spring Drive 9R15 Movement Accuracy
While doing a project involving survival analysis a while back, I learned a useful technique for reducing survival analysis to binary regression. The following discussion is adapted from Chapter 7 of Survival Analysis by David G. Kleinbaum. To illustrate how the technique works, consider a small data set with three subjects. Subjects 1 and 3 are … More Survival Analysis using Logistic Regression
Incentive drives behaviour. If you have ever wondered why (brick-and-mortar) grocery retailers historically don’t pay much attention to understanding customer preferences — it’s getting better but things are still moving slowly — then you only have to look at how most of them make their money. It turns out that the main sources of supermarket profits … More Sources of Profit in Retail
Sometimes I can’t help but wonder whether data science as a discipline is still in the dark ages. From my vantage point as a practicing data scientist, I see that too many projects still rely on personal heroics for success. By and large, ours is still a discipline dominated by the need for stars and … More Agile Data Science: Applying Kanban in the Analytics Life Cycle
Most of us have heard of this little puzzle in retail analytics: Assuming we know products A and B are usually purchased together from market basket analysis, how do we place them on shelves to maximise sales? The often-stated “sophisticated” answer is that A and B should be placed far away from each other because … More A Note on Shopping Efficiency and Product Placement
Content-based recommendation engines are typically done in two steps. In the first, a user-preference model is constructed using a set of predefined features. (For example, in the context of retail, we may have features like Price Sensitivity, Promotion Sensitivity, Coupon Redeemer, Calorie Counter, Working Mums, etc.) In the second step, products are mapped, either directly … More A Text-Based and Self-Adapting Product Recommendation Engine
An interesting story I learned recently while reading the excellent book Time’s Pendulum by Jo Ellen Barnett is the history of how we came to have 24 hours in a day, 60 minutes in an hour, and 60 seconds in a minute. Apparently, the 24-hour convention began in ancient Egypt, and the sixty divisions of minutes … More How Come There Are 60 Minutes In An Hour
The Pareto Principle states that, for many events, roughly 80% of the effects come from 20% of the causes. The principle certainly finds expression in grocery sales data and the following diagram is quite typical of what one can observe. In retail parlance, the top product classes or SKUs are called the big heads and … More Big Heads and Long Tails in Retail
Mental Models 4 Life 