The Finance Book Club

1.2 Why Machine Learning projects usually fail

1. Failure rate in financial ML is higher even than in quant finance (and that’s high)

1.2.1 The Sisyphus paradigm

1. One mistake underlies all the main failures
2. Discretionary managers rely on their judgment and intuition
3. They are usually siloed to prevent them from influencing each other
4. Eventually groupthink overwhelms them
5. Boardrooms treat quants like discretionary PMs and ask them to individually produce a unique strategy
6. This backfires because each PhD will frantically search for investment opportunities and eventually settle for
   1. (1) a false positive that looks great in an overfit backtest or
   2. (2) standard factor investing, which is an overcrowded strategy with a low Sharpe ratio, but at least has academic support.

I applaud you for reviewing Advances in Financial Machine Learning. It's going to be helpful to ordinary traders through Chapter 3 at least. In particular, I thought it was very interesting to see the types of financial data he likes and what he doesn't (you'll see what I mean, I don't want to give it away ). Chapter 5 is where I gave in my first time through because I didn't have the ML background to benefit from it. Now I do, but portions will still be a challenge if I'm honest. I had planned to reread this starting in September, but I'll try to follow along now since you're making the endeavor.

It's an amazing book and gives a window into how one of the top hedge funds in the world operates. However, what you see through that window is intimidating.

1.2.2 The Meta-Strategy Paradigm

1. If you have been asked to develop ML strategies on your own, the odds are stacked against you.
2. It takes almost as much effort to produce one true investment strategy as to produce a hundred, and the complexities are overwhelming:
   1. data curation and processing,
   2. HPC infrastructure,
   3. software development,
   4. feature analysis,
   5. execution simulators,
   6. backtesting

3. You are like a worker at a BMW factory who has been asked to build an entire car by using all the workshops around you.
4. This book was written as a research manual for teams, not for individuals.
5. You will learn how to set up a research factory
6. The role of each quant is to specialize in a particular task
7. Teamwork yields discoveries at a predictable rate (note he doesn’t say ‘profitable discoveries’)
8. There is no reliance on lucky strikes
9. These discoveries are the outcome of team efforts where everyone contributes.
10. What do you think has a higher chance of success, this proven paradigm of organized collaboration or the Sisyphean alternative of having every single quant rolling their immense boulder up the mountain?

1.3.1 Structure by Production Chain

This next metaphor is maybe one of the more important revelations in the book club journey.

 

1. Visible gold is an infinitesimal portion of the overall amount of gold on Earth.
2. However gold mining still produces tonnes of ore by refining microscopic flecks from raw material
3. El Dorado was always there . . . if only Pizarro could have exchanged his sword for a microscope. The discovery of investment strategies has undergone a similar evolution.
4. A decade ago it was relatively common for an individual to discover macroscopic alpha (i.e., using simple mathematical tools like econometrics), currently the chances of that happening are quickly converging to zero.
5. The only true alpha remaining is microscopic (markets approaching efficiency. However maybe M should have specified, the only true ‘long-term’ alpha.)

If this claim is accurate then we must consider the possible outcomes for individual retail discretionary traders. They would be delusional to expect long-term gains larger than the risk- free rate.

1. microscopic alpha does not mean smaller overall profits. Microscopic alpha today is much more abundant than macroscopic alpha has ever been in history.

Next: M describes the stations needed in a ‘quant factory’.

1.3.1.1 Data Curators

1. This is the station responsible for collecting, cleaning, indexing, storing, adjusting, and delivering all data to the production chain.
2. Team members are experts in market microstructure and data protocols such as FIX.

1.3.1.2 Feature Analysts

1. This is the station responsible for transforming raw data into informative signals.
2. Team members are experts in information theory, signal extraction and processing, visualization, labeling, weighting, classifiers, and feature importance techniques.
3. For example, feature analysts may discover that the probability of a sell-off is particularly high when:
   1. (1) quoted offers are cancelled-replaced with market sell orders, and
   2. (2) quoted buy orders are cancelled-replaced with limit buy orders deeper in the book. (are they?)

   
   

4. Such a finding is not an investment strategy on its own,
5. A common error is to believe that feature analysts develop strategies.
6. feature analysts collect and catalogue libraries of findings that can be useful to a multiplicity of stations.

1.3.1.3 Strategists

1. In this station, informative features are transformed into actual investment algorithms. A strategist will parse through the libraries of features looking for ideas to develop an investment strategy.
2. The goal of the strategist is to make sense of all these observations and to formulate a general theory that explains them.
3. The strategy is merely an experiment designed to test the validity of this theory.
4. Strategists are data scientists with a deep knowledge of financial markets and the economy.
5. A theory must identify the economic mechanism that causes an agent to lose money to us. Is it a behavioral bias? Asymmetric information?
6. Features may be discovered by a black box, but the strategy is developed in a white box.
7. Once a strategy is finalized, the strategists will prepare code that utilizes the full algorithm and submit that prototype to the backtesting team
8. It would be unreasonable for a book to reveal specific investment strategies.

1.3.1.4 Backtesters

1. This station assesses the profitability of an investment strategy under various scenarios. One of the scenarios of interest is how the strategy would perform if history repeated itself.
2. However, the historical path is merely one of the possible outcomes of a stochastic process, and not necessarily the most likely going forward. Alternative scenarios must be evaluated. Does this mean that M assumes the market is a stochastic (non-deterministic) process?
3. They are data scientists with a deep understanding of empirical and experimental techniques.
4. Analysis must evaluate the probability of backtest overfitting by taking into account the number of trials it took to distill the strategy.
5. Backtest results are communicated to management and not shared with anyone else.

1.3.1.5 Deployment Team

1. The deployment team is tasked with integrating the strategy code into the production line. Some components may be reused by multiple strategies, especially when they share common features.
2. Team members are algorithm specialists
3. Their job is to ensure that the deployed solution is logically identical to the prototype they received.
4. Also they optimize the implementation sufficiently, such that production latency is minimized.
5. As production calculations often are time sensitive, this team will rely heavily on
   1. process schedulers,
   2. automation servers (Jenkins),
   3. vectorization,
   4. multithreading,
   5. multiprocessing,
   6. graphics processing unit (GPU-NVIDIA),
   7. distributed computing (Hadoop),
   8. high-performance computing (Slurm), and
   9. parallel computing techniques in general

   
   

next: portfolio oversight

Thanks for your support! I also will shrink from the highly technical chapters but if you have additional insights based on your ML training please feel free to chime in.

1.3.1.6 Portfolio Oversight

Life cycle of a strategy

1. Embargo
   1. Initially, the strategy is run on data observed after the end date of a backtest

2. Paper trading
   1. On a live real-time feed

3. Graduation
   1. Real trades
   2. Performance is evaluated precisely, including attributed risk, returns, and costs.

4. Re-allocation
   1. Based on the production performance, funds are diverted from other graduated strategies
   2. A strategy's allocation follows a concave function. The initial allocation (at graduation) is small. As time passes, and the strategy performs as expected, the allocation is increased. Over time, performance decays, and allocations become gradually smaller.

5. Decommission
   1. Nothing beautiful lasts

1. It is preferable to release new variations of a strategy and run them in parallel with old versions. Each version will go through the above lifecycle, and old strategies will receive smaller allocations as a matter of diversification, while taking into account the degree of confidence derived from their longer track record.

1.3.2 Structure by Strategy Component

1. Many investment managers believe that the secret to riches is to implement an extremely complex ML algorithm. They are setting themselves up for disappointment.
2. If it was as easy as coding a state-of-the art classifier, most people in Silicon Valley would be billionaires.

1.3.2.1 Data

1. Problem: Garbage in, garbage out.
2. Solution: Work with unique, hard-to-manipulate data.

1.3.2.2 Software

1. Problem: A specialized task requires customized tools.
2. Solution: Develop your own classes. Using popular libraries means more competitors tapping the same well.

Notice how much emphasis M puts on data ‘freshness’ and avoiding competition. I have my doubts about the relative importance of this idea. If you recall our discussion on third order thinking. A trend can only start or end when there is a broad consensus and so being the first to make some fundamental discovery is only of value if you’re in a respected position where you can (eventually) transmit that understanding to a liquidity pool. If you’re early to a trend you can easily go broke before it takes off. Early is only beneficial if you’re able to front-run someone’s large orders.

However maybe M is not talking about market data at all, but rather data about other market participants. Market metadata, if you will. Then it makes perfect sense.

1.3.2.3 Hardware

1. Problem: ML involves some of the most computationally intensive tasks in all of mathematics.
2. Solution: Become an HPC expert. If possible, partner with a National Laboratory to build a supercomputer. Be sure to let us know in the comments when you achieve this!
   Attached Image (click to enlarge)

   
   
   
   

1.3.2.4 Math

1. Problem: Mathematical proofs can take years, decades, and centuries. No investor will wait that long.
2. Solution: Use experimental math. Solve hard, intractable problems, not by proof but by experiment. For example, Bailey, Borwein, and Plouffe [1997] found a spigot algorithm for π (pi) without proof, against the prior perception that such mathematical finding would not be possible. Note that the example cited is not financial in nature.
3. An algorithm that is optimal in-sample can perform poorly out-of-sample. There is no mathematical proof for investment success.

1.3.2.5 Meta-Strategies

1. Problem: Amateurs develop individual strategies, believing that there is such a thing as a magical formula for riches. In contrast, professionals develop methods to mass-produce strategies. The money is not in making a car, it is in making a car factory.
2. Solution: Think like a business. Your goal is to run a research lab like a factory, where true discoveries are not born out of inspiration, but out of methodic hard work. That was the philosophy of physicist Ernest Lawrence, the founder of the first U.S. National Laboratory. Again not a financial example. How many hedge funds or banks are run like this?
3. Most discoveries in finance are false, due to multiple testing and selection bias.
4. When misused, ML algorithms will confuse statistical flukes with patterns.
5. Combined with the low signal-to-noise ratio that characterizes finance, all but ensures that careless users will produce false discoveries at an ever-greater speed.

1.6 FAQs

1. Millions of years of evolution (a genetic algorithm) have fine-tuned our ape brains to survive in a hostile 3-dimensional world where the laws of nature are static. Now, when it comes to identifying subtle patterns in a high-dimensional world, where the rules of the game change every day, all that fine-tuning turns out to be detrimental.
2. ML algorithms can spot patterns in a 100-dimensional world as easily as in our familiar 3-dimensional one.
3. it is possible that the best results are achieved by combining discretionary PMs with ML algorithms.
4. This is what has come to be known as the “quantamental” way. (At least by M. I think he’s trying to get it to catch on.)
5. Econometrics is the application of classical statistical methods to economic and financial series. The essential tool of econometrics is multivariate linear regression, an 18th-century technology that was already mastered by Gauss before 1794 (Stigler [1981]).
6. econometrics is a primary reason economics and finance have not experienced meaningful progress over the past 70 years. (!!) sounds like someone has an axe to grind.
7. Economics is stuck in a dark age because it hasn’t made scientific progress.
8. What if economists finally started to consider nonlinear functions? Where is our Kepler? It’s you, M!
9. A ML algorithm learns patterns in a high-dimensional space without being specifically directed.
10. While we all laugh when we see an algorithm make a silly mistake, keep in mind, algorithms have been around only a fraction of our millions of years. Every day they get better at this, we do not. Humans are slow learners, which puts us at a disadvantage in a fast changing world.

Q:What do you say to people who dismiss ML algorithms as black boxes?
A: People mistrust what they don’t understand. M actually compares ML skeptics to Luddites.

 

1. The book is agnostic with regards to the particular ML algorithm you choose.
2. This is the first book to provide a complete and systematic treatment of ML methods specific for finance (!)
3. If there was a precise method to prevent backtest overfitting, we would be able to take backtests to the bank. A backtest would be almost as good as cash, rather than a sales pitch.
4. Backtest overfitting is arguably the most important open problem in all of mathematical finance. It is our equivalent to “P versus NP” in computer science.
5. Why would you read a book that deals with CAPM, APT, asset allocation techniques, risk management, etc. when the empirical results that support those arguments were selected without determining their false discovery probabilities?
6. If backtest overfitting is an issue in econometric analysis, the flexibility of ML makes it a constant threat to your work.

Next: chapter 2: Financial Data Structures

Great philosophical thread "The Finance Book Club".

I trade live and at this point trading is more of the hobby then making money. But I do love good profits and good PF.

I disagree with some of the statements in the latest comments, especially on back testing and optimizing.

Core of my strategy is to optimize my ea for previous 3 weeks of data and trade forward one week with the best possible inputs that brought greatest profits previous 3 weeks, "hoping" that momentum of previous 3 weeks will rub off on the week forward. I optimize up to 150 parameters/inputs for entries, MM(SL, Trailings, TP), Session Times, etc.. and choose only 12-20 of the ones that bring additional incremental profits and better PF. It works! To be together with the market and "Momentum" of the flow of the market are the principles here.

I learned allot about proper optimization from John Ehlers.

My greatest inspiration for mindfulness are teaching of Mark Douglas, especially this wording. "If you have an edge, STOP thinking", and execute that "random" probability trade.

Chapter 2

1. In general, you do not want to consume someone else's processed dataset, as the likely outcome will be that you discover what someone else already knows or will figure out soon. Ideally your starting point is a collection of unstructured, raw data that you are going to process in a way that will lead to informative features.

The Four Essential Types of Financial Data

1. Fundamental data
   1. Assets
   2. Liabilities
   3. Sales Costs/earnings
   4. Macro variables

2. Market Data
   1. Price/yield/implied volatility
   2. Volume Dividend/coupons
   3. Open interest
   4. Aggressor side Quotes/cancellations

3. Analytics
   1. Analyst recommendations
   2. Credit ratings
   3. Earnings expectations
   4. News sentiment

4. Alternative Data
   1. Satellite/CCTV images
   2. Google searches
   3. Twitter/chats
   4. Metadata

Do you do this manually or have you automated the process?

2.2.1 Fundamental data

 

1. A common beginner's error is to assume that this (fundamental) data was published at the end of the reporting period. That is never the case. (?) For example, fundamental data published by Bloomberg is indexed by the last date included in the report, which precedes the date of the release (often by 1.5 months). In other words, Bloomberg is assigning those values to a date when they were not known. You could not believe how many papers are published every year using misaligned fundamental data, especially in the factor-investing literature. Once you align the data correctly, a substantial number of findings in those papers cannot be reproduced. What’re we talking here? 20%? 99%?
2. A second aspect of fundamental data is that it is often backfilled or reinstated. “Backfilling” means that missing data is assigned a value, even if those values were unknown at that time.
3. A company may issue multiple corrections for a past quarter's results long after the first publication, and data vendors may overwrite the initial values with their corrections. The problem is, the corrected values were not known on that first release date.
4. Fundamental data is extremely regularized and low frequency. Being so accessible to the marketplace, it is rather unlikely that there is much value left to be exploited. And yet thousands of news traders would disagree.

2.2.2 Market Data

1. Market data includes all trading activity that takes place in an exchange (like CME) or trading venue (like MarketAxess).
2. Ideally, your data provider has given you a raw feed, with all sorts of unstructured information, like FIX messages that allow you to fully reconstruct the trading book, or the full collection of BWIC (bids wanted in competition) responses. Maybe Peter Caleb knows what this is, but he won’t tell us.
3. Every market participant leaves a characteristic footprint in the trading records, and with enough patience, you will find a way to anticipate a competitor's next move. Data leakage. The paramount concern of all institutionals.
4. For example, TWAP algorithms leave a very particular footprint that is used by predatory algorithms to front-run their end-of-day trading (usually hedging) activity (Easley, López de Prado, and O'Hara [2011]).
5. Human GUI traders often trade in round lots, and you can use this fact to estimate what percentage of the volume is coming from them at a given point in time, then associate it with a particular market behavior.
6. FIX data is abundant and difficult to process so M loves it.

I automated the whole process up to the point, where it takes only 1 hour on the weekend to prepare a new set.

Self-adjusting EAs are an interesting research project. If ML has some promise I would expect it to be in adjusting inputs to meet changing conditions faster than a human.

2.2.3 Analytics

1. You could think of analytics as derivative data, based on an original source, which could be fundamental, market, alternative, or even a collection of other analytics. What characterizes analytics is not the content of the information, but that it is not readily available from an original source, and that it has been processed for you in a particular way.
2. Investment banks and research firms sell valuable information that results from in-depth analyses of companies' business models, activities, competition, outlook, etc. Some specialized firms sell statistics derived from alternative data, for example, the sentiment extracted from news reports and social media.
3. Analytics may be costly, the methodology used in their production may be biased or opaque, and you will not be the sole consumer.

2,2,4 Alternative Data

1. differentiate among alternative data produced by
   1. individuals (social media, news, web searches, etc.),
   2. business processes (transactions, corporate data, government agencies, etc.), and
   3. sensors (satellites, geolocation, weather, CCTV, etc.)

   
   

2. What truly characterizes alternative data is that it is primary information, that is, information that has not made it to the other sources.
3. Two problematic aspects of alternative data are their cost and privacy concerns. All that spy craft is expensive, and the surveilled company may object, not to mention bystanders. Alternative data offers the opportunity to work with truly unique, hard-to-process datasets.
4. Remember, data that is hard to store, manipulate, and operate is always the most promising. Or the hardest to immediately refute?
5. You will recognize that a dataset may be useful if it annoys your data infrastructure team. Perhaps your competitors did not try to use it for logistic reasons, gave up midway, or processed it incorrectly.

2.3 Bars

1. In order to apply ML algorithms on your unstructured data, we need to parse it, extract valuable information from it, and store those extractions in a regularized format. Most ML algorithms assume a table representation of the extracted data. Finance practitioners often refer to those tables’ rows as “bars.” We can distinguish between two categories of bar methods:
   1. (1) standard bar methods, which are common in the literature, and
   2. (2) more advanced, information-driven methods,

   
   

The rest of this section M describes ‘bars’ more like chart bars, not table rows, but maybe in a data series there is no real difference? I’d love to know for sure.

2.3.1 standard bars

1. very popular in the financial industry, to the point that most data vendors’ APIs offer several of them. The purpose of these methods is to transform a series of observations that arrive at irregular frequency (often referred to as “inhomogeneous series”) into a homogeneous series derived from regular sampling. So, chart bars, yes?

2.3.1.1 Time Bars

1. Time bars are obtained by sampling information at fixed time intervals, e.g., once every minute.
2. The information collected usually includes:
   1. Timestamp
   2. Volume-weighted average price (VWAP)
   3. Open (i.e., first) price
   4. Close (i.e., last) price
   5. High price
   6. Low price
   7. Volume traded, etc.

   
   

3. Although time bars are perhaps the most popular among practitioners and academics, they should be avoided for two reasons.
   1. First, markets do not process information at a constant time interval. The hour following the open is much more active than the hour around noon (or the hour around midnight in the case of futures). As biological beings, it makes sense for humans to organize their day according to the sunlight cycle. But today's markets are operated by algorithms that trade
   2. time bars oversample information during low-activity periods and undersample information during high-activity periods.

   
   

4. time-sampled series often exhibit poor statistical properties, like serial correlation, heteroscedasticity, and non-normality of returns

2.3.1.2 Tick Bars

1. The sample variables listed earlier (timestamp, VWAP, open price, etc.) will be extracted each time a pre-defined number of transactions takes place, e.g., 1,000 ticks.
2. Mandelbrot and Taylor [1967] were among the first to realize that sampling as a function of the number of transactions exhibited desirable statistical properties: “Price changes over a fixed number of transactions may have a Gaussian distribution. Price changes over a fixed time period may follow a stable Paretian distribution, whose variance is infinite. Since the number of transactions in any time period is random, the above statements are not necessarily in disagreement.”
3. Multiple studies have confirmed that sampling as a function of trading activity allows us to achieve returns closer to IID Normal (independent and identically distributed)
4. When constructing tick bars, you need to be aware of outliers. Many exchanges carry out an auction at the open and an auction at the close. This means that for a period of time, the order book accumulates bids and offers without matching them. When the auction concludes, a large trade is published at the clearing price, for an outsized amount. This auction trade could be the equivalent of thousands of ticks, even though it is reported as one tick. Just one reason never to trade near the end of the day.