Chronos: Learning the Language of Time Series with Abdul Fatir Ansari - #685

0:00This is also one of the problems in the literature. In many papers, they are using the same set of datasets. And at some point, improvements on those benchmarks don't really mean anything because there is no real absolute improvement in performance over time.

0:25All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by Abdul Fethir Ansari. Abdul is a machine learning scientist at AWS AI Labs in Berlin. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Abdul, welcome to the podcast. Thanks a lot. Thanks a lot, Sam, for inviting me. I'm looking forward to digging into our conversation. This time around, we'll be talking about your work in modeling time series, and in particular, the Kronos paper. Before we jump into that, I'd love to have you share a little bit about your background and how you came to work in the field.

1:05Yeah, sure. Thanks a lot. My background is actually a bit interesting because my bachelor's was in civil engineering. It was not really computer science or machine learning. But after that, I moved to Singapore where I got my PhD from NUS. And there I worked mostly on deep generative models. And that's the field I'm broadly interested in, everything related to generative models, which basically means modeling, representation, learning, inference, these kinds of things. And for time series forecasting, I actually did an internship at the same team at Amazon. That's where I got interested and I got introduced to time series modeling.

1:38And since then, I've been working on time series forecasting. And then I joined Amazon full time, the same team, and I've been working on time series since then. Awesome. Awesome. Was there a particular event in your academic experience that was the bridge between civil engineering and machine learning? Yeah, I wouldn't say so. I think I was always interested in computer science and machine learning since the beginning. It's just a weird sequence of events that led me to get into civil engineering. Okay, okay. So let's talk a little bit about the paper. What was the inspiration for taking on this particular project?

2:15I think the major inspiration actually came from the things that we see in production time series systems, because typically people use very simple models in production systems. And the reason for that is there is a lot of work that needs to be done to put in a new state of the art model into any production system, because first you need to set up some kind of data pipeline to collect data, then train models on this data and then do a lot of evaluation before you can do some reasonable change of models. And this is a lot of steps and people tend to kind of not do all of this stuff and use very simple models.

2:52And time series forecasting is a bit special in the sense that where very simple models, in fact, something like seasonal naive, which basically takes the previous season as the forecast, seems to do very well for many kinds of time series data set. So people tend to choose and use these simpler models compared to state-of-the-art deep learning models. And there's also this kind of hesitancy amongst people to not use state-of-the-art models because in the academic domain, you have a few data sets that people kind of overfit on. And it's very easy to get misled by those results and then hope that that model actually works on real-world time series, which often is not the case.

3:30So you referenced real-world time series. What are some of the types of problems that you are trying to apply real-world time series analysis to? There are actually a lot of problems both inside Amazon and outside where people use time series forecasting. Like most importantly, it's used for some kind of demand forecasting, like, for example, for energy demand forecasting or also for retail forecasting. I mean, Amazon is one of the biggest e-commerce companies. So there's also this retail forecasting component, which is also not unique to Amazon, but in general for labor planning, you need to do forecasting and many other things.

4:07So there are actually a lot of problems that use time series forecasting, at least in the loop, not really as an end goal. So you would want to use your forecast somehow to take some kind of action. So it's not really just to get a forecast, but to actually use that forecast somehow in a downstream system. And you mentioned some of the traditional approaches to time series. Can you provide kind of an overview of how the traditional statistical approaches tend to work? Sure. So I think for the existing models for time series forecasting, I would broadly classify into two categories. But of course, there are many other different kinds of classifications.

4:47One of them are so-called statistical local models. And the idea for these models is that you fit one model for each individual time series. So you can just look at one time series. So models like ETS, ARIMA, these models basically fit one model or one system for each time series. The second kind of models, I would say, are task-specific deep learning models, like DeepAR, PESH-TSD, or all the recent models that are coming out. I would call them task-specific deep learning models because they look at a bunch of time series that are somehow related. And then they train one model, so-called global model, across all of these time series.

5:26And then you can use this model for inference on some downstream time series. So these are the two broad categories. Of course, there are positives and negatives for both of these techniques. for simple statistical models I would say the positives are that they are extremely strong baselines like I mentioned previously especially in the case where you have limited data they tend to do very well and also for low frequency time series like monthly, yearly weekly this kind of data they seem to do very well because typically there you don't really have a lot of data but as soon as you get a lot of data deep learning models tend to do better because then you can train your model on a large corpus of such time series and then you can hope to get better improvements.

6:08There's another point of difference between these two models in terms of flexibility. I would say the first model, the local models, they train one model for each time series, which makes them slow and less flexible. And also they have some kind of design assumptions for how time series should be decomposed, what kind of error distributions you are modeling. But in the case of deep learning models, of course, all bets are off, right? You can really have any kind of arbitrary design assumption in a deep learning model. So in that sense, they are more flexible and they can also be used for multiple time series together.

6:43So these are some positives and negatives. Of course, more recently, you have these pre-trained models, just like Kronos. The idea there is to train one model on a large corpus of time series data and then hope that on an unseen downstream task or downstream time series, it would do well. And that's what we see, at least in our work. You mentioned overfitting as something that you see a lot of with, is that applied to traditional models in particular or the deep learning models that you mentioned? I would say overfitting is probably more of a problem for deep learning models. when you have limited data and a large-ish model, you can tend to overfit very easily.

7:28And this is what we see typically. And this is also one of the problems in the literature. Not necessarily any specific paper, but in many papers, they are using the same set of data sets. And at some point, improvements on those benchmarks don't really mean anything because there is no real absolute improvement in performance over time. And the field kind of is converging to some performance. There's always some delta that you can show compared to some selected baselines. But in terms of absolute performance, things don't seem to be improving. So we definitely need better data sets, better benchmarks for sure to say that this model is better than some other baseline model.

8:14Got it. Got it. And so the approach that you took with the Kronos work is to try to leverage language models. Is this the first paper that tries to leverage LLMs, or in what ways have folks tried to leverage LLMs to solve the same type of problem? This is definitely not the first work. I would like to say that this is one of the good attempts, but this is definitely not the first work. In fact, if you look at Neodips last year, there were several papers that are trying to leverage language models somehow. So I would broadly classify them into two categories. Some models that just try to use pre-trained language models directly to prompt them in a certain way for time series forecasting.

8:58So one of the good examples of such a model or such a method would be LLM time, which was in Neodips last year. It basically uses a LAMA 70 billion or GPT-3 model for basically doing forecasting. you convert your time series somehow into a string of numbers, and then you just prompt the model using this string. The second category of model that uses LLMs for forecasting would be like time LLM or GPT-40S. So these models take some backbone language model architecture like GPT-2, and then they have some kind of fine-tuning scheme that you can adapt the model weights for time series forecasting. So these are the two main categories that use language models for forecasting.

9:39But even beyond that, there are other works like, for example, Moirai from Salesforce and also Laglama and a few other works, TimeGPT from Nixla. These models basically, they train a transformer-based model on a large corpus of time series data. They're not really language models per se. They're just like large transformer models trained on a lot of time series data. So in some sense, Kronos is somewhere in between these kinds of works. So it uses a language model architecture, but it also trained from scratch on a lot of time series data. So it's not really using a pre-trained language model like some of the previous works.

10:18Got it. Got it. Got it. And it seems like one of the core ideas around the way that you've attacked this problem is the tokenization scheme. Yeah, exactly. One of the key motivations of our work was to be as simple slash as lazy as possible. So the idea is, why reinvent the wheel? Why build something new? We just wanted to verify what the things, the infrastructure, the architectures, the libraries, everything that's already there for natural language processing. Can we directly use it for time-size forecasting? In fact, in this work, we don't make any changes to the language model architecture.

10:55And it's also surprising that we also don't make any changes to the objective function, right? for language modeling, the objective function that's used typically is cross-entropy. So it's a classification-based loss. And it might be counterintuitive to say that for time series forecasting, we are using cross-entropy or a classification loss. So essentially, we are doing regression via classification. And it tends to do really well, which is surprising and not surprising because there are also some existing works that already do this. For example, if you are familiar with WaveNet, it was a generative model for audio.

11:28And audio signals are also time series. They also have a similar kind of approach to basically use like bins for modeling the audio signal. And we are also using a similar kind of approach. So at the core of it, it's really simple. So we don't have any time series specific design choices or design modifications from the architecture perspective. And also from the loss function perspective. We just encode the time series somehow into a discrete signal from a fixed vocabulary. And then we train the language model on a lot of such tokenized time series data. And what does that mean or look like to tokenize a time series data set or to map it to a fixed vocabulary?

12:16Yeah. So, of course, tokenization is a broad general term. but the specific scheme that we are using in our work is we take the original time series which can be at any arbitrary level so it can be really millions of something or between zero and one so it can be at an arbitrary scale so the first step is to somehow normalize the time series into a reasonable range and this normalization can be can be a design choice in our work we use mean scaling which just scales the time series by its absolute mean and then what what that does is it squishes the time series down into some reasonable range.

12:52And once it comes into a reasonable range, we select some bounds on the real line and we basically construct uniformly spaced bins between these bounds. And we take the scale time series and we quantize them into these bounds. And after quantization, you basically get the indices of these bins, which also kind of looks like the original time series. But of course, there is some loss of precision here because the bins have non-zero width. So there is some loss of precision, but you have somehow converted your original continuous signal into a discrete signal from a fixed vocabulary. And then once you have done this kind of tokenization, you can essentially train any kind of language model architecture on this.

13:33So it doesn't really matter which architecture you select. So in the paper, we really focused on the T5 encoder-decoder architecture, which seems to do very well. But it's not really a restriction. In fact, you can choose an architecture which is not even transformer-based. So, for example, recent works that do state-based modeling for lateral language processing like Mamba, you could also use such kinds of architectures. So it's not really a restrictive framework. Is there a simpler possible way that you could have done the tokenization? That seems like a very straightforward approach to it. I'm not sure if you can be simpler than that.

14:11But if you have ideas, I'm open to discussing them. But I feel like this is the simplest possible. In fact, just for quantization, in prior literature, there are some other techniques that one could use. So, for example, here you have uniformly spaced bins. But what you could do is something called data-dependent binning. Something like quantile binning, where basically you select your bin edges such that equal number of points fall in each bin. But the reason we didn't select such kind of approach is then this approach really depends on the training corpus. And in the downstream inference time, you could have such data which probably was never seen during training and the distribution would be completely different.

14:54And you might see some worse performance on downstream data. So that's the reason we went with the simplest possible thing. But I'm not sure if you can go even simpler. It's extremely simple. That's what really struck me. Like I can imagine ways to get arbitrarily more complex, but that seems like the most basic place to start. And, you know, as it often happens, it surprises me that it works. Yeah, it's definitely very surprising. And so you mentioned that you use T5, but you didn't need to use T5. Is there a reason why you chose T5 for the paper? And were there other language models that you tried?

15:35I have a very boring answer for it. The boring answer is that I just started with T5. So when we did our initial experiments, we just started with T5. And the reason for starting with T5 is because T5 is extremely popular in the NLP community. I mean, these decoder-only models are only getting popular very recently, but T5 has been popular since its beginning. The second reason is all of these recent decoder-only models, their configurations and everything were typically available in very large sizes. But T5 comes from a tiny size, which has only 8 million parameters, to an Excel size, which has a few billion parameters.

16:13So there was a wide range of configurations, and we knew that these configurations work very well for NLP. So that's the main reason that we selected the T5 because it came in a lot of different configurations that people already knew works well for NLP. So we didn't want to redesign or reinvent the wheel. That's the main reason why we selected T5. Got it. Got it. Another element that the paper calls out as being important to the work is the use of generated data or data augmentation. Can you talk a little bit about how you incorporated that? Sure. the modeling framework or any kind of loss function, whatever, architecture design, I think it's only one side of the story.

16:55And as more and more papers in language modeling are also saying that data is the key, right? Data is the solution to real good quality language models. I would say same is the case for time series forecasting. And one of the special limitations for time series forecasting is that you don't actually have so much good quality publicly available time series data. Many companies have some internal data sets that probably they won't release that are probably much more high quality. But in public domain, we don't have such high quality data sets. And you say that, but you also mentioned in the paper that you tested against 48 data sets, which seems like a big number of data sets.

17:36Yeah, exactly. It seems like a big, it seems like a, so by the way, we tested on 42, but it seems like a big number. but some of those data sets may have only a few time series, like 10. So at the end of the day, in total, we collected 890 ,000 time series. But there is probably an over-representation of certain domains and a very limited or under-representation of some other domains. So this data set, I think comparing to the language modeling data sets, which is essentially at the scale of the entire internet, this is significantly less. When you say 890 ,000 time series, I guess the other unknown variable is the length of those time series.

18:19Exactly. To compare this to other approaches to training language models, does it make sense to boil it down into a number of tokens that were used in training? Do you know that number? If I remember correctly, it's something around 84 billion. But I would say that for language modeling, the tokens are much more high quality and diverse compared to time series forecasting. Because here, for example, there is an over-representation of Wikipedia page visits data set or weather bench, which is weather data set, right? So it might have very similar characteristics for multiple cities, and it may not bring so much additional information compared to the case of language modeling.

19:01And that's the main reason why we had to try these augmentation schemes to basically improve the performance even further. Got it. So just to make sure I understand that in the case of language modeling, each additional web page or book or aspect of data is kind of fairly diverse. Whereas it sounds like you're saying that many of the time series in your data set are essentially different slices of the same generating function. So, you know, visits to, you know, the same website, that kind of thing. Yeah, I mean, exactly. I don't really have data to back this up, but based on the sizes of the data sets.

19:44So for example, the weather events data set that we used was fairly large. The Wikipedia page, which is data set that we used, is also fairly large. So it contributes a lot to this 890 ,000 time series. And the diversity, I would say, comparing to the language modeling data sets is low, but I don't have specific data points to back this up. Fair enough. So you were talking about incorporating data augmentation. Yeah. So in particular, we looked at two augmentation schemes. One of them essentially just takes random real-world time series from different data sets. They may or may not be from different data sets.

20:23And then just takes convex combinations of these time series. And the idea here is to just diversify the types of patterns that the model sees during training. So, for example, from some data set, maybe you see some kind of trend. From some other data set, you see some other interesting seasonal patterns. If you take a convex combination of these two data sets, you will get a time series with combinations of these two types of patterns. And that's essentially what this augmentation scheme does. We also have an example in the paper showing what exactly it does. The second scheme that we used is to generate new synthetic data using Gaussian processes.

21:00So it's an extremely simple scheme. the core idea is to have some kind of bank of Gaussian process kernels. And so these kernels would represent some base functions or base time series. So for example, from a linear kernel, if you draw a signal, you get a linear function. From a periodic kernel, you get some kind of seasonal time series. And the idea is to randomly draw these kernels from this bank and combine it using additional multiplication, which also gives you a valid kernel. And once you have these more complex kernels, you can draw samples from these kernels. So let's say you have two linear kernels and you take their multiplication.

21:35So you get like a quadratic kernel, which would give you some kind of quadratic trend. But then if you add it to a periodic kernel, you get some quadratic trend with seasonal properties. So you can basically construct very complex time series using this very simple scheme. And in the paper, we show that these augmentation schemes are extremely helpful, especially in the case of zero shot performance, which is what really is the thing that we care about at the end of the day. So on an unseen time series, how does the model perform? And adding such kind of augmentation schemes and synthetic data really improves the performance of the model on unseen time series tasks.

22:13And so what percentage of the ultimate training data set was synthetic versus the data sets that you sourced? So we have a 90-10 split. So 90 % of data comes from the real augmentations and 10 % of data comes from synthetic data set. So this was the original scheme that we used for the main models. But then we also did some ablation, which shows that this is a near optimal choice. So if you select 90 % real and 10 % synthetic, you get almost the best performing model on all metrics. But we also trained with different percentages. For example, we also trained a model that was purely trained on synthetic data, just to see what we can gain on only training the model from synthetic data.

22:58And interestingly, this model also doesn't do very bad. So it's actually one of the models that does reasonably well. It's better than some baselines, which is very, very interesting. Of course, it's not as good as the model trained with both real and synthetic data, but there could be potentially ways to improve the quality of synthetic data to reduce this gap between real and synthetic data. And this, I would say, is slightly easier compared to the case of language modeling, because synthetic data in the case of language modeling really means you first train a big model, and then from this big model, you generate synthetic data.

23:35But here, there's a very simple scheme. You don't, so it's not a chicken and egg problem here, because here you don't really need data a priori. You can come up with a scheme like using Gaussian processes, like we did in the paper, to draw reasonably looking time series data. Of course, what data, what time series is good or bad, this problem is really unsolved, at least at this point, if this time series would really help in the downstream performance. But that's an interesting way to augment using purely synthetic data sets. Okay. I thought I heard you mentioned, were you distinguishing between the real data sets that has been augmented using the combinations and synthetic?

24:22When you talk about the 90-10 split, is the 90 strictly real data sets and the 10 is a combination of the TS Mixup and KernelSynth or was TS Mixup stuff included in the 90? So the 90 is actually purely TS mixup. And because in TS mixup, you have a parameter k, which basically tells you how many time series you want to draw. And this k can range from one up to some value k. So if you sample k equals to one, you get the original time series. So original data is also represented. But this 90 % includes all the augmentations, possibly with original time series, but also with combinations. Got it, got it.

25:03So you're distinguishing real and augmented data via TS MixUp from the synthetic data via KernelSynth, which is the Gaussian process stuff. Yeah. The 10 % is only KernelSynth. Got it. Okay. Awesome. And then so how do you, you evaluated against a series of historically relevant academic benchmarks? Yes. So we used many data sets from, for example, Monash benchmark, which has a lot of time series data sets, but also some other data sets that are popular, like electricity data set, ETT data set. There's several other data sets that are popular in the literature. Our specific benchmarking or evaluation setup is different from these existing bugs.

25:51Like, for example, there's a long-term time-series forecasting benchmark that's quite popular in many papers. But we strictly went with a new benchmark setup so as to not give any model any kind of advantage. So we just use these data sets, but the evaluation setup is different and it's the same across all the models that we're using. Okay, can you talk a little bit about the evaluation setup then? Yeah. So for the evaluation setup, basically we just selected, so for each kind of dataset based on its frequency, we selected some splitting points so that not to leak any data for any model. And then basically with some prediction length, that's it.