Back

Artificial intelligence gives weather forecasters a new edge

158 points3 monthsnytimes.com
neonate3 months ago
lainga3 months ago

Sigh... don't make me bind a hotkey to paste my spiel about how these non-primitive-equation-based models are bad at long tails and freak weather...

Ed.: too late! https://news.ycombinator.com/item?id=40577332

Majromax3 months ago

Long tails and freak weather are the hottest topics of research in the area of data-driven weather forecasting. ECMWF, highlighted in this article, is attempting to extend its ML forecast system to ensemble predictions (https://www.ecmwf.int/en/about/media-centre/aifs-blog/2024/e...).

If these methods work, they'll likely improve our ability to model long tails. Traditional NWP is extremely expensive, so cutting-edge models can have either high resolution xor large ensembles. You need high resolution for the detail, but you need large ensembles to see into the tails of the distribution; it's a persistent problem.

In inference, ML-based models run a bit over two orders of magnitude faster than traditional NWP, with the gains split between running on GPUs (possibly replicable) and fantastic levels of numerical intensity thanks to everything being matrix-matrix products (much harder to replicate with conventional algorithms). That opens a lot of freedom to expand ensemble sizes and the like.

kardos3 months ago

The limitations of the non-physics-based models is under rapid exploration and the bounds of their utility will be a lot more clear in a year or two. For now, they seem to outperform physics-based models in larger scales. The future may be a hybrid: https://arxiv.org/abs/2407.06100v2

lainga3 months ago

Interesting, thanks!

Well, as I said, I would expect them to outperform at large scales, specifically because they're learning and memoising large, stable patterns (ed. in the sense of teleconnections) at low wavenumbers.

I hope they have a switch to turn it off if we ever mess up and go back to a single-cell Hadley configuration :)

uoaei3 months ago

Hybrid models are almost certainly the future, i.e., developing structural equations and filling in data or modeling gaps with relatively small NNs for approximation.

Majromax3 months ago

Hybrid models have the conceptual edge, but it's not yet obvious that they'll become the dominant AI forecasting paradigm.

The hybridization acts as a strong regularizer. This is a good thing, but it's not yet obvious that it's a necessary thing for short to medium-term forecasts. There seems to be enough extant data that pure learning models figure out the dynamics relatively easily.

Hybrid models are more obviously appropriate if we think about extending forecasts to poorly-constrained environments, like non-modern climates or exoweather. You can run an atmospheric model like WRF but with parameters set for Mars (no, not for colonization, but for understanding dust storms and the like), and we definitely don't have enough data to train a "Mars weather predictor."

The difficulty in training NeuralGCM is that one has to backpropagate over many dynamics steps (essentially all the time between data snapshots) to train the NN parameterizations. That's very memory-intensive, and for now NeuralGCM-like models run at coarser resolutions than fully-learned peers.

mistrial93 months ago

a senior weather modeler that is US-based but works on some of the Euro models spoke recently about the use of AI from his perspective. The comments were more than half communication skills with the audience, so it was not a math lecture. Overall I believe that this UC Berkeley speaker said almost exactly what the parent-post says.

johnthescott3 months ago

my bs detector smells a non-elephant animal.

markovs_gun3 months ago

How do ML models do with chaotic systems in general? I am a beginner in terms of ML, but based on my limited knowledge it seems like ML models would generally do fairly poorly with chaotic systems. Then again, maybe with enough data it doesn't really matter.

jrockway3 months ago

I think there is potential in this approach, but I don't think we have the training data yet. We have 100+ years of storm tracks, but we don't have 100+ years of surface observations at the mesonet granularity (like, every city has a reliable weather station). We only measure the upper atmosphere from a few points twice a day. I think that the "chaos" and "butterfly effect" type influences can be controlled, but probably not without really granular surface data, including over the ocean.

So while GPUs are ready to crunch the numbers, we don't actually have the numbers yet.

bbstats3 months ago

This is the opposite of what the model is bad at

npalli3 months ago

Seems to be a "pattern" in AI for weather forecasting --

  1. Due to AI hype and funding (even predecessor hype cycles like "big data", ML, even GOF time-series statistics), generate 100s of AI models.
  2. Of these, a few perform better than current models and are seen as the "future".
  3. One year later models in 2. fail catastrophically.
  4. Go back to step 1.
darby_nine3 months ago

At first blush, AI seems to be fantastic for weather prediction:

1. Hugely chaotic model, which means deterministic prediction is extremely expensive, and accurate prediction more so.

2. Certain trends seem to dominate the observed time period, hence the field of meteorology.>

3. AI can clue you in as to which trends seem to be most likely to dominate, or if uncertainty is too high to predict, without the investment of major labor costs.

While I'm not sure what the market offers, I can say that if the demand-side seems unsatisfied this is the perfect place for a state-operated solution to enter: a zero-margin service where quality in the long-term is demanded over short-term profits.

EDIT: just clarifying terms

EDIT2: Meteorology -> meteorology

mjburgess3 months ago

Well with "AI" all we're talking about is theory-free frequentist modelling.

We can already do that for the weather: just take the mode of the last 10 years of the weather on this day, and some sort of weighted mean for temp/humidity/etc.

All "AI" is doing here is overfitting this process. On the overfit range, yes, you'll get better peformance. But the baseline performance here, from taking averages, is already pretty good.

What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data.

partitioned3 months ago

This is simply not true and an uninformed opinion of what modeling physical events with AI looks like.

You seem to assume it’s a purely data-driven approach, it is not. You could use a physics informed neural network (PINN) that utilizes both approaches. That is, it uses both historical data (and likely synthetic data from physics models), as well as physical equations in the loss function (in this case atmospheric as fluid equations) as part of the training. It can truly be the best of both worlds if approached correctly.

That being said, 99% of AI out there is just masters thesis level data in -> prediction out, but that is far from what the useful AI models that are currently being developed to predict and forecast dynamical physical systems.

Additionally, you can generate synthetic data with the physical models of “edge” and “tail” events to train the model on. This by itself allows the ML model to be able to model almost all events that we can physically model, so at its base it’s at least as useful as the big O order models we use while being orders of magnitude faster. This doesn’t even account for using the physical equations to assist in training the model directly (through architecture tricks or in the loss function).

Source: I work on AI models that merge data and physics for dynamical physical systems

Majromax3 months ago

> That being said, 99% of AI out there is just masters thesis level data in -> prediction out, but that is far from what the useful AI models that are currently being developed to predict and forecast dynamical physical systems.

I think one surprising outcome of the recent wave of ML-based weather forecasting models is just how accurate the "dumb" approaches are. Pangu-Weather and a couple of its peers are essentially vision transformers without much explicit physics-informed regularization.

+1
mjburgess3 months ago
Majromax3 months ago

Although it's obviously difficult to crack open an ML model, they do perform enough computation to have potentially learned something like the dynamical equations for the atmosphere.

At the same time, some ML models are surprisingly parsimonious. Graphcast has about 37 million trainable parameters, but its output is a forecast (increment) of six variables on a 37-level, quarter-degree lat/lon grid. That's about 235 million outputs for a single forecast date, so it's safe to conclude that Graphcast cannot memorize its training set.

Researchers are also explicitly interested in probing the out-of-sample behaviour of ML models for plausibility. A paper last year by Hakim and Masanam (https://arxiv.org/abs/2309.10867) put Pangu-Weather through some simplified but out-of-sample test cases and saw physically plausible outputs, so the ML models have at least not fallen at the first hurdle.

Meanwhile, it's also not quite correct to give traditional models an automatic pass for out-of-sample behaviour. The large-scale dyanmical equations of the atmosphere are well-understood, but so much of the chaos comes from poorly-resolved, poorly-modeled, or poorly-constrained processes near and below the grid scale. The microstructure of clouds, for example, is completely invisible to models that must necessarily run at kilometer or tens-of-kilometer scales. Operational weather models rely on parameterizations to close the system and statistical correlations to assimilate observational data.

As far as I'm aware, all of the operational weather models missed the rapid intensification of hurricane Otis last year, an out-of-sample event with deadly consequences.

+1
mjburgess3 months ago
+1
Kon-Peki3 months ago
jebarker3 months ago

> What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data.

Yes, it seems like weather forecasting is a simulation problem not a low-shot prediction problem. I assume it's one of the computationally irreducible problems Stephen Wolfram talks about

prasoonds3 months ago

This has not really been my experience in my information bubble. But I admit I haven't kept up too well with the latest models and their failure modes.

Can you have provide any examples of the scenario you described?

mandibeet3 months ago

Initial successes can lead to failure but I think this is the pattern to future development

zuckerma3 months ago

So true.

kelsey987654313 months ago

IBM tried with Watson based modeling on weather.com and it was a spectacular failure. Hopefully things have improved enough to beat traditional weather models, but I honestly don't know enough about this area to speculate. Classical modeling, I.E. non neural network models, have been iteratively improved on for decades by the brightest minds in weather science and are barely able to eek out something close to a 50% accuracy in forecast. About 5 years ago that was 10x better than the AI based modeling available, It would be cool to see some advancement here!

jurgenaut233 months ago

IBM failed pretty much at everything AI-related except for Jeopardy and Chess (and even this was brute-forcing more than learning).

mdp20213 months ago

Should we not get comparisons and absolute values based on true|false positives|negatives?

The most relevant parts seem to be:

> In seconds... [the ML based] GraphCast can produce a 10-day forecast that would take a supercomputer [crunching through traditional deterministic methods] more than an hour

> GraphCast outperformed the best forecasting model of the European Center for Medium-Range Weather Forecasts more than 90 percent of the time

> Dr. Lam said the study found that GraphCast locked in on landfall in Nova Scotia three days before the supercomputers reached the same conclusion

> the European center was considered the world’s top weather agency because comparative tests have regularly shown its forecasts to exceed all others in accuracy

mrweasel3 months ago

> Weather experts say the A.I. systems are likely to complement the supercomputer approach because each method has its own particular strengths.

So it's not really that exciting, it's just a new tools for weather forecasting adding to the existing tool chain. I feel like this is only a news story because it's AI. New mathematical models are developed, but don't make it into the mainstream news, because math is boring.

namaria3 months ago

Yeah statistical games with historical datasets on the verge of a new climate epoch threatening us with unprecedented weather events is a great way to be caught by surprise when we can least afford to.

arnaudsm3 months ago

Umbrella marketing terms like this are misleading, and hype adjacent tech like GenAI.

"AI" is a family of statistical techniques, and weather forecasting has been using many of them for decades. Are GNNs displayed here more "AI" than the usual stochastic monte-carlo simulations ? I guess the word "Math" is not sexy anymore

snowpid3 months ago

I currently develop a ml model to forecast temperature (spatial resolution 10 m x 10m) by leveraging these models (in this case Climax by Microsoft https://github.com/microsoft/climax ). Feel free to ask!

1minusp3 months ago

Do you use WRF as an input? I think graphcast uses some kind of NWP input (qhich in my mind i've equated to WRF). Thank you for offering to answer questions!

snowpid3 months ago

We use a commercial meteorology service (aka Openweathermap with resolution 500m x 500m ) as one input so they might use WRF data (and indirectly my model as well). OWM was chosen, because they provide an easy api for providing general weather data.

lispisok3 months ago

Last I checked the AI models still used the same initial conditions used in models like the GFS and ECMWF which is a combination of a previous forecast adjusted with observations.

1minusp3 months ago

So GFS->graphcast = more accurate coarse-grained prediction (0,25 degree iirc), -> WRF for fine grained interpolation of forecast?

badkitty993 months ago

Quick let's fire all the weather professionals

mystified50163 months ago

Honestly I'd be surprised if we weren't already using ML in weather models. Seems obvious enough that it should have popped up years ago, before "AI".

7thaccount3 months ago

They've been using NN for this field for a long time as I understand it. Electricity demand, wind...etc. Plenty of vendors already out there since before AI became trendy again.

Majromax3 months ago

> Honestly I'd be surprised if we weren't already using ML in weather models.

Statistical modeling has long been used for "post-processing" the raw, gridded outputs of numerical models. This performs fairly well, and it is intended to correct for local factors that are not observable at the scale of the underlying model. For example, local topology might channel winds into a few primary directions, but a O(100m)-scale valley is too fine to show up in the large-scale grid of a global weather model.

You can also consider data assimilation to be a form of machine learning. ECMWF's main, traditional weather model essentially uses a form of backpropagation to find the Jacobian of observation values with respect to the input weather state, and from there it's straightforward (but far from simple) to optimize the weather system. Other centres take related approaches, such as ensemble Kalman filtering (when the underlying model is not so easily differentiated with respect to its inputs).

The remarkable product of the last year or two has been the discovery that full, relatively unconstrained ML models can do a good job at the core NWP prediction task. It was not at all obvious that the historic record (the ERA5 reanalysis, mostly: https://www.ecmwf.int/en/forecasts/dataset/ecmwf-reanalysis-...) was large enough for pure data-driven models to learn appropriate responses.

devwastaken3 months ago

In the U S. There is a lack of data input to create meaningful predictions because there is a lack of Doppler radar stations.

https://www.ncei.noaa.gov/maps/radar/

Need far more sites to gain full coverage, but no doubt they'll be strongly opposed by red disinformation.

1minusp3 months ago

Any comments from experts on how much better these models are compared to running base WRF over GFS forecasts?

Stephen_0xFF3 months ago

The images with the forecast don’t look like any forecasts I was watching. They predicted pretty well even on July 1st. The majority of models predicted it would go up like it did. The trend between EPS, GEFS, UKMET, and CMC all had it going up. Not saying an AI LLM won’t help, but this seems a little disingenuous.

tonetegeatinst3 months ago

I feel the need to insert some "edge compute" joke.

hansvm3 months ago

Slightly on-topic:

We have efficiently computable state-update equations available for weather. If neural nets are actually able to outperform those in general (and not just because of some form of sampling bias), why would that be?

1. They're exploiting information not available in the raw inputs to those weather models. Perhaps the number of cars driving vs parked in garages on a given day, contributing some local heat and perturbing PDE solutions away from the optimum. If this is the case, there's probably a fruitful line of research in using neural nets to estimate unmeasured parameters and using those directly in the applicable physics.

2. They just look better, e.g. by comparing raw neural outputs to sanitized weather reports or by using a metric biased toward low prediction variance at the expense of accuracy or something (this could be as simple as grading the neural net's probability distribution against the weather model's raw predicted output too, not considering the epistemological tweaked predictions the physics model also produced, thus implicitly giving the neural net and edge because the physics model can't get partial credit, even when worthy of it's and the AI can). Those sorts of mistakes are shockingly easy to make, even with much thought and careful review. Time will tell, I hope.

3. The gains are in some form of auxiliary data, like cheaply extrapolating a coarse forecast into a fine-grained forecast (superresolution/hallucination), or cheaply approximating a single weather estimate without fleshing out the details of the entire grid. This is potentially incredibly valuable, at the obvious cost of occasionally being very wrong. Use such predictions with care.

I don't think that list is exhaustive, but I'm not holding my breath too much either. Gaining an extra day of hurricane touchdown accuracy, for example, takes immense amounts of data. We've slowly made those gains, but for a neural net to do noticeably better there would have to be major problems in our original problem formulation.

Slightly off-topic:

I'd love an easy way to get conditional probabilities in my forecasts. Some sort of raw, fine-grained data allowing those computations would be a god-send. When a forecast says there's a 30% chance of rain each hour, do they mean we're definitely getting rain and don't know which hour (like the very tall, narrow thunderstorms often traversing west->east in tornado alley), do they mean we're getting a spotty amount of rain that entire period (drizzling off and on like in southeast Alaskan summers), do they mean there's a 30% chance the storm hits us and rains continuously and a 70% chance it skirts by (it's commonly easy to predict there will be a storm but not necessarily exactly where or when with respect to communities/times on the boundaries)? Those have drastic impacts on my plans for the day, and the raw data has that nuance, but it's not obtainable from any weather report I've seen.

ramon1563 months ago

Thanks nytime for another polarising article

surfingdino3 months ago

Why 'polarising'?

mandibeet3 months ago

My niece recently asked me why it's important to make weather forecasts. I had never really thought about it myself. But it is a very important job! Knowing the weather can affect decisions about travel, outdoor events and work schedules; forecasts help people prepare for severe weather conditions, such as storms, hurricanes or extreme temperatures, potentially saving lives and reducing injuries!

surfingdino3 months ago

Armies, emergency services, farmers, pilots, and sailors can't do their job without it.

nomel3 months ago

> pilots

Pilots usually need to fly regardless. They reroute based on present data, not future. Bad weather still usually cancels all flights at an airport since they can't afford to close down the x% of the time when the local forecast is wrong. I suppose the airport might get the de-icers out of the garage early, but prediction, rather than current state, seems less important for pilots.

> emergency services

Do you have an example? I believe staffing is almost constant for these, since most are union, until extra are needed. I would suspect the only difference would be "it's going to rain heavy next week. you guys on call should expect to be called".

surfingdino3 months ago

There are military, commercial, and private pilots. Each group has different look-ahead needs and approaches to planning and routing flights. If you want to give your brain a little exercise and learn new stuff, read up on various levels of pilots' licenses, endorsements, type ratings, etc. Add to that limits imposed by the aircraft (max operating altitude, speed, de-icing equipment, weight, etc.), airspace classes (how high can you/have to fly, will you be cleared to fly above weather), and jurisdictions (does my aircraft rating allow me to land at my destination given the weather forecast in that area). When you do that, you will understand how important weather forecasting is for pilots.

As for emergency services, it's good to know if/when things are going to get biblical, because you may need to transport additional machinery and personnel to another area, or you may need to prep temporary shelters.

nomel3 months ago

Sorry, I was only thinking in commercial context. I mistakenly excluded military with "armies" and private with "job".

> If you want to give your brain a little exercise and learn new stuff

Absolutely unnecessary.

shaggie763 months ago

I've never known someone so mindful of the weather forecast as my uncle the farmer: is it too soon to plant? Should I irrigate or wait just a few days longer? If I don't harvest now will my fields get too muddy for the tractor?

_neil3 months ago

Agriculture is a big one. We have an independent forecaster locally that posts on Facebook. People tend to check in with him around hurricane and snow storm times. It seems like his paying clients are mainly farmers.

SoftTalker3 months ago

Is the farmer's almanac no longer popular?

ohthatsnotright3 months ago

As a gag gift, maybe.

thesis3 months ago

Ah yes. The old 0% chance of rain today when it’s currently raining.

gorgoiler3 months ago

Nothing suits a generative AI better, in terms of quotidian prognostication, than telling you whether it will be vaguely sunny-ish in the next 24 to 48 hours.

Whatever next? Perhaps that most egregious example of just about vaguely getting it right enough of the time to seem plausible: AI horoscopes?

mdp20213 months ago

Irrelevant to the article content.

1) It is ML based forecasting, which does not fall in the category of "generative" in the typical contextual sense;

2) the relevant forecasting responses are falsifiable in a Popperian framework.