What is a good weather forecast? That’s a question that most people probably don’t think much about because the answer seems obvious – a correct one. But then again, most people are not CTOs at DTN. Lars Ewe is, and his answer may be different than most people’s. With 180 meteorologists on staff providing weather forecasts worldwide, DTN is the largest weather company you’ve probably never heard of.
Example: DTN is not included in ForecastWatch’s “Global and Regional Weather Forecast Accuracy Overview 2017 – 2020The report assesses 17 weather forecast providers based on a comprehensive set of criteria and a thorough method of data collection and evaluation. So how come a company that started in the 1980s serves a global audience and has always had a strong focus? on the weather, is not evaluated?
Weather forecast as big data and internet of things problem
The name DTN stands for ‘Digital Transmission Network’ and is a nod to the company’s origins as a radio farm information service. Over time, the company has made technological evolution, focused on providing what it calls “operational intelligence” for a number of industries, and has gone global.
Ewe has held previous positions in senior positions at a variety of companies, including AMD, BMW and Oracle. He values data, data science and the ability to provide insights to deliver better results. Ewe called DTN a global technology, data and analytics company, whose goal is to provide actionable near real-time insights to customers to better run their businesses.
Again as a service from DTN® (WAAS®) approach should be seen as an important part of the broader goal, according to Ewe. “We have hundreds of engineers dedicated not only to weather forecasting, but also to the insights,” Ewe said. He also explains that DTN invests in making its own weather forecasts for various reasons, even though it could outsource them.
Many available weather forecasting services are either not global, or they have weaknesses in certain areas, such as image resolution, according to Ewe. DTN, he added, uses all publicly available and many proprietary data inputs to generate its own predictions. DTN also extends that data with its own data entry, as it owns and operates thousands of weather stations around the world. Other data sources include satellite and radar, weather balloons, and aircraft.
DTN provides customers worldwide with a range of operational intelligence services and weather forecasting is an important parameter for many of them. DTN
Some examples of the higher order services that DTN’s weather forecasting can provide are storm impact analysis and shipping guidance. Storm impact analysis is used by utilities to better predict outages and plan and staff accordingly. Shipping guidance is used by shipping companies to calculate optimal routes for their ships, both from a safety point of view, but also from a fuel efficiency point of view.
At the heart of the approach is the idea of using DTN’s forecasting technology and data and then merging it with customer-specific data to provide tailored insights. While there are basic services DTN can provide as well, the more specific the data, the better the service, Ewe noted. What could that data be? Everything that helps DTN’s models to perform better.
It could be the position or shape of ships or the health of the infrastructure network. As such concepts are used repeatedly in DTN’s models, the company is moving toward a digital twin approach, Ewe said.
In many ways, weather forecasting is really a big data problem these days. To some extent, Ewe added, it’s also an Internet of Things and a data integration problem, where you try to access a set of data, integrate it, and store it for further processing.
As a result, weather forecasting involves not only the domain expertise of meteorologists, but also the work of a team of data scientists, data engineers, and machine learning/DevOps experts. As with any big data and large-scale data science task, there is a tradeoff between accuracy and viability.
Good enough weather forecast to scale
Like most CTOs, Ewe enjoys working with technology, but also needs to be aware of the business side of things. Maintaining accuracy that’s just right, or “good enough,” without making cuts while making it financially viable is a very complex exercise. DTN tackles this in a number of ways.
One way is by reducing redundancy. As Ewe explained, over time and through mergers and acquisitions, DTN came to own more than five forecasting engines. As was usually the case, each of these had its strengths and weaknesses. The DTN team took the best elements of each and consolidated them into one global forecasting engine.
Another way is by optimizing hardware and reducing associated costs. DTN worked with AWS to develop new hardware instances suited to the needs of this highly demanding use case. Using the new AWS instances, DTN can run weather forecast models on demand at unprecedented speed and scale.
In the past, it was only possible to run weather forecast models at fixed intervals, once or twice a day, because it took hours to run them. Now, according to Ewe, models can be run on demand and generate a one-hour global forecast in about a minute. Equally important, however, is the fact that these bodies are more economical to use.
As for the actual science of how the DTN models work – they include: both data-driven, machine learning models and models with meteorology expertise† Ewe noted that DTN takes an ensemble approach, uses different models and weighs them as necessary to arrive at a definitive result.
However, that outcome is not binary – rain or no rain, for example. Rather, it’s probabilistic, meaning it assigns probabilities to potential outcomes — 80% chance of 6 Beaufort winds, for example. The reasoning behind this has to do with what those predictions are used for: operational intelligence.
That means helping customers make decisions: should this offshore drilling facility be evacuated or not? Should this ship or plane be diverted or not? Should this sporting event take place or not?
According to Ewe, the ensemble approach is of crucial importance in order to be able to process predictions in the risk equation. Feedback loops and automating the selection of the right models with the right weights in the right circumstances is what DTN is actively working on.
This is also where the “good enough” aspect comes into play. The real value, as Ewe put it, is in consuming the predictions those models generate downstream. “You want to be very careful when balancing your investment levels because the weather is just one input parameter to the next downstream model. Sometimes that extra half degree of precision might not even make a difference to the next model. Sometimes, it does. “
Coming full circle, Ewe noted that DTN’s attention to the day-to-day operations of its customers’ businesses, and how the weather affects those operations, enables the highest level of safety and monetary return for customers. “That has proven to be much more valuable than having a third party measure the accuracy of our forecasts. It’s our daily customer interaction that measures how accurate and valuable our forecasts are.”