At the beginning of the summer, the AirHelp ranking of the best airlines was much talked about. Some saw it as a new opportunity to praise their favorite airline, others to criticize the ones they don’t like, but in the end, many voices were raised to discuss its relevance.

Airline rankings: a never-ending debate

That’s the thing about airline rankings: they are as much an opportunity for passionate debate as they are for discussion, not to say doubt. The opportunity for us, at TravelGuys, to say what we think about it.

Let’s go back to the AirHelp rankings to start and look at the methodology. The average delay is an objective data and may be subject to discussion. Quality of service is another. They talk about taking into account multiple indicators but it remains a bit obscure. And obviously this is not based on customer feedback, even though we know that when it comes to service, it’s perception that counts: the data (since that’s what we’re talking about) gives, when it’s relevant and accurate, a cold, “absolute” truth, which can be in total contradiction with the customer’s feelings. The last criterion is the efficiency of the customer service, measured on how complaints are handled.

Let’s focus on this last point. Passenger compensation is AirHelp’s business. They therefore have easy access to the compensation data of their own customers, but without any guarantee that this constitutes a representative database of all passengers and airlines.

When you read a study, ask yourself who is funding it

Generally speaking, there are two types of players in the research/ranking world: those whose job it is and those whose job it is not.

There are players whose job is to make rankings, we will talk about them later. There are others, such as AirHelp, whose job is quite different but who produce studies or rankings related to their sector of activity: this is called marketing. Let’s be clear: the goal is not for the ranking to be reliable but for the business to be talked about and for its reputation to increase. And here the shot is perfectly successful.

I forgot a third case: those who are not in the business but who do things properly. For example, in March Flighright did a study on the rights of air passengers with the Odoxa Institute. Odoxa is a reputable pollster and guarantees in this case that the sample surveyed is indeed representative and that the questions are not biased. It’s still marketing, it’s still a question of notoriety, you can make the figures say what you want but the figures are sincere and representative.

In short, no matter what the subject is, when a ranking or a study comes out, always look at who is behind it…it will often help you guess what comes out without even reading it. A study on online banking by in Fintech, a study on speed limits made by the government or on the contrary a car manufacturer.

In short, do what you want with the AirHelp ranking, it contains things that are a priori logical, others much more questionable but they are clear on the methodology. If the reader wants to fool himself, he is free to do so.

But are other rankings more reliable?

Desperately seeking a Skytrax agent

The authoritative ranking, the Michelin of the airline industry, is Skytrax. It is the reference but it is not without its faults either. No one understands that Lufthansa, for example, gets 5 stars on the basis of the promise of a product that has just been presented and will not be put into service for years. Even the German carrier must have thought it was a joke.

Turkish, for example, which for years has made the many awards it has received from Skytrax its battle horse, has just decided to no longer appear in the ranking. If the top of the ranking seems coherent, we can be surprised by some incongruities after the 5th position…

Specialized forums like FlyerTalk are full of discussions on the subject and for many the Skytrax awards go to those who pay the most.

As proof, Skytrax works in two ways. Customer reviews posted on their website and impromptu passenger surveys. I recently did a little survey around me, which is only worth what it is worth. With a group of people, many of whom have been flying around 100 times a year for years:

– no one leaves reviews at Skytrax

– only one has ever seen a SKytrax agent (we’re talking about a person who at one point in his life flew enough to be platinum on the 3 big airline alliances!)

If Skytrax is the reference, it is surely for lack of better because too many grey areas remain as to the objectivity of the ranking.

What about the APEX Awards?

Another frequently mentioned ranking is the Airline Passenger Experience Association (APEX) ranking. It is an association of professionals of the sector (basically the service providers and suppliers of the airlines) that rewards the things actually accomplished over a year.

Positive point: the knowledge of the matter.

Negative points: it’s a bit consanguineous because at the end of the day things are awarded that the airlines have done with the help of some of them and, above all, the voice of the customer is absent from the scheme. This is quite something when you present yourself as the association of the passenger experience.

But their awards are no more debatable than Skytrax, if not less so.

The difficulty of evaluating an airline

An airline is not a uniform whole, many parameters are considered.

First the type of flight. Between long and medium haul it can be night and day depending on positioning, choices and priorities. One can be very good at long-haul and cut back on medium-haul services. Or the opposite: being good on medium haul but not being able to deliver a consistent experience on long haul.

The main airport. Delays are not always the fault of the airlines, far from it. But when you have your main base at a congested or poorly managed airport you will have a higher delay rate than your competitors who only operate a few flights a week there.

The aircraft. Not all devices are equally comfortable. Then there is what the airlines do with it: the interior design is their responsibility. And sometimes within the same fleet, for the same aircraft, several types of cabins coexist. Take the 777 of Air France between the so-called “Best” cabins and the NEV 4 with slide seat that can be found in business class depending on the destination it’s night and day. Ditto in eco or pitch, seat inclination and screen size have nothing to do with each other.

The travel class. Well yes, one travels better in the front than in the rear. But some airlines are very acceptable on the eco class and have difficulties to propose a service in relation with the price charged in the front classes while for others it is exactly the opposite.

The Crew. You have the best aircraft with the best cabin and the service offered looks great? Bad luck: a bad crew can ruin your flying experience. On the other hand, a good crew can make you forget an “average” product.

So many factors make that no two flights are alike and that it is difficult to make generalizations about an airline and therefore to say in an absolute way that this one is systematically better than that one. Yes Emirates is better than Lufthansa or Air France. But you will also find examples of passengers who had exactly the opposite experience.

Very often when someone asks me what I think of an airline I answer “it depends, where are you going? This means everything.

The important thing is the passenger experience, the rest is gruel for experts

If there is a certain variability in what an airline delivers, we should still be able to draw reliable statistics, at least in trend, on the mass of flights. Yes, but…

It doesn’t matter if an airline is rated 7.2/10 for its crews and 8/10 overall. The reality is that on a given flight it will be 9 or 4…. More often 9 than 4 but 4 is possible. And for the passenger who will be confronted with the second situation, his experience will not be an 8/10 but a 4 and he won’t care that 80% of the passengers had a better experience than him.

Rankings are made to help the customer to choose his flights but his flight will not be an average, it will be an element of this average. What these rankings should give us is not just the average but the distribution around the average, in other words the variability of the quality of the service provided. Many airlines are capable of the best, the question is how far they can deviate from their standard. And for some this happens far too often. And maybe that’s what we need to know.

So let’s continue to have passionate debates at every ranking release but let’s just take them for what they are.

Photo: customer experience by Black Salmon via Shutterstock.