By Nell Mackenzie

LONDON (Reuters) -A wager that stock markets would stay calm has cost retail traders, hedge funds and pension funds billions after a selloff in global stocks, highlighting the risks of piling into a popular bet.

The CBOE VIX index, which tracks the stock market's expectation of volatility based on S&P 500 index options, posted its largest-ever intraday jump and closed at its highest since October 2020 on Monday as U.S. recession fears and a sharp position unwind have wiped off $6 trillion from global stocks in three weeks.

Investors in 10 of the biggest short-volatility exchange traded funds saw $4.1 billion of returns erased from highs reached earlier in the year, according to calculations by Reuters and data from LSEG and Morningstar.

These were bets against volatility that made money as long as the VIX, the most-watched gauge of investor anxiety, remained low.

Wagers on volatility options became so popular that banks, in an effort to hedge the new business they were receiving, might have contributed to market calm before the trades suddenly turned negative on Aug. 5, investors and analysts said.

Billions flew in from retail investors but the trades also garnered the attention of hedge funds and pension funds.

While the total number of bets is difficult to pin down, JPMorgan estimated in March that assets managed in publicly traded short volatility ETFs roughly totaled $100 billion.

"All you have to do is just look at the intra-day rate of change in the VIX on Aug. 5 to see the billions in losses from those with short vol strategies," said Larry McDonald, author of How to Listen When Markets Speak.

But McDonald, who has written about how bets against volatility went wrong in 2018, said publicly available data on ETF performance did not fully reflect losses incurred by pension funds and hedge funds, which trade privately through banks.

On Wednesday, the VIX had recovered to around 23 points, well off Monday's high above 65, but holding above levels seen just a week ago.

VOLATILITY'S RISE

One driver behind the trading strategy's popularity in recent years has been the rise of zero-day expiry options - short-dated equity options that allow traders to take a 24-hour bet and collect any premiums generated.

Starting in 2022, investors including hedge funds and retail traders, have been able to trade these contracts daily instead of weekly, allowing more opportunities to short volatility while the VIX was low. These contracts were first included in ETFs in 2023.

Many of these short-term options bets are based around covered calls, a trade that sells call options while investing in securities such as U.S. large-cap stocks. As stocks rose, these trades earned a premium as long as market volatility remained low and the bet looked likely to succeed. The S&P 500 rose over 15% from January to July 1 while the VIX fell 7%.

Some hedge funds were also taking short volatility bets through more complicated trades, two investor sources told Reuters.

A popular hedge-fund trade played on the difference between the low volatility on the S&P 500 index compared to individual stocks that approached all-time highs in May, according to Barclays research from that time.

Hedge-fund research firm PivotalPath follows 25 funds that trade volatility, representing about $21.5 billion in assets under management of the roughly $4-trillion industry.

Hedge funds tended to bet on a VIX rise, but some were short, its data showed. These lost 10% on Aug. 5 while the total group, including hedge funds that were short and long volatility, had a return of between 5.5% and 6.5% on that day, PivotalPath said.

'DAMPENED VOLATILITY'

Banks are another key player standing in the middle of these trades for their larger clients.

The Bank of International Settlements in its March quarterly review suggested that banks' hedging practices kept Wall Street's fear gauge low.

Post-2008 regulations limit banks' ability to warehouse risk, including volatility trades. When clients want to trade price swings, banks hedge these positions, the BIS said. This means they buy the S&P when it falls and sell when it rises. This way, big dealers have "dampened" volatility, said the BIS.

In addition to hedging, three sources pointed to occasions where banks hedged volatility positions by selling products that allowed the bank to even out its trades, or remain neutral.

Marketing documents seen by Reuters show that Barclays, Goldman Sachs and Bank of America this year were offering complex trade structures, which included both short- and long-volatility positions.

Some, according to the documents, do not have a constant hedge built into the trade to buttress against losses and are protected "periodically," the papers say. This might have exposed investors to higher potential losses as the VIX spiked on Aug. 5.