By Byron Kaye and Rishav Chatterjee

(Reuters) - Australia's No. 4 lender ANZ said on Monday that surging demand for its institutional banking services propelled its annual profit to a record but an aggressive campaign to sell more mortgages flattened its margin, sending its shares lower.

As Australia's banks redirect focus away from their traditional earnings engine of mortgages where interest rate hikes have spurred competition, ANZ has benefited from an institutional payments platform that it says processes some of the world's biggest cross-border transactions.

That pushed operating income from the bank's institutional unit in front of its retail unit, by dollar value, from March to September, and helped the Melbourne-listed company grow annual profit 14% to A$7.4 billion ($4.7 billion), just missing a Visible Alpha consensus forecast of A$7.56 billion.

But analysts expressed concern about a faster-than-expected narrowing of profit margin from the bank's retail unit, the only one of Australia's so-called "big four" lenders which has persisted with offering cash handouts to lure mortgage customers looking for a cheaper deal.

Even as ANZ's institutional division grew its net interest margin (NIM) - the interest it collects on loans minus interest paid to deposit-holders - the bank's overall NIM declined 10 basis points to 1.65% in the six months to September, more than NIMs reported this month by rivals Westpac and National Australia Bank (OTC:NABZY).

ANZ shares were down 3.1% by midsession, against a flat overall market, amid worries about whether the bank was sacrificing profitability for mortgage volume which had lagged in previous years.

"We expect questions to be raised about margin/volume management in the Australia retail division, particularly due to NIM pressure stemming from ANZ's relatively aggressive growth in Australia home lending," E&P Capital analyst Azib Khan said.

ANZ CEO Shayne Elliott denied forgoing margin to grow mortgages faster than the market, and challenged comments from other banks which have said they were intentionally slowing mortgage growth while competition eroded profit.

"The fact that others have stepped back from the market: I think there's a lot of people rationalising their loss of share," he said on a call with journalists.

"That's for them to answer, not me. All I know is we've been winning more customers than we have historically."

ANZ declared a final dividend of 94 Australian cents per share, up from 74 cents a year ago.

(This story has been refiled to correct typo in the headline)