Three-time Grand Slam champion Murray, 36, was beaten 7-6 (7-3) 6-1 by Rublev in a high-quality match in California.
Murray, who has said he is not planning to "play much past this summer" as he considers retirement, missed four set points when leading the first set 5-4.
He then quickly faded in the second after losing serve in the fourth game.
Murray, who finished runner-up to Rafael Nadal at the tournament in 2009, thanked the Indian Wells crowd on what he acknowledged would be his final visit.
"I've loved coming to this place over the years," he said, praising the support from fans which had allowed him "to play in some amazing atmospheres and create brilliant memories - because without them it's not the same".
Elsewhere, Australian Open champion Jannik Sinner progressed with a straight-set victory over Australian Thanasi Kokkinakis.
The Italian world number three, 22, took just 81 minutes to seal a 6-3 6-0 win and will play German Jan-Lennard Struff or Croatia's Borna Coric in the third round.
Defending champion Carlos Alcaraz, the second seed, dropped the first set against Italy's Matteo Arnaldi but the Spaniard recovered to win 6-7 (7-5) 6-0 6-1.
German sixth seed Alexander Zverev won 6-4 6-4 against Australia's Christopher O'Connell, while Greek 11th seed Stefanos Tsitsipas defeated French qualifier Lucas Pouille 6-3 6-2.
Having claimed a straight-set win over David Goffin to reach the second round, Murray delighted the Indian Wells crowd in a high-quality opening set against Rublev and a stunning backhand passing shot on a crucial point allowed him to hold serve for a 5-4 lead.
It looked as though that may prove pivotal as he maintained momentum to create a total of four set points in the following game.
But Rublev hung on, forcing a tie-break which he would clinch at the second opportunity, to leave Murray empty-handed despite a promising display in a 66-minute first set.
Well supported on court two, Murray continued to display flashes of brilliance and beat Rublev with a perfectly executed lob as the players fought for an advantage in the second set.
But, first to face pressure points in a service game he had led 40-0, Murray could not withstand a third break point as Rublev moved 3-1 ahead.
The 26-year-old Rublev, who won an appeal after being defaulted from last week's Dubai Tennis Championships, would not let up from there - breaking Murray again before eventually taking his fourth match point.
Former world number one Murray has said he hopes to compete at another Olympic Games before he retires, but to achieve that he must be inside the top 56 of the ATP rankings on 10 June.
The two-time Olympic champion, who turns 37 in May, is currently ranked 61st and won just two of his eight matches in the lead-up to Indian Wells.
Copyright © Batbricks7© 2020