The Briton, 36, secured his ninth break point to take a 6-5 lead in the opening set before serving out to love.
But Machac hit back to win the second set and then clinched the match in a third-set tie-break, having previously been broken when serving to progress.
Murray received treatment on an ankle injury late in the third set and was also agitated by movement in the crowd.
Elsewhere, British number one Cameron Norrie lost 7-5 6-1 to third seed Daniil Medvedev in just 81 minutes.
Murray's match against 23-year-old Czech Republic player Machac lasted three hours and 27 minutes and saw him save match point to force a tie-break.
After the Briton secured the opening set, Machac broke Murray for the first time at 6-5 in the second - which was interrupted by a brief rain delay - to draw level.
But Murray's frustration grew with people in the crowd moving and umpire Carlos Bernardes at the end of that set.
In response to an unheard reply from Bernardes to Murray's "useless, useless" comment, the Scot responded: "What are you talking about? It's quite obvious when people are standing up between every single point."
Machac raced into a 3-0 lead in the decider, but Murray retrieved a break and saved match point when serving to stay in the match before succumbing to defeat when he lost the tie-break.
The Scot had worryingly gone down with an ankle injury at the net after hitting the winner to level at 5-5 in the third set.
He received treatment from the physio and managed to continue but could not overcome a resolute Machac, who is ranked two places lower than 58th-placed Murray.
Norrie, 28, conceded two early breaks in the opening set against Medvedev and left himself with too much to do.
He was blown away by Medvedev's serve in the second set, with the Russian winning eight of nine first-serve points.
The world number three was happy to progress, but expressed concerns about the balls being used after the win.
"It's a bit strange," said Medvedev. "The court is pretty fast but the balls get old very fast.
"So at one moment in the rally, you feel like there isn't much you can do, so you can't go for the winner because the risk reward isn't there.
"It's the same for the opponent, so we just hit, hit."
Meanwhile, Australian Open winner Jannik Sinner lost the first set to the Netherlands' Tallon Griekspoor before the Italian recovered after a rain break to win 5-7 7-5 6-1.
Norway's seventh seed Casper Ruud progressed with a 6-3 6-4 victory over Spain's Alejandro Davidovich Fokina.
However, Denmark's Holger Rune suffered a 6-1 6-1 defeat by Hungary's
Copyright © Batbricks7© 2020