Williams, watched on by her family in the stands, lost a tight first set before leveling the match in the second on Centre Court, the site of seven of her 23 grand slam singles titles.
She was a break up in the third, but Tan showed her resilience by fighting back to take a thrilling final set in a nail-biting tie break.
Asked if it was the final Wimbledon match of her decorated career, Williams said that was a question she “can’t answer.”
“Who knows where I’ll pop up?” she added.
On Tuesday, she made her long-awaited return to singles, one year on from retiring in her last first-round match at Wimbledon.
Momentum shifted between the two players over the course of the three-hour, 10-minute encounter — first in Tan’s favor as she edged the first set, then towards Williams as she rallied in the second.
It looked as if the 40-year-old Williams would prevail in the deciding set as she served for the win, but the memorable return wasn’t to be.
Tan broke back at 5-5, and despite failing to convert a match point at 6-5, made no mistake when she had a second chance in the tie break.
Ranked No. 115 in the world, Tan was playing in her first match in the main draw of Wimbledon and ninth across all grand slams.
“When I saw the draw, I was really scared,” she said after the match. “It’s Serena Williams — she’s a legend. I was like, ‘Oh my god, how can I play?’ And if I can win one game or two games, it’s really good for me.”
Under the lights of Centre Court, Tan comfortably did more than that — triumphing against one of the greatest athletes sport has ever seen.
“For my first Wimbledon — it’s wow. Just wow,” she added, struggling to find the words to capture her emotions.
It remains unclear whether this was Williams’ final outing at Wimbledon. Asked to sum up her legacy at the tournament, she was succinct, if understated.
“I think I’m pretty solid on the grass,” she said. “Maybe not today, but pretty solid out there.”
Nadal, Swiatek advance
In Tuesday’s earlier games on Centre Court, there were victories for this year’s French Open champions Rafael Nadal and Iga Swiatek.
Nadal, a two-time winner at Wimbledon, survived a scare to beat Francisco Cerundolo 6-4 6-3 3-6 6-4 as he begun his pursuit of a 23rd grand slam title, while Swiatek recorded her 36th consecutive win by beating Jana Fett 6-0 6-3.
Nadal, who missed last year’s tournament due to injury, received a warm reception on Centre Court, where he was sternly tested by Argentina’s Cerundolo.
The Spaniard looked on course for a routine win when he was 2-0 and a break up in the third set, but the free-swinging Cerundolo fought back with a double break to take the set.
The match looked to be heading for a decider when the world No. 41 broke in the fourth, only for Nadal to save his best until last and seal the win with a double break.
Swiatek, meanwhile, dominated the first set against Fett in her first match since the French Open, but had to overcome a tough period in the second when Fett came close to securing a double break.
The top seed recovered from the shaky start to the set and rattled off five games in a row, increasing her winning run to 36 matches.