The 2-1 loss came as thousands of West Ham fans staged a protest march before kickoff against the board.
Afterwards, Potter said: "It’s a tough period, there’s no getting away from it, with the results, especially here.
"It’s a tough one for the team, it’s a tough one for the supporters, who I thought were brilliant with us today, to be honest. I thought the way they pulled with the team in the second half was fantastic. I feel for them because it’s tough.
"The scoreline is what it is and we’re disappointed with that of course. I thought we started the game quite well, but then struggled to get some pressure and then they found spaces and getting their pocket players in, and five against four in the last line destabilised us a little bit.
"But we tried to change it and we did at half-time, it affected it a little bit and we were a lot better. I thought the first part of the second half was really positive.
"It’s just disappointing that we conceded and we probably just huffed and puffed a little bit at that point."
Not good enough
Potter alsoo said, "It’s this level and at the moment it’s not enough and we have to face up to that.
"We have to do more, we have to work harder, we have to find a solution to it. We have to be more stable defensively and offensively and find more ways to look stronger as a team. That’s where the work is now.
"We know we have to keep working. You have to face these moments up when it’s tough and it is tough at the moment.
"I said to the players, 'we have to stick together, work hard, keep on working on finding the solution and come out of this tough period stronger for it'. That’s the aim now."