Peterborough Sports compete in National League North and former Hull City manager Brown admits he didn't see himself working in the nation's sixth tier.
"Did I want to come back to this level? No, but football has a weird way of playing its hand and I'm now here, working with one of the most supportive chairmen I've worked with," said Brown to BBC Sport.
"Everybody remembers the highest level (with Hull), but if I did something here with Peterborough Sports, there's no doubt in my mind it would be a great achievement.
"If I can keep Peterborough Sports in this division, first and foremost... if we survive this year, which is our first goal, then it's all about next year and can we get promoted? Can we move this club on?"
For himself, Brown also has ambitions he's still eager to fulfil.
"I saw (former Luton and Tottenham boss) David Pleat watching a Peterborough United Under-23s game and he said 'how are you dealing with them being part-time?' Brown said.
"He actually gave me something that I'm going to try to use, just a little snippet of information that he used when he was a part-time manager in non-league football.
"He's 14 years my elder, I didn't realise he had gone part-time, but it was in the early stages of his career not at the end.
"I don't want it to be the end of my career, I want this to still be a platform for getting back to the Football League."