if we consider Einstein’s conception, all of space-time is one-piece or one set (so to speak). Now not all sets have borders. In math, open sets, such as an open line segment, does not contain its endpoints (e.g. the open set from 0 to 1 exclusive contains neither 0 nor 1). The universe may be like that, a set without beginning nor end even if it had a ‘big bang’. Dr. Penrose proposed something more interesting with his metric free model whereby at some point (pardon the pun), the end is the beginning (since all energy and entropy is configurational and a rescaling shows a bijection between the two).

In any case, natural language has ambiguities that mathematical language tries to avoid. ‘Universe’ and ‘beginning’ sound straightforward but models seen in math and physics indicate those concepts are indeed too subtle for natural language alone.