Don't we also have to somehow set the upstream branch?

Clone works by looking for the advertised remote branch whose SHA-1 is the same as the advertised remote's HEAD branch.  The first such branch with the same SHA-1 is used as the name for the local branch, rather than assuming the default of "master".

If no branches exist in the remote, then yes, we assume master.  Otherwise its based on what HEAD was also named.