"Assuming there is s Joseph."
A silly and unwarranted challenge, given that John twice identifies Jesus as "the son of Joseph" (1:45; 6:42), but never identifies Mary as Jesus' mother in his 3 references to Jesus' "mother" (2:3, 5; 19:26)! Does your church know you question whether Joseph was Jesus' father? The triple attestation of Joseph as Mary's husband in 3 Gospels offsets the fact that Mark refers to Mary as Jesus' mother once, but not to Joseph. Nor would we expect a reference to Joseph, since Joseph is apparently already deceased and Mark has no virgin birth narrative.
GordW: "All of which assumes some remembered history about the birth story at all."
My rebuttal is inspired by English scholar Richard Bauckham's excellent book, "Jude and the Relatives of Jesus in the Early Church:"
Julius Africanus has ancient Jewish Christian sources that imply just such a "remembered history:
"From the Jewish villages of Nazareth and Kokhaba, they (Jesus' family members) traveled around the rest of the land and interpreted the genealogy they had [from family tradition]."
The reference to the obscure Galilean village "Kokhaba" just 16 kilometers from Nazareth lends credibility to this report.
Kokhaba was a main center of ancient Jewish Christianity (so Epiphanius, Pan. 29.7.7). Paul already attests the itinerant missionary work of Jesus' brothers:
"Do we not have a right to be accompanied by a believing wife, as do the other apostles and THE BROTHERS OF THE LORD and Cephas (Peter) (1 Cor. 9:5)?"
We don't know which genealogy Jesus' brothers interpreted, but that is irrelevant because the main purpose of both the Matthean and Lucan genealogies is to establish Joseph (and therefore Jesus) as a descendent of David and hence as the promised Davidic Messiah. Of course, the genealogies go with the virgin birth narratives, which are also intended to establish Jesus as the Davidic Messiah born in Bethlehem, the city of David. This likely means that Jesus' brothers originally circulated virgin birth stories, which they would have heard from their mother, Mary.
The text of Matthew does not support prior residency in Galilee."
But that does not mean that Matthew is unaware of Joseph's prior residence in Nazareth. Remember, Matthew's agenda is to establish Joseph as a descendent from David to facilitate Jesus' status as the expected Davidic Messiah. So it does not serve Matthew's purpose to admit that Joseph was fleeing to Bethlehem, the city of David, from Nazareth to protect Mary from her perceived adultery scandal (see Mark 6:3). And of course, Matthew never denies that Joseph originally lived in Nazareth.
That may be a literal virtue to you ... but consider the effects of so many lies and delucions ... the squeeze is on ...