A 1986 paper by Frank Westheimer, cited 400+ times, abstract here and pdf here, explains why phosphates are preferred. In particular, he notes how the negative charge of the phosphate ester makes it relatively more resistant to hydrolysis, while it still can act as a leaving group if enzymatically activated. Then, in an intriguing section, he discusses why various other alternatives would not make sense, including arsenic:
Another compound that must be considered as a basis for a possible genetic material is arsenic acid, which is also tribasic. The poisonous effects, however, of compounds of arsenic probably cannot be avoided, since these effects are centered in the lower valence states of arsenic, and the reduction of pentavalent arsenic is much easier than that of pentavalent phosphorus. In any case, arsenic esters are totally unsuitable; the hydrolysis of esters of arsenic acid is remarkably fast. The triisopropyl ester in neutral water at room temperature is completely hydrolyzed in a couple of minutes. Apparently the hydrolysis of the diesters is even faster than that of the triesters.The idea is that esters of arsenic are too liable to be cut by water, thus making them poor linkers for bases of DNA and RNA. But this is assuming that the reaction will occur under relatively stable conditions (i.e., pH and temp), and perhaps those conditions are somehow altered in this particular bacterium, sufficiently lowering the hydrolysis rate of arsenic. We will have to wait and see, but in the meantime where are the prediction markets when we need them?