First you need alot of power to shoot them, but that means it's heavy and basically bolted down to the floor/ship etc. That is fin if your basically having it shoot in 1 set direction and only increasing the height of the ballista. If you have it able to move left and right as well, shooting it will totally thrash your aim (it'll jerk it while being shot...could even break it for that matter.
Second, just like any long distance weapon, you generally need a few shots before you get a good feel for where it's gonna go, heck even bows at long range generally need that (unless you have practiced the shot at that range many times before). The likely hood that any of them would have hit a moving dragon so far away.....was basically non-existant.
Third, actual ships fought in the real world, and those ballistas were WAY more effective then cannon shots lol, at a much...much further range. Why wouldn't anyone have ever used them if that was the case (heck didn't even need gunpowder!!).
Also as mentioned by many, even if such a contraption could exist that would shoot with enough power to reach even close to the distances, and that release wouldn't have destroyed the weapon or thrashed any kind of aim, the only way to get that power would be to wind it up with alot of manpower, and it would take a long time between shots....a very long time.
That is of course outside of the whole stealth fleet against a FREAKING DRAGON IN THE SKY, I can ...sort of understand how maybe Danny's own fleet might not see them (not really to be honest it's not like they are on a river or whatever), but how do you sneak up on a dragon?
To be honest those weapons were more effective in some ways then modern weapons we currently have lol. They had at least the range of a mounted machine gun for instance, and seemed to do more damage against a large craft then one would do. They freaking destroyed the enemy ships into pieces.
So yeah, unless magic was involved in making them (and such magic has not really been seen in the GoT universe yet), I don't think there is any way they could have done what they did in the show, other then the writers just decided it was gonna happen.