Was Invisibility Plausible?

If non-magical folk set their eyes on Hogwarts they would see the castle marked condemned and dangerous. We take it that if they dared walked passed 'do not enter' signs they wouldn't see the glorious Great Hall, common rooms, etc. They would see only rubble, brick, mortar.

Platform 9 3/4 Quarters, perhaps at King's Cross Station is another obvious example of concealment. To get to the station for Hogwarts Express kids and grown adults run through a wall in the middle of a busy train station to land at their hidden train landing. Rowling's solution: Wizards and witches would be on-hand to perform memory charms to anyone who witnessed individuals disappearing through the brick wall.

This is the same case when Harry and Ron flew Arthur Weasley's car into the sky in Chamber of Secrets. Before switching the car into invisibility, camouflaging it into clear blue skies, several muggles saw it flying overhead. The Ministry of Magic took immediate actions to wipe the sighting out of their minds. No harm, no foul.

However, in other instances a lack of invisibility or concealment are a bit unexplainable.

A giant husky hairy...giant is walking with a young boy into an abandoned and pretty-much condemned building. Nobody notices? Nobody alerts the authorities of this unusual sighting?

At the Quidditch World Cup in Goblet of Fire the Ministry of Magic takes notes that muggles again might be out in the middle of their open meadowy fields vacationing..? It's not really explained why or how witches and wizards in security posts would deal if muggles saw the festivities, but again it's assumed memory charms.

In Book 5 Order of the Phoenix, St. Mungo's Hospital is concealed as a rundown shop in a busy London marketplace. A woman with her friend hastily notes that this place is never ever opened. When the Order of the Phoenix takes Harry to St. Mungo's, and the Weasleys visit Arthur in the hospital, are there memory charm experts on hand every single day to make humans forget seeing people disappear into abandoned buildings?

Are they wherever wizards might be seen doing something magical and logically unexplainable? How would the magical and muggle government deal if memory charms backfired on their targets - a la Gilderoy Lockhart using Ron's defected wand and his charm backfiring?

For individual cases, invisibility cloaks are extremely rare. Only Harry Potter - as far as we know - is the owner of one. He was able to wear it and scamper around Hogwarts unnoticed to visit Hagrid and leave to Hogsmeade, among other activities. The Disillusionment Charm is utilized to make objects near chameleon-like but these are known to wear away over time and aren't as reliable if there is spell or charm difficulties.

Clearly, magical folk are capable of hiding entire buildings. Considering the locations where some agencies or buildings are closely located near muggle-heavy destinations, would it have been more feasible if the wizarding world existed entirely on its own - in an alternate society perhaps? Should  invisibility been a talent witches and wizards could perform at will?