Preservation of Ruins

I live on private property (an HOA) that has plantation ruins - walls and a sugar mill. My understanding was that there were, unfortunately, no rules regarding preservation for ruins on properties purchased as far back as ours was. We have spent several thousand $$ to keep the high walls from collapsing, so we are being very diligent in keeping what's left. The low walls are incredibly sturdy and aren't going anywhere in the next century or so. But I was recently told it was illegal to even walk on the ruins' walls. Is this true?

I have searched online, including LexisNexis, and can't find laws addressing the ruins. I'm probably looking in the wrong place? Can anyone steer me in the right direction as to laws regarding these ruins? We want to do the right thing, while still enjoying these beautiful structures. Just not sure how far we need to go. (And, to avoid any "just stay the he!! off of them" comments, I'm simply referencing landscapers, the occasional visitors who want a pic, and an adventurous child or two. Not steady traffic, by any means).