I posted this to help another player to address a similar issue, and it seems relevant to post here as well. Here are my tips:
First, you need to be well supplied. I would say 40 cannonballs along with 25 pieces of wood should be more than enough.
Second, you need to use the environment to your advantage as much as possible. My favorite tactic is to kite them into an area where you can play to the sloop's advantages and take advantage of the skellies questionable sailing techniques. Think about cutlass cay, rapier cay, or shipwreck bay. Those islands are great because they are an island situated within a perilous set of rocks that border them. With your sloop, you can easily slice and weave between the rocks and islands, and force the skelly gally to take on water as they continually crash into obstructions they are not nimble enough to move through.
Third, you need to ensure the majority of your broadsides hit below the waterline for the gally. Skellies cannot bail water (as others have noted here), and that means the better your shots are, the faster that irreversible damage pours into the ship. With enough supplies and good shooting, victory is only a matter of time.
Fourth, if you can find cursed cannonballs; especially a ballast ball, you can end the fight much more quickly. Pepper the gally until it has taken on a deck of water (and is riding a level below the water line) and hit it with the ballast ball. From there, 3-4 follow up shots should sink it outright.
Finally, if you stumble on a passive skelly ship going about its business, wait until it pulls close to an island and slows up (this requires you to have a gunpowder keg). Pull your ship alongside it (they won't shoot), jump into the water and board their ship. You can sprint down two decks and light the GP keg off on the bottom deck. Even if you die immediately afterword, you still have done a whole floor of irreversible damage to the skelly ship. I have never had an issue getting back onto my ship to finish the fight even in the event of an untimely demise running this scheme.
Anyways those are some tips I have. So long as I am prepared, I end up winning the majority of those fights solo; especially against the passive ones.
Don't be discouraged, with proper tactics, you will be more than a match for em @EmperorAss
I do agree with many on this thread that the skellies should give up after a while; the relentless pursuit is a bit much if you don't feel like engaging. I feel escape should always be an option since it is for nearly every other type of encounter (kraken, megs, players, etc.)