You apparently have asked a difficult and somewhat "personal" question. The use of "Terminal Tackle" (See terminal tackle/wikipedia:http://en.wikipedia..../Fishing_tackle ) is not a cut and dry answer.
You see, it all depends on how a person learned to fish. Some were taught with and some without.
So, I guess we'll look at the "Facts", as I see them.
Not all situations are equal and not all fish are equal either. The use of terminal tackle depends on the species, conditions and "how educated the fish are".
Species: For freshwater fishing certain species can be far more "aware" of terminal tackle. Trout for instance can be very fussy when it comes to terminal tackle. Brown Trout are very discriminating. Speckles can be somewhat fussy. Rainbows can be at times discriminating and at others they'd bust on a Buick.
Smallmouth can be fussy, but it all depends on the situation. Certainly, rigs like the Carolina or Texas Rig don't seem to phase the fish too much. Yet other times you are forced to present a whacky rigged worm with little in the way of terminal tackle.
Chain Pickerel...they're just feeding machines and I have never considered them a discriminating fish. I've seen a Pickerel hit a white perch with a big spinner hanging out it's mouth. Not so discriminating.
Other species of freshwater fish are usually not terminal tackle shy, in my experience.
Conditions: First, let's start with water clarity. Most of your terminal tackle is a visibility issue. So the effect it has on your fishing will depend on how clear the water is where you are fishing. Muddy or stained the fish aren't going to notice as much as if you are fishing gin clear water. Thankfully (or not) you won't run into many gin clear conditions here.
The second condition I'd consider is the sky. Is it sunny, cloudy, windy? Depending on the species, the reflective "flash" or silhouette of the terminal tackle on your line may make a difference. Again depending on the species targetted. So you might want to consider the colour of your weight or swivel in these situations.
The last condition I would consider is that of the fish. Are they actively feeding? Are they finicky?
The last factor, "How educated are they", is very important too. Fish aren't as dumb as some would like to think. The more pressure an area gets, the more educated the fish will get. Places that don't see a lot of pressure from other anglers will produce far less discriminating fish. Easy access and strongly pressured areas will produce fish that will be more discriminating and less quick to the hook.
It's these educated fish that need to be addressed with a little more thought. Sometimes it just takes throwing something they haven't seen before. Other times it will take some finessing.
Basically, all fish and all fishing conditions are not equal. The more you know about your quarry, the better the decisions you will make when setting up your tackle.
Personally, when I use a swivel it's usually the smallest I can get away with depending on the weight of the fish I'm expecting to catch. For most situations you can get away with a #12 swivel. I prefer black for the colour.
I like to use foam or rubber core weights over split shot when possible. Less chance of damaging the line and they are adjustable.
So that's my take on an answer to your question. Guaranteed others may feel differently, depending on their own experiences.
Take from it what you think is logical.