If the spell is on a pc’s class’s spell list and it has components (verbal, somatic, or material), they know if the spell is being cast (e.g. if the wizard sees someone casting Fireball, they recognize it from the arm movements, magic words, and the smell of guano and sulfur). If it isn’t, giving the player a DC (10 + level of the spell) Intelligence (Arcana) check to identify it would also be fair.

So, RAW here is how a round with a spell is supposed to work:

1. Character A announces that they are casting a spell. The name of the spell, and other information such as its level are not mentioned.
2. There is a short pause to allow someone to use a Reaction.
3. If no one uses a Reaction, Character A either rolls or tells people what they need to roll.
   • Side note: This is where someone could technically cheat by changing their spell slot level, and is one of the many reasons why Counterspell is a terribly-designed spell.
4. After this roll (and any effect that would apply to those rolls), Character A describes the effect and can optionally state what the spell was:

You all take 36 Fire damage, as an explosion of flame blooms at your feet from my character’s 6th-level Fireball.

RAW, Counterspell would occur during that second step. The creature that casts it has no idea what the spell they’re countering is, beyond context clues (i.e. they’ve seen that armored spell caster has been casting spells that heal their allies earlier).

As you said, there are rules to identify a spell. They were added by either Tasha’s Cauldron of Everything or Xanathar’s Guide to Everything. A character can use their Reaction to identify the spell. This usually means you’ll need 2 spellcasters working in conjunction for Counterspell to work with an identified spell.

As for how I run it at my table: I don’t. I really don’t like it. It’s anti-fun, and the awkward pause and wording that’s required to cast spells in case someone wants to counter it. There’s some equally awkward metagaming thats required if someone accidentally blurts out the name of the spell, and it plays really poorly with how most VTT software handles spells (most just spit them out in the chat for everyone to see). It is just so un-fun that I just ban the spell outright at my table and it makes everything much simpler.

Having said that, if I do play at a table where it isn’t banned, we usually go about it as I described above. The Reaction needed to identify the spell is an intentional design decision to prevent spellcasters from identifying every spell cast their way before deciding to counter them, and needing 2 spellcasters to work together to “cleanly” cast it is perfectly fine, in my opinion. Spellcasters are already bonkers in this game, there’s no reason to empower them further by letting them save-up their Counterspells until they’re absolutely critical.

It’s just important that every player is on the same page and doesn’t blurt-out their spell names whenever they cast a spell.

From the sessions ive been in and ran, the player/npc can make a arcana check to see if they know the spell. If they fail the check they can still use their reaction cast counterspell but they wont be able to tell what spell (an consequently what level to cast counterspell) is it a firebolt or meteor storm, who knows, want to risk casting it?

I’ve found Counterspell to be a pretty positive addition to my encounters. It encourages players to watch when enemies use their reactions, or try to bait a reaction. Rather than frustrating, the players have turned it into part of the game.

I’m also kinda meh about the whole arcana check thing. Casters generally have high DPS/versatility. I feel no need for them to be able to horde their spell slots further. When the party is facing an enemy caster, they announce that they’re casting and I tell them if the enemy caster counterspells.

I’m aware of people who decouple the “cast a spell” step from the “announce the spell” step, and implement rules to identify which spells are being cast ahead of time. I think it slows the game down too much for my liking, but that’s an option.

To make Counterspell more fair and give it a degree of interaction, I run counterspell as a contested check (d20 + spellcasting modifier + level of the spell/counterspell). It gives players an active role in the outcome, and it feels less cheap when the NPC negates their spell.

The easy way we fixed it when we still played dnd is that casters know what the spells are, even if it’s not in their spell list. You know if the enemy is casting firebolt or finger of death. Consider that if it’s possible to counterspell counterspell, which only has somatic casting components, then probably there is sufficient time to tell what each spell is. This fixes the bad feel of counterspelling because the spell is known by having everyone do it. Everyone knows what the spells are so you can choose to be strategic and get the most out of your counterspell.

If you want to justify it, you can imagine that all fire spells have some fire movement or incantation, add to that something related to size or expansion, plus a bead of fire forming, that’s a spicy fireball.

I personally only give NPC casters spells that fit them thematically or have in game reasons for having that spell. If counterspell is one of those, then I will use it to the full extent that an accurate portrayal of that NPC allows.

In your situation, if you are having the players roll to recognize spells, an easy solution, albeit one that slows things slightly, is having the NPC do this as well. I personally don’t bother with using a reaction to recognize the spell I just call for the check if they ask (within reason of course). The roll can be made behind the screen, but it should help things feel more fair for the players. I know most tables will use an arcana check with a DC of 10+spell level. Some tables will just have the caster roll against their intelligence score. Roll it however you feel is appropriate.

NGL (and I only have dm experience in 3 modules with a handful of players so take with a large grain of himalayan pink salt mmm) - I generally use mages and counterspell specifically to discourage lazy play. Blast the last 4 cultist encounters with fireballs? Oh man is there a counterspell ready caster in the next lol. Midlevel bosses and BBEGs aren’t dumb, they know magic exists, and they’ll retain talent. Letting that talent show illustrates the stakes, as well.

But I don’t use it to counter healing or well role-played hijinks.

This reminds me of an instance where my WM sorcerer counterspelled what I assessed to be a high-level spell, but then the bad guy counterspelled my counterspell and won. However, because we were playing with WM variant rules where the chance of it triggering are higher when higher level spells are used, a WM surge happened at that exact moment too and caused absolute mayhem.

A fun mechanic for spell casters to identify what is being used against them might be to have them automatically recognize any cantrips, or put it behind a low perception skill check, like DC 5 or 7, and increase the DC by 2 for every spell level or slot thereafter. In addition, the DC could be decreased by the highest level of spell known/available to that player.

This could be a special reaction only available to spellcasters that represents their instinctual familiarity with magic. I wouldn’t make it cost a reaction point, but I would limit its use to something like their proficiency bonus with a LR or SR recharge.

So, for instance, a BBEG casts disintegrate(lvl 5). A spell caster in the group uses this special reaction to recognize the spell, and knows one 4th level spell. DC would be DC = (5 + (2 * ESL)) - GSL, where ESL is “Evil Spell Level” and GSL is “Good Spell Level”. So being a lvl 5 spell, we would get:

DC = (5 + (2 * 5)) - 4

DC = (5 + 10) - 4

DC = 15 - 4

DC = 11

However, for that one dude in the party that took a multiclass level or two in warlock and would only know a lvl 1 spell, his DC would be 14.

Thats just my spitball answer though, but probably what I would start with.

The Pathfinder 2 way.

PF2 has the general Counteract mechanic:

Critical Success Counteract the target if its counteract level is no more than 3 levels higher than your effect’s counteract level.
Success Counteract the target if its counteract level is no more than 1 level higher than your effect’s counteract level.
Failure Counteract the target if its counteract level is lower than your effect’s counteract level.
Critical Failure You fail to counteract the target.

(critical success and failures being when you roll ten or more higher or lower than the DC)
You expend a spell slot containing the same spell and roll to counter the cast spell, with varying possible degrees of success.

You can later specialize it in various interesting ways with feats, for example one that lets you spend a prepared spell from merely the same school, or with a spell that is especially thematically appropriate, or redirect the countered spell, or just eat it.

It’s a whole mechanic that you can build around, it’s exciting when it happens, you feel like you’ve earned it. It’s a reward for clever play as opposed to a button that you push.

I’ve heard of people implementing a rule where a spell being counter spelled doesn’t make them lose their turn. The person who is counter spelled loses the spell slot and the ability to cast leveled spells that turn, but doesn’t lose their action. Seems kind of a fun side step to the RAW rules.