LinkedIn: RegEx Challenge 5 - Twisted Nested Brackets

Challenge:

Imagine a programming language that only consists of brackets. For this challenge you'll be working with sequences of <>, {}, [], and (). The rules are simple but tricky:

🔸 <> can only contain {} or []
🔸 {} can only contain [] or ()
🔸 [] can only contain () or <>
🔸 () can only contain <> or {}

Brackets of the same type cannot appear directly inside each other. Sequences can be nested arbitrarily deep. Your task: identify which sequences are valid. For reference some valid and non-valid ones:

String                         | Match
-------------------------------|---------
<{}[]>                         | ✅
<[{()}]>                       | ✅
{[]()}                         | ✅
[(){}]                         | ✅
(<>{})                         | ✅
<{[()<>]}>                     | ✅
{[(<>)()]}                     | ✅
[({}<>)()]                      | ✅
<<>>                            | ❌
{{}}                            | ❌
[[]]                            | ❌
(())                            | ❌
<()>                            | ❌
{[()<>]<[]>}                    | ❌
[(<>{})<[]>]                    | ❌
(<{[()]}> <>)                   | ❌
<{[{()}]>}                      | ❌
<{{[()]}}>                      | ❌

Pattern:

(?(DEFINE)(?<A><((?&C)|(?&S))*>)(?<C>\{((?&S)|(?&P))*\})(?<S>\[((?&P)|(?&A))*\])(?<P>\(((?&A)|(?&C))*\)))^((?&A)|(?&C)|(?&S)|(?&P))$

Or, written a bit more clearly:

(?(DEFINE)
	(?<A><((?&C)|(?&S))*>)
	(?<C>\{((?&S)|(?&P))*\})
	(?<S>\[((?&P)|(?&A))*\])
	(?<P>\(((?&A)|(?&C))*\))
)
^((?&A)|(?&C)|(?&S)|(?&P))$

(?(DEFINE)) is one of the most underrated features in PCRE. It lets you declare named subpatterns without matching anything right away — effectively turning regex into a modular grammar system.

When you wrap your definitions in (?(DEFINE)), PCRE pre-registers all named groups before matching begins. This means each subpattern can safely refer to the others — even if they appear later in the pattern. In other words, you can finally build mutually recursive grammars directly in regex.

In the this challenge, (?(DEFINE)) makes it possible to describe four different bracket types (<>, {}, [], ()) that can contain each other according to strict nesting rules. Without DEFINE, this kind of cross-referencing recursion simply wouldn’t be possible in a single regex.

It’s regex, but with the structure of a real grammar — a small leap from pattern matching to parsing.

How the Pattern Works:

The pattern defines four named subpatterns inside (?(DEFINE)):

• A matches <…> brackets that may contain either {} or [] pairs.

• C matches {…} that may contain either [] or () pairs.

• S matches […] that may contain either () or <> pairs.

• P matches (…) that may contain either <> or {} pairs.

Each rule uses recursion like (?&A) or (?&C) to allow nested structures, and the * quantifier means a pair can contain multiple valid children.

^((?&A)|(?&C)|(?&S)|(?&P))$ Starts the actual match — it tells the regex engine to match exactly one complete top-level structure of any of the four bracket types, fully respecting the recursive rules defined above.

Awesome right!?