Question: Apparently, specific patterns with specific subjects can drive DIRegEx into stack overflow errors. How can I avoid these?

Answer: DIRegEx uses a recursive matching algorithm which can run out of stack space with extremely demanding patterns. The recursive algorithm was still chosen over an iterative implementation for performance reasons: Extensive testing revealed that DIRegEx runs multiple times faster with recursion than without.

Even though stack overflows are a real problem, they happens so rarely with common regex patterns and subjects that most DIRegEx users will never notice. In case you ever do, these steps can help to avoid them. Obviously, all 3 options combined yield best results:

1. Increase your application's stack size. Adding {$MAXSTACKSIZE $00400000}, preferably in your *.dpr file, should stop overflows even for extremely demanding patterns. {$MAXSTACKSIZE $00200000} enables most normally demanding patterns to run well and is a reasonable precaution setting.
2. Lower the TDIRegEx.MatchLimit and TDIRegEx.MatchLimitRecursion option properties. They cause matching to abort with PCRE_ERROR_MATCHLIMIT or PCRE_ERROR_RECURSIONLIMIT if the respective thresholds have been reached. The values can be set via TDIRegEx or the Extra field using the native API.
3. Optimize your regular expression to avoid nested subpatterns with unlimited repeats. The section "Atomic Grouping and Possessive Quantifiers" in the DIRegEx "Syntax Details" help page contains details and examples. The techniques described there will also help your patterns to run faster!

In Windows, the stack size is defined on a per-thread basis when the thread is created. This means that the calling thread's stack size applies to DIRegEx even if it is compiled into an external *.bpl or *.dll link library. It defaults to the application's stack size when called from the main thread.

If the calling application's stack size is too small and you are unable to change its stack size because you are writing a plugin or extension for a larger application, you might want to run DIRegEx in a newly created thread. The Windows CreateThread() function's dwStackSize parameter allows to change the initially committed stack space, which you should choose according to your needs.

Unfortunately, it is not possible to predict the required stack size in advance. It is highly dependent on the number of potential matches in the subject text. A pattern which works well with larger text can still fail with shorter ones if it encounters lots of failed matches which must be backtracked.

This page contains an interesting e-mail conversion about the internals of TDIRegExSearchStream and descendent classes (German).