egghelp/eggheads community

Learning regexpr

2005-07-13T23:50:12-04:00

attention should be paid to some subtle aspects of regexps, for example so-called "greedy matching"

by default, regexp characters '+' and '*' will match as much as possible ("greedy matching"), which might mean rather unexpected results, most likely in HTML parsing constructs like this:

Code:

% set str "foosome textbar"% regsub -all {(.*)} $str ""%

here, we need to strip the tags and their contents, leaving the text in-between; however, because of the greedy matching, we end up with all of the characters between the first opening tag and the last closing tag stripped, effectively leaving us with an empty string - definitely not what we wanted!

the solution is to add '?' specifier after the asterisk, to avert the greedy matching and force '+'/'*' to match as little as possible:

Code:

% set str "foosome textbar"% regsub -all {(.*?)} $str ""some text%

Statistics: Posted by demond — Wed Jul 13, 2005 11:50 pm

Learning regexpr

2005-07-08T04:15:32-04:00

Here are some quick and easy examples of substitutions. We can use 'regsub' (regular substitution) or 'string map'.

Nevertheless regular substitutions are slower, yet more advanced, complicated and effective than string map. However they both can be utilized to accomlish the same thing.

If you want to remove a character from a string:

Code:

#regsubregsub -all {a} $data "" data#Will remove all occurences of character "a" in the string $data.regsub -all {a} $data "b" data#Will replace all occurences of character "a" in the string by "b".Similarly,#string mapstring map {"a" ""} $data#Will remove all occurences of character "a" in the string $data.#string mapstring map {a b} $datastring map {"a" "b"} $data#Will replace all occurences of character "a" in the string by "b".

Mostly, regsub and string map are used in filters, to filter out certain parts, characters or words in texts.

Similar to regexp, regsub expressions can be used as

for matching each character individually.

Code:

#This will remove all occurences of a, b, c and d in the string $data.regsub -all {[abcd]} "sgfdszasbdgds" "" data#Note: We are using the -all switch here so it will return '5' as per the matches.You can also strip control codes (colors, bolds, underlines etc) from strings using regsub, string map filters as you might have seen in most posts on the forum. Here are some I found on the forum:#For removing colorsregsub -all {\003([0-9]{1,2}(,[0-9]{1,2})?)?} $str "" str#For removing control codesregsub -all {\017|\037|\002|\026|\006|\007} $str "" str#For removing control codesset str [string map {"\017" "" "\037" "" "\002" "" "\026" "" "\006" "" "\007" ""} $str]

You might have noticed, removing colors takes advanced regsub logics, which string map can't accomplish as above.

Note: It is best to indicate control codes in their ascii codes.

Then normally, string map and regsub can be used as filters to strip out certain special characters or to escape them with extra \'s.

Here are common examples I found to escape special characters by creating small filters.

Code:

#regsubproc filter {data} {regsub -all -- \\\\ $data \\\\\\\\ dataregsub -all -- \\\[ $data \\\\\[ dataregsub -all -- \\\] $data \\\\\] dataregsub -all -- \\\} $data \\\\\} dataregsub -all -- \\\{ $data \\\\\{ dataregsub -all -- \\\$ $data \\\\\$ dataregsub -all -- \\\" $data \\\\\" datareturn $data}#Taken from: http://www.peterre.com/characters.html#string mapproc filter {data} { set data [string map {\\ \\\\ [ \\\[ ] \\\] \{ \\\{ \} \\\} $ \\\$ \" \\\"} $data]}#Taken from: spambuster.tcl

A list of all special characters that can choke scripts if not used properly:

Code:

\, [, ], {, }, $, "

Note: regsub can be used in similar format as regexp:

Code:

regsub -all "\002|\003|\017|\026|\037" $text "" textregsub -all {\002|\003|\017|\026|\037} $text "" text

For example:

Code:

#To remove the total number of capital letters in a string:regsub -all {[A-Z]} $text "" counted#The total number of capital letters in $text will be placed in $counted and $text would have been stripped of the capital letters.Same goes similarly with numbers, [0-9] or both [a-z0-9].Also the -nocase switch is available in regsub for case sensitive matching or if you want to ignore cases while matching -- only for alphabets.

String map does not have an all switch hence it is difficult to count the total number of characters, so string map does have limitations.

For example:

Code:

regsub -all {a} $text "" counted#is similar as:set counted 0for {set count 0} {$count < [string length $text]} {incr text} { if {[string equal "a" [string index $text $count]]} {  incr counted  }}(Adv: As you can see regsub is more simpler, easier and has a smaller code)(Disadv: regsub is slower than string map)

Statistics: Posted by awyeah — Fri Jul 08, 2005 4:15 am

Learning regexpr

2005-07-08T04:40:38-04:00

I have been dealing with regexp's alot these days. Here are a few common examples which can help you in eggdrop scripts.

Suppose you want to count the number of A's (alphabet) in a string:

Code:

regexp -all {A} $string

Suppose you want to count the number of the's (word) in a string:

Code:

regexp -all {the} $string

Suppose you want to count more than one character:

Code:

regexp -all {[abcd]} $string#This code will count and add all the number of a's, b's, c's and d's found

Suppose you want the script to exeucte if any of these characters are not present:

Code:

regexp -all {[^abcd]} $string#This code will check and add all the number of a's, b's, c's and d's found. #The total number should be 0, for this statement to be true. (negative logic)

Sometimes while matching with regexp's you can use:

Code:

regexp "string" $stringregexp \[string\] $stringregexp {string} $string

I would you to use the curly brackets or the square brackets.

Counting special characters:

Code:

#To count the number of ['s or use:regexp -all \[\\\[\] $string#To count the number of {'s or use:regexp -all \[\\\\\] $string#To count the number of {'s or use:regexp -all \[{\] $stringNOTE: Generally you will only need to add 3 escape's infront of each [, ] or \ special characters. For others mostly you need not.

Note: regexp has a -nocase switch, which can be used for ignoring cases while doing matching.

Code:

regexp -nocase -all {abc} $string#andregexp -nocase -all {ABC} $string#will be considered the same then

Matching range of characters:

Code:

#To match a character in between the range of a, b, c, d, ..........z:regexp {[a-z]} $string > will give 1 for MATCH, 0 for NO-MATCH (lower case match)regexp -all {[a-z]} $string > will return total number of MATCHES (lower case match)regexp -nocase -all {[a-z]} $string > will return total number of MATCHES (case ignored)#To match a character in between the range of 0, 1, 2, 3, ..........9:regexp {[0-9]} $string > will give 1 for MATCH, 0 for NO-MATCHregexp -all {[0-9]} $string > will return total number of MATCHES#Note: The nocase switch for the [0-9] would be redundant.#To match a character in between the range of a, b, c,....z and 0, 1, 2...9:regexp {[a-z0-9]} $string > will give 1 for MATCH, 0 for NO-MATCHregexp -all {[a-z0-9]} $string > will return total number of MATCHES

Note also, it cis also necessary to use the proper matching format:

Code:

regexp {^string_here$} $string^ = Assert position at the start of the string$ = Assert position at the end of the string (or before the line break)$+ = Assert position at the end of the string (or before the line break)

The | operator is used as a LOGICAL "OR".

Code:

regexp {abc|efg|hij} $string#This will try to match "abc" or "efg" or "hij" if none is found, returns 0, if anyone is found returns 1.

The ^ operator used in a

before the first element is used as a LOGICAL "NOT".

Code:

regexp {^[^abcd]$} $string#If a, b, c and d are not present return 1, if anyone of them is, return 0.

Note if you want to find the matched patterns in the string of regexp you can use the -inline switch. But you should use it with -all in most cases.

Other examples:
If you want to match certain patterns:

Code:

regexp {^[a-z]{3,}[0-9]{2,}$} $string#This will match only if 3 or more characters are present in the range [a-z] and 2 or more characters in the range [0-9] of the string.#Example:abgfg452 > will matchas456342 > will not matchabc12 > will match

Other examples:

Code:

regexp {^[a-z]{3,5}[0-9]{2,8}$} $string#This will match only if 3, 4 or 5 characters are present in the range [a-z] and 2, 3, 4, 5, 6, 7 or 8 characters in the range [0-9] of the string.#Examples:adfsd3463 > will matchadsfsdgfs325 > will not matchwer436234 > will matchgdtweer436322512 > will not match

Other examples:

Code:

regexp {^[a-z]{4}[0-9]{3}$} $string#This will match only if 4 characters are present in the range [a-z] and 3 characters in the range [0-9] of the string.#Examples:wrew364 > will matchwe436 > will not matchwhg6743 > wil not matchwga63 > will not matchse65 > will not match

Other matchings:

Code:

regexp {https?} $stringThis will return 1 if "http" or "https" is present in the string, else return 0.

Here are some advanced examples:

Code:

regexp {([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})} $string#IP Address -- Matches 0.0.0.0 through 999.999.999.999regexp {(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)} $string#IP Address -- Matches 0.0.0.0 through 255.255.255.255regexp {(https?|ftp|file)://[-A-Z0-9+&@#/%?=~_|!:,.;]*[-A-Z0-9+&@#/%=~_|]}#Matching a urlregexp {[0-9]{5}(?:-[0-9]{4})?}#US Zipcoderegexp {[A-Z0-9._%-]+@[A-Z0-9._%-]+\.[A-Z]{2,4}}#Email addressregexp {(0[1-9]|[12][0-9]|3[01])[-/.](0[1-9]|1[012])[- /.](19|20)[0-9]{2}}#Date in formats: dd-mm-yy, dd.mm.yy, dd/mm/yyregexp {^(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6011[0-9]{14}|3(?:0[0-5]|[68][0-9])[0-9]{11}|3[47][0-9]{13})$}#Matching all major credit cards

More of these examples can be found by DOWNLOADING and installing
the SOFTWARE "REGEXYBUDDY".

Download link: http://www.regexbuddy.com/download.html

1) After downloading, install the trial version of the software.
2) After installation, run the software and click on the Library tabs.
3) In the long search list on the right panel, highlight any matching pattern
of your choice and in the left of the software, the window you would be able to see the regular expression match pattern.
4) This is a good software to learn regexp from.

Statistics: Posted by awyeah — Thu Jul 07, 2005 8:19 am

Learning regexpr

2004-08-28T15:53:11-04:00

Here is a good website to learn regular expressions (regexp) from, it includes tutorials and examples:
http://www.regular-expressions.info/

Here are some softwares only made for the purpose of testing/using regular expressions with their respective examples:
http://www.regular-expressions.info/tools.html

Statistics: Posted by awyeah — Sat Aug 28, 2004 3:53 pm

Learning regexpr

2004-08-28T13:01:56-04:00

The Mastering Regular Expressions no longer seems to be working, dose anyone have a good link to it?

Statistics: Posted by caesar — Sat Aug 28, 2004 1:01 pm

Learning regexpr

2003-07-11T13:56:25-04:00

well guys , I was about to ask how to learn regexp, but I found this post and its realy handy

nice idea Ppslim.

Statistics: Posted by Sir_Fz — Fri Jul 11, 2003 1:56 pm

Learning regexpr

2003-05-11T19:59:36-04:00

Thanks to a kind sugestion from pgpkeys (#egghelp@efnet), there is also a PDF document for you to download.

I ahvn't had a look at it myself yet, but I sure it may be of more use to sombody here.

The document is called Mastering Regular Expressions

Statistics: Posted by ppslim — Sun May 11, 2003 7:59 pm

A note to future regexp lunatics :)

2003-05-06T15:10:54-04:00

People often stick with regexp (once they learn how to use it) even when there's better (faster) methods avaliable to deal with a certain problem. 'scan' and 'string map' is the most common commands ignored by regexp fanatics.

Example using scan to chop up eggdrop's $botname:

Code:

scan $botname %\[^!\]!%\[^@\]@%s nick user host

which is ~13.5 times faster than

Code:

regexp {(.*)!(.*)@(.*)} $botname match nick user host

in my tclsh (8.3)

/rant

Statistics: Posted by user — Tue May 06, 2003 3:10 pm

Learning regexpr

2003-05-06T05:57:52-04:00

While again, not directly Tcl related, this subject is something worthwhile talking about. It will help in a hell of a lot fo ways, and make no less than 3 commands, far more fun, with 2 requiring them full stop.

This entry, actually comes from our very own "stdragon". Posted back in Sep 02 (available here), it is more worthwhile in a Tcl FAQ, than at the bottom of a failed search, or long and boring trail through the forums history.

Originally inspired by the very question "Tutorials please", "How do I use them", "What can I use them for", this was the reply, of which, is what makes this forum as a whole, such a powerful learning and help tool.

Regexps?

1. I learned to use them by experimentation in tclsh. If you're in a hurry and don't want to 'learn as you go' then sit down for an hour and play with every special character you see in re_sytnax until you know what it does.

2. You can use them for complex string matching and substitution. That's about it, but that encompasses a lot, since most things in life can be represented as a string. Usually people use it for syntax checking (e.g. "Is the input a valid email address?" or "Is this sentence 'bad' as defined by this list of user rules?"). Other common uses are getting rid of color/control codes, extracting parts of a line into variables, and performing substitutions into kick messages,etc (like the person's nick replaces %nick in the kick message).

3. Regexp returns 0 for no match and 1 for a match. Regsub returns the number of substitutions, e.g. 0 for no match and non-zero for a match.

Looking through re_syntax is good, but it's better to find some example scripts. Depending on what you want to do, most regular expressions are very simple. There are only a few special chars you have to escape (like (), |, ., *, {}, +, erm, maybe some more..).

Here's a quick mini tutorial:

( ) is used for match reporting.. regexp lets you specify 'match variables' that get filled in with what exactly matched. The matches within parenthesis are what get reported. Also it's just like math, they allow you to group other operations. An example of match reporting would be: regexp {(.*)!(.*)@(.*)} $from match nick user host

| means "or". It lets your regexp match more than one thing. For instance, "hello there|hi there" would match either "hello there" or "hi there". Using ( ) for grouping, you could say "(hello|hi) there" and it would be the same thing.

. means "any single character". So the regex "..." would match any 3 letters (including spaces). To match an actual period, escape it with a backslash \.

* means "any number of the previous thing, including 0." So if you have "baa*", what is the previous thing? "a". So that would match "ba" (0), "baa", "baaaaaaaaa", etc. However, using grouping ( ) you can match multiple things: "(baa)*" would match "baabaabaa". If you notice, .* will match anything, because . means "any char" and * means "any number". So any easy way to translate between dos-style wildcards and regexps is to replace ?'s with single dots, and * with ".*"

+ is exactly the same as *, but it requires at least 1. So "baa+" would not match "ba" anymore.

{ } is a range operator. It's just like * and + but it lets you specify how many repeats are acceptable. For instance, "ba{2,10}" would match from "baa" (2 a's) to "baaaaaaaaaa" (10 a's).

There are a few more ones that are either less useful or way more complicated. For instance the section on negative lookaheads meant very little to me until the other day. It's a silly name, what it really is is a "and not" operator. For instance, if you want to match something that "contains an a and no b" you could use a negative lookahead. (Yes for that example you can use the [ ] operator to match for ^b, but [ ] can't contain a full regular expression, whereas negative lookaheads can.)

So those are the basics. Anything more advanced would require exponentially more amounts of text I think. If you have specific questions feel free to ask.

As the man said, post any further questions if you feel fit.

Statistics: Posted by ppslim — Tue May 06, 2003 5:57 am