EditPageHistoryDiffInfoLikePages

Annotated edit history of perlretut(1) version 2, including all changes. View license author blame.

Rev	Author	#	Line
1	perry	1	PERLRETUT
		2	!!!PERLRETUT
		3	NAME
		4	DESCRIPTION
		5	Part 1: The basics
		6	Part 2: Power tools
		7	BUGS
		8	SEE ALSO
		9	AUTHOR AND COPYRIGHT
		10	----
		11	!!NAME
		12
		13
		14	perlretut - Perl regular expressions tutorial
		15	!!DESCRIPTION
		16
		17
		18	This page provides a basic tutorial on understanding,
		19	creating and using regular expressions in Perl. It serves as
		20	a complement to the reference page on regular expressions
		21	perlre. Regular expressions are an integral part of the
		22	m//, s///, qr// and
		23	split operators and so this tutorial also overlaps
		24	with ``Regexp Quote-Like Operators'' in perlop and ``split''
		25	in perlfunc.
		26
		27
		28	Perl is widely renowned for excellence in text processing,
		29	and regular expressions are one of the big factors behind
		30	this fame. Perl regular expressions display an efficiency
		31	and flexibility unknown in most other computer languages.
		32	Mastering even the basics of regular expressions will allow
		33	you to manipulate text with surprising ease.
		34
		35
		36	What is a regular expression? A regular expression is simply
		37	a string that describes a pattern. Patterns are in common
		38	use these days; examples are the patterns typed into a
		39	search engine to find web pages and the patterns used to
		40	list files in a directory, e.g., ls *.txt or
		41	dir *.*. In Perl, the patterns described by regular
		42	expressions are used to search strings, extract desired
		43	parts of strings, and to do search and replace
		44	operations.
		45
		46
		47	Regular expressions have the undeserved reputation of being
		48	abstract and difficult to understand. Regular expressions
		49	are constructed using simple concepts like conditionals and
		50	loops and are no more difficult to understand than the
		51	corresponding if conditionals and while
		52	loops in the Perl language itself. In fact, the main
		53	challenge in learning regular expressions is just getting
		54	used to the terse notation used to express these
		55	concepts.
		56
		57
		58	This tutorial flattens the learning curve by discussing
		59	regular expression concepts, along with their notation, one
		60	at a time and with many examples. The first part of the
		61	tutorial will progress from the simplest word searches to
		62	the basic regular expression concepts. If you master the
		63	first part, you will have all the tools needed to solve
		64	about 98% of your needs. The second part of the tutorial is
		65	for those comfortable with the basics and hungry for more
		66	power tools. It discusses the more advanced regular
		67	expression operators and introduces the latest cutting edge
		68	innovations in 5.6.0.
		69
		70
		71	A note: to save time, 'regular expression' is often
		72	abbreviated as regexp or regex. Regexp is a more natural
		73	abbreviation than regex, but is harder to pronounce. The
		74	Perl pod documentation is evenly split on regexp vs regex;
		75	in Perl, there is more than one way to abbreviate it. We'll
		76	use regexp in this tutorial.
		77	!!Part 1: The basics
		78
		79
		80	__Simple word matching__
		81
		82
		83	The simplest regexp is simply a word, or more generally, a
		84	string of characters. A regexp consisting of a word matches
		85	any string that contains that word:
		86
		87
		88
		89	What is this perl statement all about? is a simple double quoted string. World is the regular expression and the // enclosing /World/ tells perl to search a string for a match. The operator =~ associates the string with the regexp match and produces a true value if the regexp matched, or false if the regexp did not match. In our case, World matches the second word in , so the expression is true. Expressions like this are useful in conditionals:
		90
		91
		92	if (
		93	There are useful variations on this theme. The sense of the match can be reversed by using !~ operator:
		94
		95
		96	if (
		97	The literal string in the regexp can be replaced by a variable:
		98
		99
		100	$greeting =
		101	If you're matching against the special default variable $_, the $_ =~ part can be omitted:
		102
		103
		104	$_ =
		105	And finally, the // default delimiters for a match can be changed to arbitrary delimiters by putting an 'm' out front:
		106
		107
		108
		109	/World/, m!World!, and m{World} all represent the same thing. When, e.g., is used as a delimiter, the forward slash '/' becomes an ordinary character and can be used in a regexp without trouble.
		110
		111
		112	Let's consider how different regexps would match
		113	:
		114
		115
		116
		117	The first regexp world doesn't match because regexps are case-sensitive. The second regexp matches because the substring 'o W' occurs in the string . The space character ' ' is treated like any other character in a regexp and is needed to match in this case. The lack of a space character is the reason the third regexp 'oW' doesn't match. The fourth regexp 'World ' doesn't match because there is a space at the end of the regexp, but not at the end of the string. The lesson here is that regexps must match a part of the string ''exactly'' in order for the statement to be true.
		118
		119
		120	If a regexp matches in more than one place in the string,
		121	perl will always match at the earliest possible point in the
		122	string:
		123
		124
		125
		126	With respect to character matching, there are a few more points you need to know about. First of all, not all characters can be used 'as is' in a match. Some characters, called __metacharacters__, are reserved for use in regexp notation. The metacharacters are
		127
		128
		129	{}[[]()^$.*+?\
		130	The significance of each of these will be explained in the rest of the tutorial, but for now, it is important only to know that a metacharacter can be matched by putting a backslash before it:
		131
		132
		133
		134	In the last regexp, the forward slash '/' is also backslashed, because it is used to delimit the regexp. This can lead to LTS (leaning toothpick syndrome), however, and it is often more readable to change delimiters.
		135
		136
		137	The backslash character '' is a metacharacter
		138	itself and needs to be backslashed:
		139
		140
		141	'C:WIN32' =~ /C:\WIN/; # matches
		142	In addition to the metacharacters, there are some ASCII characters which don't have printable character equivalents and are instead represented by __escape sequences__. Common examples are t for a tab, n for a newline, r for a carriage return and a for a bell. If your string is better thought of as a sequence of arbitrary bytes, the octal escape sequence, e.g., 033, or hexadecimal escape sequence, e.g., x1B may be a more natural representation for your bytes. Here are some examples of escapes:
		143
		144
		145
		146	If you've been around Perl a while, all this talk of escape sequences may seem familiar. Similar escape sequences are used in double-quoted strings and in fact the regexps in Perl are mostly treated as double-quoted strings. This means that variables can be used in regexps as well. Just like double-quoted strings, the values of the variables in the regexp will be substituted in before the regexp is evaluated for matching purposes. So we have:
		147
		148
		149	$foo = 'house';
		150	'housecat' =~ /$foo/; # matches
		151	'cathouse' =~ /cat$foo/; # matches
		152	'housecat' =~ /${foo}cat/; # matches
		153	So far, so good. With the knowledge above you can already perform searches with just about any literal string regexp you can dream up. Here is a ''very simple'' emulation of the Unix grep program:
		154
		155
		156	% cat
		157	% chmod +x simple_grep
		158	% simple_grep abba /usr/dict/words
		159	Babbage
		160	cabbage
		161	cabbages
		162	sabbath
		163	Sabbathize
		164	Sabbathizes
		165	sabbatical
		166	scabbard
		167	scabbards
		168	This program is easy to understand. #!/usr/bin/perl is the standard way to invoke a perl program from the shell. $regexp = shift; saves the first command line argument as the regexp to be used, leaving the rest of the command line arguments to be treated as files. while ( loops over all the lines in all the files. For each line, print if /$regexp/; prints the line if the regexp matches the line. In this line, both print and /$regexp/ use the default variable $_ implicitly.
		169
		170
		171	With all of the regexps above, if the regexp matched
		172	anywhere in the string, it was considered a match.
		173	Sometimes, however, we'd like to specify ''where'' in the
		174	string the regexp should try to match. To do this, we would
		175	use the __anchor__ metacharacters ^ and
		176	$. The anchor ^ means match at the
		177	beginning of the string and the anchor $ means
		178	match at the end of the string, or before a newline at the
		179	end of the string. Here is how they are used:
		180
		181
		182
		183	The second regexp doesn't match because ^ constrains keeper to match only at the beginning of the string, but has keeper starting in the middle. The third regexp does match, since the $ constrains keeper to match only at the end of the string.
		184
		185
		186	When both ^ and $ are used at the same
		187	time, the regexp has to match both the beginning and the end
		188	of the string, i.e., the regexp matches the whole string.
		189	Consider
		190
		191
		192
		193	The first regexp doesn't match because the string has more to it than keep. Since the second regexp is exactly the string, it matches. Using both ^ and $ in a regexp forces the complete string to match, so it gives you complete control over which strings match and which don't. Suppose you are looking for a fellow named bert, off in a string by himself:
		194
		195
		196
		197
		198
		199	Of course, in the case of a literal string, one could just as easily use the string equivalence $string eq 'bert' and it would be more efficient. The ^...$ regexp really becomes useful when we add in the more powerful regexp tools below.
		200
		201
		202	__Using character classes__
		203
		204
		205	Although one can already do quite a lot with the literal
		206	string regexps above, we've only scratched the surface of
		207	regular expression technology. In this and subsequent
		208	sections we will introduce regexp concepts (and associated
		209	metacharacter notations) that will allow a regexp to not
		210	just represent a single character sequence, but a ''whole
		211	class'' of them.
		212
		213
		214	One such concept is that of a __character class__. A
		215	character class allows a set of possible characters, rather
		216	than just a single character, to match at a particular point
		217	in a regexp. Character classes are denoted by brackets
		218	[[...], with the set of characters to be possibly
		219	matched inside. Here are some examples:
		220
		221
		222	/cat/; # matches 'cat'
		223	/[[bcr]at/; # matches 'bat, 'cat', or 'rat'
		224	/item[[0123456789]/; # matches 'item0' or ... or 'item9'
		225	In the last statement, even though 'c' is the first character in the class, 'a' matches because the first character position in the string is the earliest point at which the regexp can match.
		226
		227
		228	/[[yY][[eE][[sS]/; # match 'yes' in a case-insensitive way
		229	# 'yes', 'Yes', 'YES', etc.
		230	This regexp displays a common task: perform a a case-insensitive match. Perl provides away of avoiding all those brackets by simply appending an 'i' to the end of the match. Then /[[yY][[eE][[sS]/; can be rewritten as /yes/i;. The 'i' stands for case-insensitive and is an example of a __modifier__ of the matching operation. We will meet other modifiers later in the tutorial.
		231
		232
		233	We saw in the section above that there were ordinary
		234	characters, which represented themselves, and special
		235	characters, which needed a backslash \ to represent
		236	themselves. The same is true in a character class, but the
		237	sets of ordinary and special characters inside a character
		238	class are different than those outside a character class.
		239	The special characters for a character class are
		240	-]^$. ] is special because it denotes the
		241	end of a character class. $ is special because it
		242	denotes a scalar variable. \ is special because it
		243	is used in escape sequences, just like above. Here is how
		244	the special characters ]$\ are
		245	handled:
		246
		247
		248	/[[]c]def/; # matches ']def' or 'cdef'
		249	$x = 'bcr';
		250	/[[$x]at/; # matches 'bat', 'cat', or 'rat'
		251	/[[$x]at/; # matches '$at' or 'xat'
		252	/[[\$x]at/; # matches 'at', 'bat, 'cat', or 'rat'
		253	The last two are a little tricky. in [[$x], the backslash protects the dollar sign, so the character class has two members $ and x. In [[\$x], the backslash is protected, so $x is treated as a variable and substituted in double quote fashion.
		254
		255
		256	The special character '-' acts as a range operator
		257	within character classes, so that a contiguous set of
		258	characters can be written as a range. With ranges, the
		259	unwieldy [[0123456789] and [[abc...xyz]
		260	become the svelte [[0-9] and [[a-z]. Some
		261	examples are
		262
		263
		264	/item[[0-9]/; # matches 'item0' or ... or 'item9'
		265	/[[0-9bx-z]aa/; # matches '0aa', ..., '9aa',
		266	# 'baa', 'xaa', 'yaa', or 'zaa'
		267	/[[0-9a-fA-F]/; # matches a hexadecimal digit
		268	/[[0-9a-zA-Z_]/; # matches a
		269	If '-' is the first or last character in a character class, it is treated as an ordinary character; [[-ab], [[ab-] and [[a-b] are all equivalent.
		270
		271
		272	The special character ^ in the first position of a
		273	character class denotes a __negated character class__,
		274	which matches any character but those in the brackets. Both
		275	[[...] and [[^...] must match a character,
		276	or the match fails. Then
		277
		278
		279	/[[^a]at/; # doesn't match 'aat' or 'at', but matches
		280	# all other 'bat', 'cat, '0at', '%at', etc.
		281	/[[^0-9]/; # matches a non-numeric character
		282	/[[a^]at/; # matches 'aat' or '^at'; here '^' is ordinary
		283	Now, even [[0-9] can be a bother the write multiple times, so in the interest of saving keystrokes and making regexps more readable, Perl has several abbreviations for common character classes:
		284
		285
		286	d is a digit and represents [[0-9]
		287
		288
		289	s is a whitespace character and represents [[\
		290	trnf]
		291
		292
		293	w is a word character (alphanumeric or _) and represents
		294	[[0-9a-zA-Z_]
		295
		296
		297	D is a negated d; it represents any character but a digit
		298	[[^0-9]
		299
		300
		301	S is a negated s; it represents any non-whitespace character
		302	[[^s]
		303
		304
		305	W is a negated w; it represents any non-word character
		306	[[^w]
		307
		308
		309	The period '.' matches any character but ``n''
		310
		311
		312	The dswDSW abbreviations can be used both inside
		313	and outside of character classes. Here are some in
		314	use:
		315
		316
		317	/dd:dd:dd/; # matches a hh:mm:ss time format
		318	/[[ds]/; # matches any digit or whitespace character
		319	/wWw/; # matches a word char, followed by a
		320	# non-word char, followed by a word char
		321	/..rt/; # matches any two chars, followed by 'rt'
		322	/end./; # matches 'end.'
		323	/end[[.]/; # same thing, matches 'end.'
2	perry	324	Because a period is a metacharacter, it needs to be escaped to match as an ordinary period. Because, for example, d and w are sets of characters, it is incorrect to think of [[^dw] as [[DW]; in fact [[^dw] is the same as [[^w], which is the same as [[W]. Think !DeMorgan's laws.
1	perry	325
		326
		327	An anchor useful in basic regexps is the __word anchor__
		328	b. This matches a boundary between a word character
		329	and a non-word character wW or
		330	Ww:
		331
		332
		333	$x =
		334	Note in the last example, the end of the string is considered a word boundary.
		335
		336
		337	You might wonder why '.' matches everything but
		338	- why not every character? The reason
		339	is that often one is matching against lines and would like
		340	to ignore the newline characters. For instance, while the
		341	string represents one line, we would
		342	like to think of as empty. Then
		343
		344
		345
		346
		347	This behavior is convenient, because we usually want to ignore newlines when we count and match characters in a line. Sometimes, however, we want to keep track of newlines. We might even want ^ and $ to anchor at the beginning and end of lines within the string, rather than just the beginning and end of the string. Perl allows us to choose between ignoring and paying attention to newlines by using the //s and //m modifiers. //s and //m stand for single line and multi-line and they determine whether a string is to be treated as one continuous string, or as a set of lines. The two modifiers affect two aspects of how the regexp is interpreted: 1) how the '.' character class is defined, and 2) where the anchors ^ and $ are able to match. Here are the four possible combinations:
		348
		349
		350	no modifiers (//): Default behavior. '.' matches
		351	any character except . ^
		352	matches only at the beginning of the string and $
		353	matches only at the end or before a newline at the
		354	end.
		355
		356
		357	s modifier (//s): Treat string as a single long line.
		358	'.' matches any character, even
		359	. ^ matches only at the
		360	beginning of the string and $ matches only at the
		361	end or before a newline at the end.
		362
		363
		364	m modifier (//m): Treat string as a set of multiple lines.
		365	'.' matches any character except
		366	. ^ and $ are able
		367	to match at the start or end of ''any'' line within the
		368	string.
		369
		370
		371	both s and m modifiers (//sm): Treat string as a single long
		372	line, but detect multiple lines. '.' matches any
		373	character, even . ^ and
		374	$, however, are able to match at the start or end
		375	of ''any'' line within the string.
		376
		377
		378	Here are examples of //s and //m in
		379	action:
		380
		381
		382	$x =
		383	$x =~ /^Who/; # doesn't match,
		384	$x =~ /girl.Who/; # doesn't match,
		385	Most of the time, the default behavior is what is want, but //s and //m are occasionally very useful. If //m is being used, the start of the string can still be matched with A and the end of string can still be matched with the anchors Z (matches both the end and the newline before, like $), and z (matches only the end):
		386
		387
		388	$x =~ /^Who/m; # matches,
		389	$x =~ /girl$/m; # matches,
		390	$x =~ /PerlZ/m; # matches,
		391	We now know how to create choices among classes of characters in a regexp. What about choices among words or character strings? Such choices are described in the next section.
		392
		393
		394	__Matching this or that__
		395
		396
		397	Sometimes we would like to our regexp to be able to match
		398	different possible words or character strings. This is
		399	accomplished by using the __alternation__ metacharacter .
		400	To match dog or cat, we form the regexp
		401	dogcat. As before, perl will try to match the
		402	regexp at the earliest possible point in the string. At each
		403	character position, perl will first try to match the first
		404	alternative, dog. If dog doesn't match,
		405	perl will then try the next alternative, cat. If
		406	cat doesn't match either, then the match fails and
		407	perl moves to the next position in the string. Some
		408	examples:
		409
		410
		411
		412	Even though dog is the first alternative in the second regexp, cat is able to match earlier in the string.
		413
		414
		415
		416	Here, all the alternatives match at the first string position, so the first alternative is the one that matches. If some of the alternatives are truncations of the others, put the longest ones first to give them a chance to match.
		417
		418
		419
		420	The last example points out that character classes are like alternations of characters. At a given character position, the first alternative that allows the regexp match to succeed wil be the one that matches.
		421
		422
		423	__Grouping things and hierarchical
		424	matching__
		425
		426
		427	Alternation allows a regexp to choose among alternatives,
		428	but by itself it unsatisfying. The reason is that each
		429	alternative is a whole regexp, but sometime we want
		430	alternatives for just part of a regexp. For instance,
		431	suppose we want to search for housecats or housekeepers. The
		432	regexp housecathousekeeper fits the bill, but is
		433	inefficient because we had to type house twice. It
		434	would be nice to have parts of the regexp be constant, like
		435	house, and and some parts have alternatives, like
		436	catkeeper.
		437
		438
		439	The __grouping__ metacharacters () solve this
		440	problem. Grouping allows parts of a regexp to be treated as
		441	a single unit. Parts of a regexp are grouped by enclosing
		442	them in parentheses. Thus we could solve the
		443	housecathousekeeper by forming the regexp as
		444	house(catkeeper). The regexp
		445	house(catkeeper) means match house
		446	followed by either cat or keeper. Some
		447	more examples are
		448
		449
		450	/(ab)b/; # matches 'ab' or 'bb'
		451	/(acb)b/; # matches 'acb' or 'bb'
		452	/(^ab)c/; # matches 'ac' at start of string or 'bc' anywhere
		453	/(a[[bc])d/; # matches 'ad', 'bd', or 'cd'
		454	/house(cat)/; # matches either 'housecat' or 'house'
		455	/house(cat(s))/; # matches either 'housecats' or 'housecat' or
		456	# 'house'. Note groups can be nested.
		457	/(1920)dd/; # match years 19xx, 20xx, or the Y2K problem, xx
		458	Alternations behave the same way in groups as out of them: at a given string position, the leftmost alternative that allows the regexp to match is taken. So in the last example at tth first string position, matches the second alternative, but there is nothing left over to match the next two digits dd. So perl moves on to the next alternative, which is the null alternative and that works, since is two digits.
		459
		460
		461	The process of trying one alternative, seeing if it matches,
		462	and moving on to the next alternative if it doesn't, is
		463	called __backtracking__. The term 'backtracking' comes
		464	from the idea that matching a regexp is like a walk in the
		465	woods. Successfully matching a regexp is like arriving at a
		466	destination. There are many possible trailheads, one for
		467	each string position, and each one is tried in order, left
		468	to right. From each trailhead there may be many paths, some
		469	of which get you there, and some which are dead ends. When
		470	you walk along a trail and hit a dead end, you have to
		471	backtrack along the trail to an earlier point to try another
		472	trail. If you hit your destination, you stop immediately and
		473	forget about trying all the other trails. You are
		474	persistent, and only if you have tried all the trails from
		475	all the trailheads and not arrived at your destination, do
		476	you declare failure. To be concrete, here is a step-by-step
		477	analysis of what perl does when it tries to match the
		478	regexp
		479
		480
		481
		482
		483
		484	Start with the first letter in the string 'a'.
		485
		486
		487	1
		488
		489
		490	Try the first alternative in the first group
		491	'abd'.
		492
		493
		494	2
		495
		496
		497	Match 'a' followed by 'b'. So far so good.
		498
		499
		500	3
		501
		502
		503	'd' in the regexp doesn't match 'c' in the string - a dead
		504	end. So backtrack two characters and pick the second
		505	alternative in the first group 'abc'.
		506
		507
		508	4
		509
		510
		511	Match 'a' followed by 'b' followed by 'c'. We are on a roll
		512	and have satisfied the first group. Set $1 to
		513	'abc'.
		514
		515
		516	5
		517
		518
		519	Move on to the second group and pick the first alternative
		520	'df'.
		521
		522
		523	6
		524
		525
		526	Match the 'd'.
		527
		528
		529	7
		530
		531
		532	'f' in the regexp doesn't match 'e' in the string, so a dead
		533	end. Backtrack one character and pick the second alternative
		534	in the second group 'd'.
		535
		536
		537	8
		538
		539
		540	'd' matches. The second grouping is satisfied, so set
		541	$2 to 'd'.
		542
		543
		544	9
		545
		546
		547	We are at the end of the regexp, so we are done! We have
		548	matched 'abcd' out of the string ``abcde''.
		549
		550
		551	There are a couple of things to note about this analysis.
		552	First, the third alternative in the second group 'de' also
		553	allows a match, but we stopped before we got to it - at a
		554	given character position, leftmost wins. Second, we were
		555	able to get a match at the first character position of the
		556	string 'a'. If there were no matches at the first position,
		557	perl would move to the second character position 'b' and
		558	attempt the match all over again. Only when all possible
		559	paths at all possible character positions have been
		560	exhausted does perl give give up and declare $string =~
		561	/(abdabc)(dfdde)/; to be false.
		562
		563
		564	Even with all this work, regexp matching happens remarkably
		565	fast. To speed things up, during compilation stage, perl
		566	compiles the regexp into a compact sequence of opcodes that
		567	can often fit inside a processor cache. When the code is
		568	executed, these opcodes can then run at full throttle and
		569	search very quickly.
		570
		571
		572	__Extracting matches__
		573
		574
		575	The grouping metacharacters () also serve another
		576	completely different function: they allow the extraction of
		577	the parts of a string that matched. This is very useful to
		578	find out what matched and for text processing in general.
		579	For each grouping, the part that matched inside goes into
		580	the special variables $1, $2, etc. They
		581	can be used just as ordinary variables:
		582
		583
		584	# extract hours, minutes, seconds
		585	$time =~ /(dd):(dd):(dd)/; # match hh:mm:ss format
		586	$hours = $1;
		587	$minutes = $2;
		588	$seconds = $3;
		589	Now, we know that in scalar context, $time =~ /(dd):(dd):(dd)/ returns a true or false value. In list context, however, it returns the list of matched values ($1,$2,$3). So we could write the code more compactly as
		590
		591
		592	# extract hours, minutes, seconds
		593	($hours, $minutes, $second) = ($time =~ /(dd):(dd):(dd)/);
		594	If the groupings in a regexp are nested, $1 gets the group with the leftmost opening parenthesis, $2 the next opening parenthesis, etc. For example, here is a complex regexp and the matching variables indicated below it:
		595
		596
		597	/(ab(cdef)((gi)j))/;
		598	1 2 34
		599	so that if the regexp matched, e.g., $2 would contain 'cd' or 'ef'. For convenience, perl sets $+ to the highest numbered $1, $2, ... that got assigned.
		600
		601
		602	Closely associated with the matching variables $1,
		603	$2, ... are the __backreferences__ 1,
		604	2, ... . Backreferences are simply matching
		605	variables that can be used ''inside'' a regexp. This is a
		606	really nice feature - what matches later in a regexp can
		607	depend on what matched earlier in the regexp. Suppose we
		608	wanted to look for doubled words in text, like 'the the'.
		609	The following regexp finds all 3-letter doubles with a space
		610	in between:
		611
		612
		613	/(www)s1/;
		614	The grouping assigns a value to 1, so that the same 3 letter sequence is used for both parts. Here are some words with repeated parts:
		615
		616
		617	% simple_grep '^(wwwwwwwwww)1$' /usr/dict/words
		618	beriberi
		619	booboo
		620	coco
		621	mama
		622	murmur
		623	papa
		624	The regexp has a single grouping which considers 4-letter combinations, then 3-letter combinations, etc. and uses 1 to look for a repeat. Although $1 and 1 represent the same thing, care should be taken to use matched variables $1, $2, ... only outside a regexp and backreferences 1, 2, ... only inside a regexp; not doing so may lead to surprising and/or undefined results.
		625
		626
		627	In addition to what was matched, Perl 5.6.0 also provides
		628	the positions of what was matched with the @- and
		629	@+ arrays. $-[[0] is the position of the
		630	start of the entire match and $+[[0] is the position
		631	of the end. Similarly, $-[[n] is the position of the
		632	start of the $n match and $+[[n] is the
		633	position of the end. If $n is undefined, so are
		634	$-[[n] and $+[[n]. Then this
		635	code
		636
		637
		638	$x =
		639	prints
		640
		641
		642	Match 1: 'Mmm' at position (0,3)
		643	Match 2: 'donut' at position (6,11)
		644	Even if there are no groupings in a regexp, it is still possible to find out what exactly matched in a string. If you use them, perl will set $` to the part of the string before the match, will set $ to the part of the string that matched, and will set $' to the part of the string after the match. An example:
		645
		646
		647	$x =
		648	In the second match, $` = '' because the regexp matched at the first character position in the string and stopped, it never saw the second 'the'. It is important to note that using $` and $' slows down regexp matching quite a bit, and $ slows it down to a lesser extent, because if they are used in one regexp in a program, they are generated for @- and @+ instead:
		649
		650
		651	$` is the same as substr( $x, 0, $-[[0] )
		652	$
		653
		654
		655	__Matching repetitions__
		656
		657
		658	The examples in the previous section display an annoying
		659	weakness. We were only matching 3-letter words, or syllables
		660	of 4 letters or less. We'd like to be able to match words or
		661	syllables of any length, without writing out tedious
		662	alternatives like wwwwwwwwww.
		663
		664
		665	This is exactly the problem the __quantifier__
		666	metacharacters ?, *, +, and
		667	{} were created for. They allow us to determine the
		668	number of repeats of a portion of a regexp we consider to be
		669	a match. Quantifiers are put immediately after the
		670	character, character class, or grouping that we want to
		671	specify. They have the following meanings:
		672
		673
		674	a? = match 'a' 1 or 0 times
		675
		676
		677	a* = match 'a' 0 or more times, i.e., any number of
		678	times
		679
		680
		681	a+ = match 'a' 1 or more times, i.e., at least
		682	once
		683
		684
		685	a{n,m} = match at least n times, but not
		686	more than m times.
		687
		688
		689	a{n,} = match at least n or more
		690	times
		691
		692
		693	a{n} = match exactly n times
		694
		695
		696	Here are some examples:
		697
		698
		699	/[[a-z]+s+d*/; # match a lowercase word, at least some space, and
		700	# any number of digits
		701	/(w+)s+1/; # match doubled words of arbitrary length
		702	/y(es)?/i; # matches 'y', 'Y', or a case-insensitive 'yes'
		703	$year =~ /d{2,4}/; # make sure year is at least 2 but not more
		704	# than 4 digits
		705	$year =~ /d{4}d{2}/; # better match; throw out 3 digit dates
		706	$year =~ /d{2}(d{2})?/; # same thing written differently. However,
		707	# this produces $1 and the other does not.
		708	% simple_grep '^(w+)1$' /usr/dict/words # isn't this easier?
		709	beriberi
		710	booboo
		711	coco
		712	mama
		713	murmur
		714	papa
		715	For all of these quantifiers, perl will try to match as much of the string as possible, while still allowing the regexp to succeed. Thus with /a?.../, perl will first try to match the regexp with the a present; if that fails, perl will try to match the regexp without the a present. For the quantifier *, we get the following:
		716
		717
		718	$x =
		719	Which is what we might expect, the match finds the only cat in the string and locks onto it. Consider, however, this regexp:
		720
		721
		722	$x =~ /^(.*)(at)(.*)$/; # matches,
		723	# $1 = 'the cat in the h'
		724	# $2 = 'at'
		725	# $3 = '' (0 matches)
		726	One might initially guess that perl would find the at in cat and stop there, but that wouldn't give the longest possible string to the first quantifier .*. Instead, the first quantifier .* grabs as much of the string as possible while still having the regexp match. In this example, that means having the at sequence with the final at in the string. The other important principle illustrated here is that when there are two or more elements in a regexp, the ''leftmost'' quantifier, if there is one, gets to grab as much the string as possible, leaving the rest of the regexp to fight over scraps. Thus in our example, the first quantifier .* grabs most of the string, while the second quantifier .* gets the empty string. Quantifiers that grab as much of the string as possible are called __maximal match__ or __greedy__ quantifiers.
		727
		728
		729	When a regexp can match a string in several different ways,
		730	we can use the principles above to predict which way the
		731	regexp will match:
		732
		733
		734	Principle 0: Taken as a whole, any regexp will be matched at
		735	the earliest possible position in the string.
		736
		737
		738	Principle 1: In an alternation abc..., the leftmost
		739	alternative that allows a match for the whole regexp will be
		740	the one used.
		741
		742
		743	Principle 2: The maximal matching quantifiers ?,
		744	*, + and {n,m} will in general
		745	match as much of the string as possible while still allowing
		746	the whole regexp to match.
		747
		748
		749	Principle 3: If there are two or more elements in a regexp,
		750	the leftmost greedy quantifier, if any, will match as much
		751	of the string as possible while still allowing the whole
		752	regexp to match. The next leftmost greedy quantifier, if
		753	any, will try to match as much of the string remaining
		754	available to it as possible, while still allowing the whole
		755	regexp to match. And so on, until all the regexp elements
		756	are satisfied.
		757
		758
		759	As we have seen above, Principle 0 overrides the others -
		760	the regexp will be matched as early as possible, with the
		761	other principles determining how the regexp matches at that
		762	earliest character position.
		763
		764
		765	Here is an example of these principles in
		766	action:
		767
		768
		769	$x =
		770	This regexp matches at the earliest string position, 'T'. One might think that e, being leftmost in the alternation, would be matched, but r produces the longest string in the first quantifier.
		771
		772
		773	$x =~ /(m{1,2})(.*)$/; # matches,
		774	# $1 = 'mm'
		775	# $2 = 'ing republic of Perl'
		776	Here, The earliest possible match is at the first 'm' in programming. m{1,2} is the first quantifier, so it gets to match a maximal mm.
		777
		778
		779	$x =~ /.*(m{1,2})(.*)$/; # matches,
		780	# $1 = 'm'
		781	# $2 = 'ing republic of Perl'
		782	Here, the regexp matches at the start of the string. The first quantifier .* grabs as much as possible, leaving just a single 'm' for the second quantifier m{1,2}.
		783
		784
		785	$x =~ /(.?)(m{1,2})(.*)$/; # matches,
		786	# $1 = 'a'
		787	# $2 = 'mm'
		788	# $3 = 'ing republic of Perl'
		789	Here, .? eats its maximal one character at the earliest possible position in the string, 'a' in programming, leaving m{1,2} the opportunity to match both m's. Finally,
		790
		791
		792
		793	because it can match zero copies of 'X' at the beginning of the string. If you definitely want to match at least one 'X', use X+, not X*.
		794
		795
		796	Sometimes greed is not good. At times, we would like
		797	quantifiers to match a ''minimal'' piece of string,
		798	rather than a maximal piece. For this purpose, Larry Wall
		799	created the __minimal match__ or __non-greedy__
		800	quantifiers ??,*?, +?, and
		801	{}?. These are the usual quantifiers with a
		802	? appended to them. They have the following
		803	meanings:
		804
		805
		806	a?? = match 'a' 0 or 1 times. Try 0 first, then
		807	1.
		808
		809
		810	a*? = match 'a' 0 or more times, i.e., any number
		811	of times, but as few times as possible
		812
		813
		814	a+? = match 'a' 1 or more times, i.e., at least
		815	once, but as few times as possible
		816
		817
		818	a{n,m}? = match at least n times, not more
		819	than m times, as few times as possible
		820
		821
		822	a{n,}? = match at least n times, but as
		823	few times as possible
		824
		825
		826	a{n}? = match exactly n times. Because we
		827	match exactly n times, a{n}? is equivalent
		828	to a{n} and is just there for notational
		829	consistency.
		830
		831
		832	Let's look at the example above, but with minimal
		833	quantifiers:
		834
		835
		836	$x =
		837	The minimal string that will allow both the start of the string ^ and the alternation to match is Th, with the alternation er matching e. The second quantifier .* is free to gobble up the rest of the string.
		838
		839
		840	$x =~ /(m{1,2}?)(.*?)$/; # matches,
		841	# $1 = 'm'
		842	# $2 = 'ming republic of Perl'
		843	The first string position that this regexp can match is at the first 'm' in programming. At this position, the minimal m{1,2}? matches just one 'm'. Although the second quantifier .*? would prefer to match no characters, it is constrained by the end-of-string anchor $ to match the rest of the string.
		844
		845
		846	$x =~ /(.*?)(m{1,2}?)(.*)$/; # matches,
		847	# $1 = 'The progra'
		848	# $2 = 'm'
		849	# $3 = 'ming republic of Perl'
		850	In this regexp, you might expect the first minimal quantifier .*? to match the empty string, because it is not constrained by a ^ anchor to match the beginning of the word. Principle 0 applies here, however. Because it is possible for the whole regexp to match at the start of the string, it ''will'' match at the start of the string. Thus the first quantifier has to match everything up to the first m. The second minimal quantifier matches just one m and the third quantifier matches the rest of the string.
		851
		852
		853	$x =~ /(.??)(m{1,2})(.*)$/; # matches,
		854	# $1 = 'a'
		855	# $2 = 'mm'
		856	# $3 = 'ing republic of Perl'
		857	Just as in the previous regexp, the first quantifier .?? can match earliest at position 'a', so it does. The second quantifier is greedy, so it matches mm, and the third matches the rest of the string.
		858
		859
		860	We can modify principle 3 above to take into account
		861	non-greedy quantifiers:
		862
		863
		864	Principle 3: If there are two or more elements in a regexp,
		865	the leftmost greedy (non-greedy) quantifier, if any, will
		866	match as much (little) of the string as possible while still
		867	allowing the whole regexp to match. The next leftmost greedy
		868	(non-greedy) quantifier, if any, will try to match as much
		869	(little) of the string remaining available to it as
		870	possible, while still allowing the whole regexp to match.
		871	And so on, until all the regexp elements are
		872	satisfied.
		873
		874
		875	Just like alternation, quantifiers are also susceptible to
		876	backtracking. Here is a step-by-step analysis of the
		877	example
		878
		879
		880	$x =
		881
		882
		883	Start with the first letter in the string 't'.
		884
		885
		886	1
		887
		888
		889	The first quantifier '.*' starts out by matching the whole
		890	string 'the cat in the hat'.
		891
		892
		893	2
		894
		895
		896	'a' in the regexp element 'at' doesn't match the end of the
		897	string. Backtrack one character.
		898
		899
		900	3
		901
		902
		903	'a' in the regexp element 'at' still doesn't match the last
		904	letter of the string 't', so backtrack one more
		905	character.
		906
		907
		908	4
		909
		910
		911	Now we can match the 'a' and the 't'.
		912
		913
		914	5
		915
		916
		917	Move on to the third element '.*'. Since we are at the end
		918	of the string and '.*' can match 0 times, assign it the
		919	empty string.
		920
		921
		922	6
		923
		924
		925	We are done!
		926
		927
		928	Most of the time, all this moving forward and backtracking
		929	happens quickly and searching is fast. There are some
		930	pathological regexps, however, whose execution time
		931	exponentially grows with the size of the string. A typical
		932	structure that blows up in your face is of the
		933	form
		934
		935
		936	/(ab+)*/;
		937	The problem is the nested indeterminate quantifiers. There are many different ways of partitioning a string of length n between the + and *: one repetition with b+ of length n, two repetitions with the first b+ length k and the second with length n-k, m repetitions whose bits add up to length n, etc. In fact there are an exponential number of ways to partition a string as a function of length. A regexp may get lucky and match early in the process, but if there is no match, perl will try ''every'' possibility before giving up. So be careful with nested *'s, {n,m}'s, and +'s. The book ''Mastering regular expressions'' by Jeffrey Friedl gives a wonderful discussion of this and other efficiency issues.
		938
		939
		940	__Building a regexp__
		941
		942
		943	At this point, we have all the basic regexp concepts
		944	covered, so let's give a more involved example of a regular
		945	expression. We will build a regexp that matches
		946	numbers.
		947
		948
		949	The first task in building a regexp is to decide what we
		950	want to match and what we want to exclude. In our case, we
		951	want to match both integers and floating point numbers and
		952	we want to reject any string that isn't a
		953	number.
		954
		955
		956	The next task is to break the problem down into smaller
		957	problems that are easily converted into a
		958	regexp.
		959
		960
		961	The simplest case is integers. These consist of a sequence
		962	of digits, with an optional sign in front. The digits we can
		963	represent with d+ and the sign can be matched with
		964	[[+-]. Thus the integer regexp is
		965
		966
		967	/[[+-]?d+/; # matches integers
		968	A floating point number potentially has a sign, an integral part, a decimal point, a fractional part, and an exponent. One or more of these parts is optional, so we need to check out the different possibilities. Floating point numbers which are in proper form include 123., 0.345, .34, -1e6, and 25.4E-72. As with integers, the sign out front is completely optional and can be matched by [[+-]?. We can see that if there is no exponent, floating point numbers must have a decimal point, otherwise they are integers. We might be tempted to model these with d*.d*, but this would also match just a single decimal point, which is not a number. So the three cases of floating point number sans exponent are
		969
		970
		971	/[[+-]?d+./; # 1., 321., etc.
		972	/[[+-]?.d+/; # .1, .234, etc.
		973	/[[+-]?d+.d+/; # 1.0, 30.56, etc.
		974	These can be combined into a single regexp with a three-way alternation:
		975
		976
		977	/[[+-]?(d+.d+d+..d+)/; # floating point, no exponent
		978	In this alternation, it is important to put 'd+.d+' before 'd+.'. If 'd+.' were first, the regexp would happily match that and ignore the fractional part of the number.
		979
		980
		981	Now consider floating point numbers with exponents. The key
		982	observation here is that ''both'' integers and numbers
		983	with decimal points are allowed in front of an exponent.
		984	Then exponents, like the overall sign, are independent of
		985	whether we are matching numbers with or without decimal
		986	points, and can be 'decoupled' from the mantissa. The
		987	overall form of the regexp now becomes clear:
		988
		989
		990	/^(optional sign)(integer f.p. mantissa)(optional exponent)$/;
		991	The exponent is an e or E, followed by an integer. So the exponent regexp is
		992
		993
		994	/[[eE][[+-]?d+/; # exponent
		995	Putting all the parts together, we get a regexp that matches numbers:
		996
		997
		998	/^[[+-]?(d+.d+d+..d+d+)([[eE][[+-]?d+)?$/; # Ta da!
		999	Long regexps like this may impress your friends, but can be hard to decipher. In complex situations like this, the //x modifier for a match is invaluable. It allows one to put nearly arbitrary whitespace and comments into a regexp without affecting their meaning. Using it, we can rewrite our 'extended' regexp in the more pleasing form
		1000
		1001
		1002	/^
		1003	[[+-]? # first, match an optional sign
		1004	( # then match integers or f.p. mantissas:
		1005	d+.d+ # mantissa of the form a.b
		1006	d+. # mantissa of the form a.
		1007	.d+ # mantissa of the form .b
		1008	d+ # integer of the form a
		1009	)
		1010	([[eE][[+-]?d+)? # finally, optionally match an exponent
		1011	$/x;
		1012	If whitespace is mostly irrelevant, how does one include space characters in an extended regexp? The answer is to backslash it '\ ' or put it in a character class [[ ] . The same thing goes for pound signs, use # or [[#]. For instance, Perl allows a space between the sign and the mantissa/integer, and we could add this to our regexp as follows:
		1013
		1014
		1015	/^
		1016	[[+-]?\ * # first, match an optional sign *and space*
		1017	( # then match integers or f.p. mantissas:
		1018	d+.d+ # mantissa of the form a.b
		1019	d+. # mantissa of the form a.
		1020	.d+ # mantissa of the form .b
		1021	d+ # integer of the form a
		1022	)
		1023	([[eE][[+-]?d+)? # finally, optionally match an exponent
		1024	$/x;
		1025	In this form, it is easier to see a way to simplify the alternation. Alternatives 1, 2, and 4 all start with d+, so it could be factored out:
		1026
		1027
		1028	/^
		1029	[[+-]?\ * # first, match an optional sign
		1030	( # then match integers or f.p. mantissas:
		1031	d+ # start out with a ...
		1032	(
		1033	.d* # mantissa of the form a.b or a.
		1034	)? # ? takes care of integers of the form a
		1035	.d+ # mantissa of the form .b
		1036	)
		1037	([[eE][[+-]?d+)? # finally, optionally match an exponent
		1038	$/x;
		1039	or written in the compact form,
		1040
		1041
		1042	/^[[+-]?\ *(d+(.d*)?.d+)([[eE][[+-]?d+)?$/;
		1043	This is our final regexp. To recap, we built a regexp by
		1044
		1045
		1046	specifying the task in detail,
		1047
		1048
		1049	breaking down the problem into smaller parts,
		1050
		1051
		1052	translating the small parts into regexps,
		1053
		1054
		1055	combining the regexps,
		1056
		1057
		1058	and optimizing the final combined regexp.
		1059
		1060
		1061	These are also the typical steps involved in writing a
		1062	computer program. This makes perfect sense, because regular
		1063	expressions are essentially programs written a little
		1064	computer language that specifies patterns.
		1065
		1066
		1067	__Using regular expressions in Perl__
		1068
		1069
		1070	The last topic of Part 1 briefly covers how regexps are used
		1071	in Perl programs. Where do they fit into Perl
		1072	syntax?
		1073
		1074
		1075	We have already introduced the matching operator in its
		1076	default /regexp/ and arbitrary delimiter
		1077	m!regexp! forms. We have used the binding operator
		1078	=~ and its negation !~ to test for string
		1079	matches. Associated with the matching operator, we have
		1080	discussed the single line //s, multi-line
		1081	//m, case-insensitive //i and extended
		1082	//x modifiers.
		1083
		1084
		1085	There are a few more things you might want to know about
		1086	matching operators. First, we pointed out earlier that
		1087	variables in regexps are substituted before the regexp is
		1088	evaluated:
		1089
		1090
		1091	$pattern = 'Seuss';
		1092	while (
		1093	This will print any lines containing the word Seuss. It is not as efficient as it could be, however, because perl has to re-evaluate $pattern each time through the loop. If $pattern won't be changing over the lifetime of the script, we can add the //o modifier, which directs perl to only perform variable substitutions once:
		1094
		1095
		1096	#!/usr/bin/perl
		1097	# Improved simple_grep
		1098	$regexp = shift;
		1099	while (
		1100	If you change $pattern after the first substitution happens, perl will ignore it. If you don't want any substitutions at all, use the special delimiter m'':
		1101
		1102
		1103	$pattern = 'Seuss';
		1104	while (
		1105	m'' acts like single quotes on a regexp; all other m delimiters act like double quotes. If the regexp evaluates to the empty string, the regexp in the ''last successful match'' is used instead. So we have
		1106
		1107
		1108
		1109	The final two modifiers //g and //c concern multiple matches. The modifier //g stands for global matching and allows the the matching operator to match within a string as many times as possible. In scalar context, successive invocations against a string will have `//g jump from match to match, keeping track of position in the string as it goes along. You can get or set the position with the pos() function.
		1110
		1111
		1112	The use of //g is shown in the following example.
		1113	Suppose we have a string that consists of words separated by
		1114	spaces. If we know how many words there are in advance, we
		1115	could extract the words using groupings:
		1116
		1117
		1118	$x =
		1119	But what if we had an indeterminate number of words? This is the sort of task //g was made for. To extract all words, form the simple regexp (w+) and loop over all matches with /(w+)/g:
		1120
		1121
		1122	while ($x =~ /(w+)/g) {
		1123	print
		1124	prints
		1125
		1126
		1127	Word is cat, ends at position 3
		1128	Word is dog, ends at position 7
		1129	Word is house, ends at position 13
		1130	A failed match or changing the target string resets the position. If you don't want the position reset after failure to match, add the //c, as in /regexp/gc. The current position in the string is associated with the string, not the regexp. This means that different strings have different positions and their respective positions can be set or read independently.
		1131
		1132
		1133	In list context, //g returns a list of matched
		1134	groupings, or if there are no groupings, a list of matches
		1135	to the whole regexp. So if we wanted just the words, we
		1136	could use
		1137
		1138
		1139	@words = ($x =~ /(w+)/g); # matches,
		1140	# $word[[0] = 'cat'
		1141	# $word[[1] = 'dog'
		1142	# $word[[2] = 'house'
		1143	Closely associated with the //g modifier is the G anchor. The G anchor matches at the point where the previous //g match left off. G allows us to easily do context-sensitive matching:
		1144
		1145
		1146	$metric = 1; # use metric units
		1147	...
		1148	$x =
		1149	The combination of //g and G allows us to process the string a bit at a time and use arbitrary Perl logic to decide what to do next.
		1150
		1151
		1152	G is also invaluable in processing fixed length
		1153	records with regexps. Suppose we have a snippet of coding
		1154	region DNA , encoded as base pair letters
		1155	ATCGTTGAAT... and we want to find all the stop
		1156	codons TGA. In a coding region, codons are 3-letter
		1157	sequences, so we can think of the DNA snippet
		1158	as a sequence of 3-letter records. The naive
		1159	regexp
		1160
		1161
		1162	# expanded, this is
		1163	doesn't work; it may match an TGA, but there is no guarantee that the match is aligned with codon boundaries, e.g., the substring GTT GAA gives a match. A better solution is
		1164
		1165
		1166	while ($dna =~ /(www)*?TGA/g) { # note the minimal *?
		1167	print
		1168	which prints
		1169
		1170
		1171	Got a TGA stop codon at position 18
		1172	Got a TGA stop codon at position 23
		1173	Position 18 is good, but position 23 is bogus. What happened?
		1174
		1175
		1176	The answer is that our regexp works well until we get past
		1177	the last real match. Then the regexp will fail to match a
		1178	synchronized TGA and start stepping ahead one
		1179	character position at a time, not what we want. The solution
		1180	is to use G to anchor the match to the codon
		1181	alignment:
		1182
		1183
		1184	while ($dna =~ /G(www)*?TGA/g) {
		1185	print
		1186	This prints
		1187
		1188
		1189	Got a TGA stop codon at position 18
		1190	which is the correct answer. This example illustrates that it is important not only to match what is desired, but to reject what is not desired.
		1191
		1192
		1193	__search and replace__
		1194
		1195
		1196	Regular expressions also play a big role in __search and
		1197	replace__ operations in Perl. Search and replace is
		1198	accomplished with the s/// operator. The general
		1199	form is s/regexp/replacement/modifiers, with
		1200	everything we know about regexps and modifiers applying in
		1201	this case as well. The replacement is a Perl double
		1202	quoted string that replaces in the string whatever is
		1203	matched with the regexp. The operator =~
		1204	is also used here to associate a string with s///.
		1205	If matching against $_, the $_ =~ can be
		1206	dropped. If there is a match, s/// returns the
		1207	number of substitutions made, otherwise it returns false.
		1208	Here are a few examples:
		1209
		1210
		1211	$x =
		1212	In the last example, the whole string was matched, but only the part inside the single quotes was grouped. With the s/// operator, the matched variables $1, $2, etc. are immediately available for use in the replacement expression, so we use $1 to replace the quoted string with just what was quoted. With the global modifier, s///g will search and replace all occurrences of the regexp in the string:
		1213
		1214
		1215	$x =
		1216	If you prefer 'regex' over 'regexp' in this tutorial, you could use the following program to replace it:
		1217
		1218
		1219	% cat
		1220	% simple_replace regexp regex perlretut.pod
		1221	In simple_replace we used the s///g modifier to replace all occurrences of the regexp on each line and the s///o modifier to compile the regexp only once. As with simple_grep, both the print and the s/$regexp/$replacement/go use $_ implicitly.
		1222
		1223
		1224	A modifier available specifically to search and replace is
		1225	the s///e evaluation modifier. s///e wraps
		1226	an eval{...} around the replacement string and the
		1227	evaluated result is substituted for the matched substring.
		1228	s///e is useful if you need to do a bit of
		1229	computation in the process of replacing text. This example
		1230	counts character frequencies in a line:
		1231
		1232
		1233	$x =
		1234	This prints
		1235
		1236
		1237	frequency of ' ' is 2
		1238	frequency of 't' is 2
		1239	frequency of 'l' is 2
		1240	frequency of 'B' is 1
		1241	frequency of 'c' is 1
		1242	frequency of 'e' is 1
		1243	frequency of 'h' is 1
		1244	frequency of 'i' is 1
		1245	frequency of 'a' is 1
		1246	As with the match m// operator, s/// can use other delimiters, such as s!!! and s{}{}, and even s{}//. If single quotes are used s''', then the regexp and replacement are treated as single quoted strings and there are no substitutions. s/// in list context returns the same thing as in scalar context, i.e., the number of matches.
		1247
		1248
		1249	__The split operator__
		1250
		1251
		1252	The __split__ function can also optionally use a matching
		1253	operator m// to split a string. split /regexp/,
		1254	string, limit splits string into a list of
		1255	substrings and returns that list. The regexp is used to
		1256	match the character sequence that the string is
		1257	split with respect to. The limit, if present,
		1258	constrains splitting into no more than limit number
		1259	of strings. For example, to split a string into words,
		1260	use
		1261
		1262
		1263	$x =
		1264	If the empty regexp // is used, the regexp always matches and the string is split into individual characters. If the regexp has groupings, then list produced contains the matched substrings from the groupings as well. For instance,
		1265
		1266
		1267	$x =
		1268	Since the first character of $x matched the regexp, split prepended an empty initial element to the list.
		1269
		1270
		1271	If you have read this far, congratulations! You now have all
		1272	the basic tools needed to use regular expressions to solve a
		1273	wide range of text processing problems. If this is your
		1274	first time through the tutorial, why not stop here and play
		1275	around with regexps a while... Part 2 concerns the more
		1276	esoteric aspects of regular expressions and those concepts
		1277	certainly aren't needed right at the start.
		1278	!!Part 2: Power tools
		1279
		1280
		1281	OK , you know the basics of regexps and you
		1282	want to know more. If matching regular expressions is
		1283	analogous to a walk in the woods, then the tools discussed
		1284	in Part 1 are analogous to topo maps and a compass, basic
		1285	tools we use all the time. Most of the tools in part 2 are
		1286	are analogous to flare guns and satellite phones. They
		1287	aren't used too often on a hike, but when we are stuck, they
		1288	can be invaluable.
		1289
		1290
		1291	What follows are the more advanced, less used, or sometimes
		1292	esoteric capabilities of perl regexps. In Part 2, we will
		1293	assume you are comfortable with the basics and concentrate
		1294	on the new features.
		1295
		1296
		1297	__More on characters, strings, and character
		1298	classes__
		1299
		1300
		1301	There are a number of escape sequences and character classes
		1302	that we haven't covered yet.
		1303
		1304
		1305	There are several escape sequences that convert characters
		1306	or strings between upper and lower case. l and
		1307	u convert the next character to lower or upper
		1308	case, respectively:
		1309
		1310
		1311	$x =
		1312	L and U converts a whole substring, delimited by L or U and E, to lower or upper case:
		1313
		1314
		1315	$x =
		1316	If there is no E, case is converted until the end of the string. The regexps Lu$word or uL$word convert the first character of $word to uppercase and the rest of the characters to lowercase.
		1317
		1318
		1319	Control characters can be escaped with c, so that a
		1320	control-Z character would be matched with cZ. The
		1321	escape sequence Q...E quotes, or protects
		1322	most non-alphabetic characters. For instance,
		1323
		1324
		1325	$x =
		1326	It does not protect $ or @, so that variables can still be substituted.
		1327
		1328
		1329	With the advent of 5.6.0, perl regexps can handle more than
		1330	just the standard ASCII character set. Perl
		1331	now supports __Unicode__, a standard for encoding the
		1332	character sets from many of the world's written languages.
		1333	Unicode does this by allowing characters to be more than one
		1334	byte wide. Perl uses the UTF-8 encoding, in
		1335	which ASCII characters are still encoded as
		1336	one byte, but characters greater than chr(127) may
		1337	be stored as two or more bytes.
		1338
		1339
		1340	What does this mean for regexps? Well, regexp users don't
		1341	need to know much about perl's internal representation of
		1342	strings. But they do need to know 1) how to represent
		1343	Unicode characters in a regexp and 2) when a matching
		1344	operation will treat the string to be searched as a sequence
		1345	of bytes (the old way) or as a sequence of Unicode
		1346	characters (the new way). The answer to 1) is that Unicode
		1347	characters greater than chr(127) may be represented
		1348	using the x{hex} notation, with hex a
		1349	hexadecimal integer:
		1350
		1351
		1352	use utf8; # We will be doing Unicode processing
		1353	/x{263a}/; # match a Unicode smiley face :)
		1354	Unicode characters in the range of 128-255 use two hexadecimal digits with braces: x{ab}. Note that this is different than xab, which is just a hexadecimal byte with no Unicode significance.
		1355
		1356
		1357	Figuring out the hexadecimal sequence of a Unicode character
		1358	you want or deciphering someone else's hexadecimal Unicode
		1359	regexp is about as much fun as programming in machine code.
		1360	So another way to specify Unicode characters is to use the
		1361	__named character__ escape sequence N{name}.
		1362	name is a name for the Unicode character, as
		1363	specified in the Unicode standard. For instance, if we
		1364	wanted to represent or match the astrological sign for the
		1365	planet Mercury, we could use
		1366
		1367
		1368	use utf8; # We will be doing Unicode processing
		1369	use charnames
		1370	One can also use short names or restrict names to a certain alphabet:
		1371
		1372
		1373	use utf8; # We will be doing Unicode processing
		1374	use charnames ':full';
		1375	print
		1376	use charnames
		1377	use charnames qw(greek);
		1378	print
		1379	A list of full names is found in the file Names.txt in the lib/perl5/5.6.0/unicode directory.
		1380
		1381
		1382	The answer to requirement 2), as of 5.6.0, is that if a
		1383	regexp contains Unicode characters, the string is searched
		1384	as a sequence of Unicode characters. Otherwise, the string
		1385	is searched as a sequence of bytes. If the string is being
		1386	searched as a sequence of Unicode characters, but matching a
		1387	single byte is required, we can use the C escape
		1388	sequence. C is a character class akin to .
		1389	except that it matches ''any'' byte 0-255.
		1390	So
		1391
		1392
		1393	use utf8; # We will be doing Unicode processing
		1394	use charnames
		1395	The last regexp matches, but is dangerous because the string ''character'' position is no longer synchronized to the string ''byte'' position. This generates the warning 'Malformed UTF-8 character'. C is best used for matching the binary data in strings with binary data intermixed with Unicode characters.
		1396
		1397
		1398	Let us now discuss the rest of the character classes. Just
		1399	as with Unicode characters, there are named Unicode
		1400	character classes represented by the p{name} escape
		1401	sequence. Closely associated is the P{name}
		1402	character class, which is the negation of the
		1403	p{name} class. For example, to match lower and
		1404	uppercase characters,
		1405
		1406
		1407	use utf8; # We will be doing Unicode processing
		1408	use charnames
		1409	Here is the association between some Perl named classes and the traditional Unicode classes:
		1410
		1411
		1412	Perl class name Unicode class name or regular expression
2	perry	1413	!IsAlpha /^[[LM]/
		1414	!IsAlnum /^[[LMN]/
1	perry	1415	IsASCII $code
		1416	You can also use the official Unicode class names with the p and P, like p{L} for Unicode 'letters', or p{Lu} for uppercase letters, or P{Nd} for non-digits. If a name is just one letter, the braces can be dropped. For instance, pM is the character class of Unicode 'marks'.
		1417
		1418
		1419	X is an abbreviation for a character class sequence
		1420	that includes the Unicode 'combining character sequences'. A
		1421	'combining character sequence' is a base character followed
		1422	by any number of combining characters. An example of a
		1423	combining character is an accent. Using the Unicode full
		1424	names, e.g., A + COMBINING RING is a combining
		1425	character sequence with base character A and
		1426	combining character COMBINING RING , which
		1427	translates in Danish to A with the circle atop it, as in the
		1428	word Angstrom. X is equivalent to PMpM*},
		1429	i.e., a non-mark followed by one or more marks.
		1430
		1431
		1432	As if all those classes weren't enough, Perl also defines
		1433	POSIX style character classes. These have the
		1434	form [[:name:], with name the name of the
		1435	POSIX class. The POSIX classes
		1436	are alpha, alnum, ascii,
		1437	cntrl, digit, graph,
		1438	lower, print, punct,
		1439	space, upper, and xdigit, and two
		1440	extensions, word (a Perl extension to match
		1441	w), and blank (a GNU
		1442	extension). If utf8 is being used, then these
		1443	classes are defined the same as their corresponding perl
		1444	Unicode classes: [[:upper:] is the same as
2	perry	1445	p{!IsUpper}, etc. The POSIX character
1	perry	1446	classes, however, don't require using utf8. The
		1447	[[:digit:], [[:word:], and
		1448	[[:space:] correspond to the familiar d,
		1449	w, and s character classes. To negate a
		1450	POSIX class, put a ^ in front of the
		1451	name, so that, e.g., [[:^digit:] corresponds to
2	perry	1452	D and under utf8, P{!IsDigit}. The
1	perry	1453	Unicode and POSIX character classes can be
		1454	used just like d, both inside and outside of
		1455	character classes:
		1456
		1457
		1458	/s+[[abc[[:digit:]xyz]s*/; # match a,b,c,x,y,z, or a digit
		1459	/^=items[[:digit:]/; # match '=item',
		1460	# followed by a space and a digit
		1461	use utf8;
		1462	use charnames
		1463	Whew! That is all the rest of the characters and character classes.
		1464
		1465
		1466	__Compiling and saving regular expressions__
		1467
		1468
		1469	In Part 1 we discussed the //o modifier, which
		1470	compiles a regexp just once. This suggests that a compiled
		1471	regexp is some data structure that can be stored once and
		1472	used again and again. The regexp quote qr// does
		1473	exactly that: qr/string/ compiles the
		1474	string as a regexp and transforms the result into a
		1475	form that can be assigned to a variable:
		1476
		1477
		1478	$reg = qr/foo+bar?/; # reg contains a compiled regexp
		1479	Then $reg can be used as a regexp:
		1480
		1481
		1482	$x =
		1483	$reg can also be interpolated into a larger regexp:
		1484
		1485
		1486	$x =~ /(abc)?$reg/; # still matches
		1487	As with the matching operator, the regexp quote can use different delimiters, e.g., qr!!, qr{} and qr~~. The single quote delimiters qr'' prevent any interpolation from taking place.
		1488
		1489
		1490	Pre-compiled regexps are useful for creating dynamic matches
		1491	that don't need to be recompiled each time they are
		1492	encountered. Using pre-compiled regexps,
		1493	simple_grep program can be expanded into a program
		1494	that matches multiple patterns:
		1495
		1496
		1497	% cat
		1498	$number = shift;
		1499	$regexp[[$_] = shift foreach (0..$number-1);
		1500	@compiled = map qr/$_/, @regexp;
		1501	while ($line =
		1502	% multi_grep 2 last for multi_grep
		1503	$regexp[[$_] = shift foreach (0..$number-1);
		1504	foreach $pattern (@compiled) {
		1505	last;
		1506	Storing pre-compiled regexps in an array @compiled allows us to simply loop through the regexps without any recompilation, thus gaining flexibility without sacrificing speed.
		1507
		1508
		1509	__Embedding comments and modifiers in a regular
		1510	expression__
		1511
		1512
		1513	Starting with this section, we will be discussing Perl's set
		1514	of __extended patterns__. These are extensions to the
		1515	traditional regular expression syntax that provide powerful
		1516	new tools for pattern matching. We have already seen
		1517	extensions in the form of the minimal matching constructs
		1518	??, *?, +?, {n,m}?, and
		1519	{n,}?. The rest of the extensions below have the
		1520	form (?char...), where the char is a
		1521	character that determines the type of
		1522	extension.
		1523
		1524
		1525	The first extension is an embedded comment
		1526	(?#text). This embeds a comment into the regular
		1527	expression without affecting its meaning. The comment should
		1528	not have any closing parentheses in the text. An example
		1529	is
		1530
		1531
		1532	/(?# Match an integer:)[[+-]?d+/;
		1533	This style of commenting has been largely superseded by the raw, freeform commenting that is allowed with the //x modifier.
		1534
		1535
		1536	The modifiers //i, //m, //s, and
		1537	//x can also embedded in a regexp using
		1538	(?i), (?m), (?s), and
		1539	(?x). For instance,
		1540
		1541
		1542	/(?i)yes/; # match 'yes' case insensitively
		1543	/yes/i; # same thing
		1544	/(?x)( # freeform version of an integer regexp
		1545	[[+-]? # match an optional sign
		1546	d+ # match a sequence of digits
		1547	)
		1548	/x;
		1549	Embedded modifiers can have two important advantages over the usual modifiers. Embedded modifiers allow a custom set of modifiers to ''each'' regexp pattern. This is great for matching an array of regexps that must have different modifiers:
		1550
		1551
		1552	$pattern[[0] = '(?i)doctor';
		1553	$pattern[[1] = 'Johnson';
		1554	...
		1555	while (
		1556	The second advantage is that embedded modifiers only affect the regexp inside the group the embedded modifier is contained in. So grouping can be used to localize the modifier's effects:
		1557
		1558
		1559	/Answer: ((?i)yes)/; # matches 'Answer: yes', 'Answer: YES', etc.
		1560	Embedded modifiers can also turn off any modifiers already present by using, e.g., (?-i). Modifiers can also be combined into a single expression, e.g., (?s-i) turns on single line mode and turns off case insensitivity.
		1561
		1562
		1563	__Non-capturing groupings__
		1564
		1565
		1566	We noted in Part 1 that groupings () had two
		1567	distinct functions: 1) group regexp elements together as a
		1568	single unit, and 2) extract, or capture, substrings that
		1569	matched the regexp in the grouping. Non-capturing groupings,
		1570	denoted by (?:regexp), allow the regexp to be
		1571	treated as a single unit, but don't extract substrings or
		1572	set matching variables $1, etc. Both capturing and
		1573	non-capturing groupings are allowed to co-exist in the same
		1574	regexp. Because there is no extraction, non-capturing
		1575	groupings are faster than capturing groupings. Non-capturing
		1576	groupings are also handy for choosing exactly which parts of
		1577	a regexp are to be extracted to matching
		1578	variables:
		1579
		1580
		1581	# match a number, $1-$4 are set, but we only want $1
		1582	/([[+-]?\ *(d+(.d*)?.d+)([[eE][[+-]?d+)?)/;
		1583	# match a number faster , only $1 is set
		1584	/([[+-]?\ *(?:d+(?:.d*)?.d+)(?:[[eE][[+-]?d+)?)/;
		1585	# match a number, get $1 = whole number, $2 = exponent
		1586	/([[+-]?\ *(?:d+(?:.d*)?.d+)(?:[[eE]([[+-]?d+))?)/;
		1587	Non-capturing groupings are also useful for removing nuisance elements gathered from a split operation:
		1588
		1589
		1590	$x = '12a34b5';
		1591	@num = split /(ab)/, $x; # @num = ('12','a','34','b','5')
		1592	@num = split /(?:ab)/, $x; # @num = ('12','34','5')
		1593	Non-capturing groupings may also have embedded modifiers: (?i-m:regexp) is a non-capturing grouping that matches regexp case insensitively and turns off multi-line mode.
		1594
		1595
		1596	__Looking ahead and looking behind__
		1597
		1598
		1599	This section concerns the lookahead and lookbehind
		1600	assertions. First, a little background.
		1601
		1602
		1603	In Perl regular expressions, most regexp elements 'eat up' a
		1604	certain amount of string when they match. For instance, the
		1605	regexp element [[abc}] eats up one character of the
		1606	string when it matches, in the sense that perl moves to the
		1607	next character position in the string after the match. There
		1608	are some elements, however, that don't eat up characters
		1609	(advance the character position) if they match. The examples
		1610	we have seen so far are the anchors. The anchor ^
		1611	matches the beginning of the line, but doesn't eat any
		1612	characters. Similarly, the word boundary anchor b
		1613	matches, e.g., if the character to the left is a word
		1614	character and the character to the right is a non-word
		1615	character, but it doesn't eat up any characters itself.
		1616	Anchors are examples of 'zero-width assertions'. Zero-width,
		1617	because they consume no characters, and assertions, because
		1618	they test some property of the string. In the context of our
		1619	walk in the woods analogy to regexp matching, most regexp
		1620	elements move us along a trail, but anchors have us stop a
		1621	moment and check our surroundings. If the local environment
		1622	checks out, we can proceed forward. But if the local
		1623	environment doesn't satisfy us, we must
		1624	backtrack.
		1625
		1626
		1627	Checking the environment entails either looking ahead on the
		1628	trail, looking behind, or both. ^ looks behind, to
		1629	see that there are no characters before. $ looks
		1630	ahead, to see that there are no characters after. b
		1631	looks both ahead and behind, to see if the characters on
		1632	either side differ in their 'word'-ness.
		1633
		1634
		1635	The lookahead and lookbehind assertions are generalizations
		1636	of the anchor concept. Lookahead and lookbehind are
		1637	zero-width assertions that let us specify which characters
		1638	we want to test for. The lookahead assertion is denoted by
		1639	(?=regexp) and the lookbehind assertion is denoted
		1640	by (?. Some examples
		1641	are
		1642
		1643
		1644	$x =
		1645	Note that the parentheses in (?=regexp) and (? are non-capturing, since these are zero-width assertions. Thus in the second regexp, the substrings captured are those of the whole regexp itself. Lookahead (?=regexp) can match arbitrary regexps, but lookbehind (? only works for regexps of fixed width, i.e., a fixed number of characters long. Thus (? is fine, but (? is not. The negated versions of the lookahead and lookbehind assertions are denoted by (?!regexp) and (? respectively. They evaluate true if the regexps do ''not'' match:
		1646
		1647
		1648	$x =
		1649
		1650
		1651	__Using independent subexpressions to prevent
		1652	backtracking__
		1653
		1654
		1655	The last few extended patterns in this tutorial are
		1656	experimental as of 5.6.0. Play with them, use them in some
		1657	code, but don't rely on them just yet for production
		1658	code.
		1659
		1660
		1661	__Independent subexpressions__ are regular expressions,
		1662	in the context of a larger regular expression, that function
		1663	independently of the larger regular expression. That is,
		1664	they consume as much or as little of the string as they wish
		1665	without regard for the ability of the larger regexp to
		1666	match. Independent subexpressions are represented by
		1667	(?. We can illustrate their behavior by
		1668	first considering an ordinary regexp:
		1669
		1670
		1671	$x =
		1672	This obviously matches, but in the process of matching, the subexpression a* first grabbed the a. Doing so, however, wouldn't allow the whole regexp to match, so after backtracking, a* eventually gave back the a and matched the empty string. Here, what a* matched was ''dependent'' on what the rest of the regexp matched.
		1673
		1674
		1675	Contrast that with an independent
		1676	subexpression:
		1677
		1678
		1679	$x =~ /(?
		1680	The independent subexpression (? doesn't care about the rest of the regexp, so it sees an a and grabs it. Then the rest of the regexp ab cannot match. Because (? is independent, there is no backtracking and and the independent subexpression does not give up its a. Thus the match of the regexp as a whole fails. A similar behavior occurs with completely independent regexps:
		1681
		1682
		1683	$x =
		1684	Here //g and G create a 'tag team' handoff of the string from one regexp to the other. Regexps with an independent subexpression are much like this, with a handoff of the string to the independent subexpression, and a handoff of the string back to the enclosing regexp.
		1685
		1686
		1687	The ability of an independent subexpression to prevent
		1688	backtracking can be quite useful. Suppose we want to match a
		1689	non-empty string enclosed in parentheses up to two levels
		1690	deep. Then the following regexp matches:
		1691
		1692
		1693	$x =
		1694	The regexp matches an open parenthesis, one or more copies of an alternation, and a close parenthesis. The alternation is two-way, with the first alternative [[^()]+ matching a substring with no parentheses and the second alternative matching a substring delimited by parentheses. The problem with this regexp is that it is pathological: it has nested indeterminate quantifiers of the form (a+b)+. We discussed in Part 1 how nested quantifiers like this could take an exponentially long time to execute if there was no match possible. To prevent the exponential blowup, we need to prevent useless backtracking at some point. This can be done by enclosing the inner quantifier as an independent subexpression:
		1695
		1696
		1697	$x =~ / ( (?
		1698	Here, (? breaks the degeneracy of string partitioning by gobbling up as much of the string as possible and keeping it. Then match failures fail much more quickly.
		1699
		1700
		1701	__Conditional expressions__
		1702
		1703
		1704	A __conditional expression__ is a form of if-then-else
		1705	statement that allows one to choose which patterns are to be
		1706	matched, based on some condition. There are two types of
		1707	conditional expression: (?(condition)yes-regexp)
		1708	and (?(condition)yes-regexpno-regexp).
		1709	(?(condition)yes-regexp) is like an 'if ()
		1710	{}' statement in Perl. If the condition is
		1711	true, the yes-regexp will be matched. If the
		1712	condition is false, the yes-regexp will be
		1713	skipped and perl will move onto the next regexp element. The
		1714	second form is like an 'if () {} else {}' statement
		1715	in Perl. If the condition is true, the
		1716	yes-regexp will be matched, otherwise the
		1717	no-regexp will be matched.
		1718
		1719
		1720	The condition can have two forms. The first form is
		1721	simply an integer in parentheses (integer). It is
		1722	true if the corresponding backreference integer
		1723	matched earlier in the regexp. The second form is a bare
		1724	zero width assertion (?...), either a lookahead, a
		1725	lookbehind, or a code assertion (discussed in the next
		1726	section).
		1727
		1728
		1729	The integer form of the condition allows us to
		1730	choose, with more flexibility, what to match based on what
		1731	matched earlier in the regexp. This searches for words of
		1732	the form or
		1733	:
		1734
		1735
		1736	% simple_grep '^(w+)(w+)?(?(2)211)$' /usr/dict/words
		1737	beriberi
		1738	coco
		1739	couscous
		1740	deed
		1741	...
		1742	toot
		1743	toto
		1744	tutu
		1745	The lookbehind condition allows, along with backreferences, an earlier part of the match to influence a later part of the match. For instance,
		1746
		1747
		1748	/[[ATGC]+(?(?
		1749	matches a DNA sequence such that it either ends in AAG, or some other base pair combination and C. Note that the form is (?(? and not (?((?; for the lookahead, lookbehind or code assertions, the parentheses around the conditional are not needed.
		1750
		1751
		1752	__A bit of magic: executing Perl code in a regular
		1753	expression__
		1754
		1755
		1756	Normally, regexps are a part of Perl expressions. __Code
		1757	evaluation__ expressions turn that around by allowing
		1758	arbitrary Perl code to be a part of of a regexp. A code
		1759	evaluation expression is denoted (?{code}), with
		1760	code a string of Perl statements.
		1761
		1762
		1763	Code expressions are zero-width assertions, and the value
		1764	they return depends on their environment. There are two
		1765	possibilities: either the code expression is used as a
		1766	conditional in a conditional expression
		1767	(?(condition)...), or it is not. If the code
		1768	expression is a conditional, the code is evaluated and the
		1769	result (i.e., the result of the last statement) is used to
		1770	determine truth or falsehood. If the code expression is not
		1771	used as a conditional, the assertion always evaluates true
		1772	and the result is put into the special variable
		1773	$^R. The variable $^R can then be used in
		1774	code expressions later in the regexp. Here are some silly
		1775	examples:
		1776
		1777
		1778	$x =
		1779	Pay careful attention to the next example:
		1780
		1781
		1782	$x =~ /abc(?{print
		1783	At first glance, you'd think that it shouldn't print, because obviously the ddd isn't going to match the target string. But look at this example:
		1784
		1785
		1786	$x =~ /abc(?{print
		1787	Hmm. What happened here? If you've been following along, you know that the above pattern should be effectively the same as the last one -- enclosing the d in a character class isn't going to change what it matches. So why does the first not print while the second one does?
		1788
		1789
		1790	The answer lies in the optimizations the REx engine makes.
		1791	In the first case, all the engine sees are plain old
		1792	characters (aside from the ?{} construct). It's
		1793	smart enough to realize that the string 'ddd' doesn't occur
		1794	in our target string before actually running the pattern
		1795	through. But in the second case, we've tricked it into
		1796	thinking that our pattern is more complicated than it is. It
		1797	takes a look, sees our character class, and decides that it
		1798	will have to actually run the pattern to determine whether
		1799	or not it matches, and in the process of running it hits the
		1800	print statement before it discovers that we don't have a
		1801	match.
		1802
		1803
		1804	To take a closer look at how the engine does optimizations,
		1805	see the section ``Pragmas and debugging''
		1806	below.
		1807
		1808
		1809	More fun with ?{}:
		1810
		1811
		1812	$x =~ /(?{print
		1813	The bit of magic mentioned in the section title occurs when the regexp backtracks in the process of searching for a match. If the regexp backtracks over a code expression and if the variables used within are localized using local, the changes in the variables produced by the code expression are undone! Thus, if we wanted to count how many times a character got matched inside a group, we could use, e.g.,
		1814
		1815
		1816	$x =
		1817	This prints
		1818
		1819
		1820	'a' count is 2, $c variable is 'bob'
		1821	If we replace the (?{local $c = $c + 1;}) with (?{$c = $c + 1;}) , the variable changes are ''not'' undone during backtracking, and we get
		1822
		1823
		1824	'a' count is 4, $c variable is 'bob'
		1825	Note that only localized variable changes are undone. Other side effects of code expression execution are permanent. Thus
		1826
		1827
		1828	$x =
		1829	produces
		1830
		1831
		1832	Yow
		1833	Yow
		1834	Yow
		1835	Yow
		1836	The result $^R is automatically localized, so that it will behave properly in the presence of backtracking.
		1837
		1838
		1839	This example uses a code expression in a conditional to
		1840	match the article 'the' in either English or
		1841	German:
		1842
		1843
		1844	$lang = 'DE'; # use German
		1845	...
		1846	$text =
		1847	Note that the syntax here is (?(?{...})yes-regexpno-regexp), not (?((?{...}))yes-regexpno-regexp). In other words, in the case of a code expression, we don't need the extra parentheses around the conditional.
		1848
		1849
		1850	If you try to use code expressions with interpolating
		1851	variables, perl may surprise you:
		1852
		1853
		1854	$bar = 5;
		1855	$pat = '(?{ 1 })';
		1856	/foo(?{ $bar })bar/; # compiles ok, $bar not interpolated
		1857	/foo(?{ 1 })$bar/; # compile error!
		1858	/foo${pat}bar/; # compile error!
		1859	$pat = qr/(?{ $foo = 1 })/; # precompile code regexp
		1860	/foo${pat}bar/; # compiles ok
		1861	If a regexp has (1) code expressions and interpolating variables,or (2) a variable that interpolates a code expression, perl treats the regexp as an error. If the code expression is precompiled into a variable, however, interpolating is ok. The question is, why is this an error?
		1862
		1863
		1864	The reason is that variable interpolation and code
		1865	expressions together pose a security risk. The combination
		1866	is dangerous because many programmers who write search
		1867	engines often take user input and plug it directly into a
		1868	regexp:
		1869
		1870
		1871	$regexp =
		1872	If the $regexp variable contains a code expression, the user could then execute arbitrary Perl code. For instance, some joker could search for system('rm -rf *'); to erase your files. In this sense, the combination of interpolation and code expressions __taints__ your regexp. So by default, using both interpolation and code expressions in the same regexp is not allowed. If you're not concerned about malicious users, it is possible to bypass this security check by invoking use re 'eval' :
		1873
		1874
		1875	use re 'eval'; # throw caution out the door
		1876	$bar = 5;
		1877	$pat = '(?{ 1 })';
		1878	/foo(?{ 1 })$bar/; # compiles ok
		1879	/foo${pat}bar/; # compiles ok
		1880	Another form of code expression is the __pattern code expression__ . The pattern code expression is like a regular code expression, except that the result of the code evaluation is treated as a regular expression and matched immediately. A simple example is
		1881
		1882
		1883	$length = 5;
		1884	$char = 'a';
		1885	$x = 'aaaaabb';
		1886	$x =~ /(??{$char x $length})/x; # matches, there are 5 of 'a'
		1887	This final example contains both ordinary and pattern code expressions. It detects if a binary string 1101010010001... has a Fibonacci spacing 0,1,1,2,3,5,... of the 1's:
		1888
		1889
		1890	$s0 = 0; $s1 = 1; # initial conditions
		1891	$x =
		1892	This prints
		1893
		1894
		1895	It is a Fibonacci sequence
		1896	Largest sequence matched was 5
		1897	Ha! Try that with your garden variety regexp package...
		1898
		1899
		1900	Note that the variables $s0 and $s1 are
		1901	not substituted when the regexp is compiled, as happens for
		1902	ordinary variables outside a code expression. Rather, the
		1903	code expressions are evaluated when perl encounters them
		1904	during the search for a match.
		1905
		1906
		1907	The regexp without the //x modifier is
		1908
		1909
		1910	/^1((??{'0'x$s0})1(?{$largest=$s0;$s2=$s1+$s0$s0=$s1;$s1=$s2;}))+$/;
		1911	and is a great start on an Obfuscated Perl entry :-) When working with code and conditional expressions, the extended form of regexps is almost necessary in creating and debugging regexps.
		1912
		1913
		1914	__Pragmas and debugging__
		1915
		1916
		1917	Speaking of debugging, there are several pragmas available
		1918	to control and debug regexps in Perl. We have already
		1919	encountered one pragma in the previous section, use re
		1920	'eval'; , that allows variable interpolation and code
		1921	expressions to coexist in a regexp. The other pragmas
		1922	are
		1923
		1924
		1925	use re 'taint';
		1926	$tainted =
		1927	The taint pragma causes any substrings from a match with a tainted variable to be tainted as well. This is not normally the case, as regexps are often used to extract the safe bits from a tainted variable. Use taint when you are not extracting safe bits, but are performing some other processing. Both taint and eval pragmas are lexically scoped, which means they are in effect only until the end of the block enclosing the pragmas.
		1928
		1929
		1930	use re 'debug';
		1931	/^(.*)$/s; # output debugging info
		1932	use re 'debugcolor';
		1933	/^(.*)$/s; # output debugging info in living color
		1934	The global debug and debugcolor pragmas allow one to get detailed debugging info about regexp compilation and execution. debugcolor is the same as debug, except the debugging information is displayed in color on terminals that can display termcap color sequences. Here is example output:
		1935
		1936
		1937	% perl -e 'use re
		1938	If you have gotten this far into the tutorial, you can probably guess what the different parts of the debugging output tell you. The first part
		1939
		1940
		1941	Compiling REx `a*b+c'
		1942	size 9 first at 1
		1943	1: STAR(4)
		1944	2: EXACT
		1945	describes the compilation stage. STAR(4) means that there is a starred object, in this case 'a', and if it matches, goto line 4, i.e., PLUS(7). The middle lines describe some heuristics and optimizations performed before a match:
		1946
		1947
		1948	floating `bc' at 0..2147483647 (checking floating) minlen 2
		1949	Guessing start of match, REx `a*b+c' against `abc'...
		1950	Found floating substr `bc' at offset 1...
		1951	Guessed: match at offset 0
		1952	Then the match is executed and the remaining lines describe the process:
		1953
		1954
		1955	Matching REx `a*b+c' against `abc'
		1956	Setting an EVAL scope, savestack=3
		1957	0
		1958	Each step is of the form n , with the part of the string matched and the part not yet matched. The 1: STAR says that perl is at line number 1 n the compilation list above. See ``Debugging regular expressions'' in perldebguts for much more detail.
		1959
		1960
		1961	An alternative method of debugging regexps is to embed
		1962	print statements within the regexp. This provides a
		1963	blow-by-blow account of the backtracking in an
		1964	alternation:
		1965
		1966
		1967
		1968	prints
		1969
		1970
		1971	Start at position 0
		1972	t1
		1973	h1
		1974	t2
		1975	h2
		1976	a2
		1977	t2
		1978	Done at position 4
		1979	!!BUGS
		1980
		1981
		1982	Code expressions, conditional expressions, and independent
		1983	expressions are __experimental__. Don't use them in
		1984	production code. Yet.
		1985	!!SEE ALSO
		1986
		1987
		1988	This is just a tutorial. For the full story on perl regular
		1989	expressions, see the perlre regular expressions reference
		1990	page.
		1991
		1992
		1993	For more information on the matching m// and
		1994	substitution s/// operators, see ``Regexp
		1995	Quote-Like Operators'' in perlop. For information on the
		1996	split operation, see ``split'' in
		1997	perlfunc.
		1998
		1999
		2000	For an excellent all-around resource on the care and feeding
		2001	of regular expressions, see the book ''Mastering Regular
		2002	Expressions'' by Jeffrey Friedl (published by O'Reilly,
		2003	ISBN 1556592-257-3).
		2004	!!AUTHOR AND COPYRIGHT
		2005
		2006
		2007	Copyright (c) 2000 Mark Kvale All rights
		2008	reserved.
		2009
		2010
		2011	This document may be distributed under the same terms as
		2012	Perl itself.
		2013
		2014
		2015	__Acknowledgments__
		2016
		2017
		2018	The inspiration for the stop codon DNA
		2019	example came from the ZIP code example in
		2020	chapter 7 of ''Mastering Regular
		2021	Expressions''.
		2022
		2023
		2024	The author would like to thank Jeff Pinyan, Andrew Johnson,
		2025	Peter Haworth, Ronald J Kimball, and Joe Smith for all their
		2026	helpful comments.
		2027	----