Penguin
Recent Changes
Annotated edit history of perlfaq6(1) version 2, including all changes. View license author blame.
Rev Author # Line
1 perry 1 PERLFAQ6
2 !!!PERLFAQ6
3 NAME
4 DESCRIPTION
5 AUTHOR AND COPYRIGHT
6 ----
7 !!NAME
8
9
10 perlfaq6 - Regexes ($Revision: 1.27 $, $Date: 1999/05/23 16:08:30 $)
11 !!DESCRIPTION
12
13
14 This section is surprisingly small because the rest of the
15 FAQ is littered with answers involving
16 regular expressions. For example, decoding a
17 URL and checking whether something is a
18 number are handled with regular expressions, but those
19 answers are found elsewhere in this document (in perlfaq9:
20 ``How do I decode or create those %-encodings on the web''
21 and perfaq4: ``How do I determine whether a scalar is a
22 number/whole/integer/float'', to be precise).
23
24
25 __How can I hope to use regular expressions without
26 creating illegible and unmaintainable code?__
27
28
29 Three techniques can make regular expressions maintainable
30 and understandable.
31
32
33 Comments Outside the Regex
34
35
36 Describe what you're doing and how you're doing it, using
37 normal Perl comments.
38
39
40 # turn the line into the first word, a colon, and the
41 # number of characters on the rest of the line
42 s/^(w+)(.*)/ lc($1) .
43
44
45 Comments Inside the Regex
46
47
48 The /x modifier causes whitespace to be ignored in
49 a regex pattern (except in a character class), and also
50 allows you to use normal comments there, too. As you can
51 imagine, whitespace and comments help a lot.
52
53
54 /x lets you turn this:
55
56
57 s{
58 into this:
59
60
61 s{
62 It's still not quite so clear as prose, but it is very useful for describing the meaning of each part of the pattern.
63
64
65 Different Delimiters
66
67
68 While we normally think of patterns as being delimited with
69 / characters, they can be delimited by almost any
70 character. perlre describes this. For example, the
71 s/// above uses braces as delimiters. Selecting
72 another delimiter can avoid quoting the delimiter within the
73 pattern:
74
75
76 s//usr/local//usr/share/g; # bad delimiter choice
77 s#/usr/local#/usr/share#g; # better
78
79
80 __I'm having trouble matching over more than one line.
81 What's wrong?__
82
83
84 Either you don't have more than one line in the string
85 you're looking at (probably), or else you aren't using the
86 correct modifier(s) on your pattern (possibly).
87
88
89 There are many ways to get multiline data into a string. If
90 you want it to happen automatically while reading input,
91 you'll want to set $/ (probably to '' for paragraphs or
92 undef for the whole file) to allow you to read more
93 than one line at a time.
94
95
96 Read perlre to help you decide which of /s and
97 /m (or both) you might want to use: /s
98 allows dot to include newline, and /m allows caret
99 and dollar to match next to a newline, not just at the end
100 of the string. You do need to make sure that you've actually
101 got a multiline string in there.
102
103
104 For example, this program detects duplicate words, even when
105 they span line breaks (but not paragraph ones). For this
106 example, we don't need /s because we aren't using
107 dot in a regular expression that we want to cross line
108 boundaries. Neither do we need /m because we aren't
109 wanting caret or dollar to match at any point inside the
110 record next to newlines. But it's imperative that $/ be set
111 to something other than the default, or else we won't
112 actually ever have a multiline record read in.
113
114
115 $/ = ''; # read in more whole paragraph, not just one line
116 while (
117 Here's code that finds sentences that begin with ``From '' (which would be mangled by many mailers):
118
119
120 $/ = ''; # read in more whole paragraph, not just one line
121 while (
122 Here's code that finds everything between START and END in a paragraph:
123
124
125 undef $/; # read in whole file, not just one line or paragraph
126 while (
127
128
129 __How can I pull out lines between two patterns that are
130 themselves on different lines?__
131
132
133 You can use Perl's somewhat exotic .. operator
134 (documented in perlop):
135
136
137 perl -ne 'print if /START/ .. /END/' file1 file2 ...
138 If you wanted text and not lines, you would use
139
140
141 perl -0777 -ne 'print
142 But if you want nested occurrences of START through END, you'll run up against the problem described in the question in this section on matching balanced text.
143
144
145 Here's another example of using ..:
146
147
148 while (
149
150
151 __I put a regular expression into $/ but it didn't work.
152 What's wrong?__
153
154
155 $/ must be a string, not a regular expression. Awk has to be
156 better for something. :-)
157
158
159 Actually, you could do this if you don't mind reading the
160 whole file into memory:
161
162
163 undef $/;
164 @records = split /your_pattern/,
165 The Net::Telnet module (available from CPAN ) has the capability to wait for a pattern in the input stream, or timeout if it doesn't appear within a certain time.
166
167
168 ## Create a file with three lines.
169 open FH,
170 ## Get a read/write filehandle to it.
2 perry 171 $fh = new !FileHandle
1 perry 172 ## Attach it to a
173 ## Search for the second line and print out the third.
174 $file-
175
176
177 __How do I substitute case insensitively on the
178 LHS while preserving case on the
179 RHS ?__
180
181
182 Here's a lovely Perlish solution by Larry Rosler. It
183 exploits properties of bitwise xor on ASCII
184 strings.
185
186
187 $_=
188 $old = 'test';
189 $new = 'success';
190 s{(Q$oldE)}
191 { uc $new (uc $1 ^ $1) .
192 (uc(substr $1, -1) ^ substr $1, -1) x
193 (length($new) - length $1)
194 }egi;
195 print;
196 And here it is as a subroutine, modelled after the above:
197
198
199 sub preserve_case($$) {
200 my ($old, $new) = @_;
201 my $mask = uc $old ^ $old;
202 uc $new $mask .
203 substr($mask, -1) x (length($new) - length($old))
204 }
205 $a =
206 This prints:
207
208
209 this is a SUcCESS case
210 Just to show that C programmers can write C in any programming language, if you prefer a more C-like solution, the following script makes the substitution have the same case, letter by letter, as the original. (It also happens to run about 240% slower than the Perlish solution runs.) If the substitution has more characters than the string being substituted, the case of the last character is used for the rest of the substitution.
211
212
213 # Original by Nathan Torkington, massaged by Jeffrey Friedl
214 #
215 sub preserve_case($$)
216 {
217 my ($old, $new) = @_;
218 my ($state) = 0; # 0 = no change; 1 = lc; 2 = uc
219 my ($i, $oldlen, $newlen, $c) = (0, length($old), length($new));
220 my ($len) = $oldlen
221 for ($i = 0; $i
222
223
224 __How can I make__ w __match national character
225 sets?__
226
227
228 See perllocale.
229
230
231 __How can I match a locale-smart version of__
232 /[[a-zA-Z]/__?__
233
234
235 One alphabetic character would be /[[^Wd_]/, no
236 matter what locale you're in. Non-alphabetics would be
237 /[[Wd_]/ (assuming you don't consider an underscore
238 a letter).
239
240
241 __How can I quote a variable to use in a
242 regex?__
243
244
245 The Perl parser will expand $variable and
246 @variable references in regular expressions unless
247 the delimiter is a single quote. Remember, too, that the
248 right-hand side of a s/// substitution is
249 considered a double-quoted string (see perlop for more
250 details). Remember also that any regex special characters
251 will be acted on unless you precede the substitution with Q.
252 Here's an example:
253
254
255 $string =
256 $string =~ s/Q$lhs/$rhs/;
257 # $string is now
258 Without the Q, the regex would also spuriously match ``di''.
259
260
261 __What is__ /o __really for?__
262
263
264 Using a variable in a regular expression match forces a
265 re-evaluation (and perhaps recompilation) each time the
266 regular expression is encountered. The /o modifier
267 locks in the regex the first time it's used. This always
268 happens in a constant regular expression, and in fact, the
269 pattern was compiled into the internal format at the same
270 time your entire program was.
271
272
273 Use of /o is irrelevant unless variable
274 interpolation is used in the pattern, and if so, the regex
275 engine will neither know nor care whether the variables
276 change after the pattern is evaluated the ''very first''
277 time.
278
279
280 /o is often used to gain an extra measure of
281 efficiency by not performing subsequent evaluations when you
282 know it won't matter (because you know the variables won't
283 change), or more rarely, when you don't want the regex to
284 notice if they do.
285
286
287 For example, here's a ``paragrep'' program:
288
289
290 $/ = ''; # paragraph mode
291 $pat = shift;
292 while (
293
294
295 __How do I use a regular expression to strip C style
296 comments from a file?__
297
298
299 While this actually can be done, it's much harder than you'd
300 think. For example, this one-liner
301
302
303 perl -0777 -pe 's{/*.*?*/}{}gs' foo.c
304 will work in many but not all cases. You see, it's too simple-minded for certain kinds of C programs, in particular, those with what appear to be comments in quoted strings. For that, you'd need something like this, created by Jeffrey Friedl and later modified by Fred Curtis.
305
306
307 $/ = undef;
308 $_ =
309 This could, of course, be more legibly written with the /x modifier, adding whitespace and comments. Here it is expanded, courtesy of Fred Curtis.
310
311
312 s{
313 /* ## Start of /* ... */ comment
314 [[^*]**+ ## Non-* followed by 1-or-more *'s
315 (
316 [[^/*][[^*]**+
317 )* ## 0-or-more things which don't start with /
318 ## but do end with '*'
319 / ## End of /* ... */ comment
320 ## OR various things which aren't comments:
321 (
322 ## OR
323 ' ## Start of ' ... ' string
324 (
325 \. ## Escaped char
326 ## OR
327 [[^'\] ## Non '\
328 )*
329 ' ## End of ' ... ' string
330 ## OR
331 . ## Anything other char
332 [[^/
333 A slight modification also removes C ++ comments:
334
335
336 s#/*[[^*]**+([[^/*][[^*]**+)*///[[^n]*(
337
338
339 __Can I use Perl regular expressions to match balanced
340 text?__
341
342
343 Although Perl regular expressions are more powerful than
344 ``mathematical'' regular expressions because they feature
345 conveniences like backreferences (1 and its ilk),
346 they still aren't powerful enough--with the possible
347 exception of bizarre and experimental features in the
348 development-track releases of Perl. You still need to use
349 non-regex techniques to parse balanced text, such as the
350 text enclosed between matching parentheses or braces, for
351 example.
352
353
354 An elaborate subroutine (for 7-bit ASCII
355 only) to pull out balanced and possibly nested single chars,
356 like ` and ', { and },
357 or ( and ) can be found in
358 http://www.perl.com/CPAN/authors/id/TOMC/scripts/pull_quotes.gz
359 .
360
361
362 The C::Scan module from CPAN contains such
363 subs for internal use, but they are
364 undocumented.
365
366
367 __What does it mean that regexes are greedy? How can I get
368 around it?__
369
370
371 Most people mean that greedy regexes match as much as they
372 can. Technically speaking, it's actually the quantifiers
373 (?, *, +, {}) that are
374 greedy rather than the whole pattern; Perl prefers local
375 greed and immediate gratification to overall greed. To get
376 non-greedy versions of the same quantifiers, use
377 (??, *?, +?,
378 {}?).
379
380
381 An example:
382
383
384 $s1 = $s2 =
385 Notice how the second substitution stopped matching as soon as it encountered ``y ''. The *? quantifier effectively tells the regular expression engine to find a match as quickly as possible and pass control on to whatever is next in line, like you would if you were playing hot potato.
386
387
388 __How do I process each word on each line?__
389
390
391 Use the split function:
392
393
394 while (
395 Note that this isn't really a word in the English sense; it's just chunks of consecutive non-whitespace characters.
396
397
398 To work with only alphanumeric sequences (including
399 underscores), you might consider
400
401
402 while (
403
404
405 __How can I print out a word-frequency or line-frequency
406 summary?__
407
408
409 To do this, you have to parse out each word in the input
410 stream. We'll pretend that by word you mean chunk of
411 alphabetics, hyphens, or apostrophes, rather than the
412 non-whitespace chunk idea of a word given in the previous
413 question:
414
415
416 while (
417 If you wanted to do the same thing for lines, you wouldn't need a regular expression:
418
419
420 while (
421 If you want these output in a sorted order, see perlfaq4: ``How do I sort a hash (optionally by value instead of key)?''.
422
423
424 __How can I do approximate matching?__
425
426
427 See the module String::Approx available from
428 CPAN .
429
430
431 __How do I efficiently match many regular expressions at
432 once?__
433
434
435 The following is extremely inefficient:
436
437
438 # slow but obvious way
439 @popstates = qw(CO ON MI WI MN);
440 while (defined($line =
441 That's because Perl has to recompile all those patterns for each of the lines of the file. As of the 5.005 release, there's a much better approach, one which makes use of the new qr// operator:
442
443
444 # use spiffy new qr// operator, with /i flag even
445 use 5.005;
446 @popstates = qw(CO ON MI WI MN);
447 @poppats = map { qr/b$_b/i } @popstates;
448 while (defined($line =
449
450
451 __Why don't word-boundary searches with__ b
452 __work for me?__
453
454
455 Two common misconceptions are that b is a synonym
456 for s+ and that it's the edge between whitespace
457 characters and non-whitespace characters. Neither is
458 correct. b is the place between a w
459 character and a W character (that is, b is
460 the edge of a ``word''). It's a zero-width assertion, just
461 like ^, $, and all the other anchors, so
462 it doesn't consume any characters. perlre describes the
463 behavior of all the regex metacharacters.
464
465
466 Here are examples of the incorrect application of
467 b, with fixes:
468
469
470
471
472 Although they may not do what you thought they did, b and B can still be quite useful. For an example of the correct use of b, see the example of matching duplicate words over multiple lines.
473
474
475 An example of using B is the pattern BisB.
476 This will find occurrences of ``is'' on the insides of words
477 only, as in ``thistle'', but not ``this'' or
478 ``island''.
479
480
481 __Why does using $
482 __
483
484
485 Once Perl sees that you need one of these variables anywhere
486 in the program, it provides them on each and every pattern
487 match. The same mechanism that handles these provides for
488 the use of $1, $2, etc., so you pay the
489 same price for each regex that contains capturing
490 parentheses. If you never use $
491 without'' capturing parentheses won't be
492 penalized. So avoid $
493 ''
494
495
496 __What good is__ G __in a regular
497 expression?__
498
499
500 The notation G is used in a match or substitution
501 in conjunction with the /g modifier to anchor the
502 regular expression to the point just past where the last
503 match occurred, i.e. the ''pos()'' point. A failed match
504 resets the position of G unless the /c
505 modifier is in effect. G can be used in a match
506 without the /g modifier; it acts the same (i.e.
507 still anchors at the ''pos()'' point) but of course only
508 matches once and does not update ''pos()'', as
509 non-/g expressions never do. G in an
510 expression applied to a target string that has never been
511 matched against a /g expression before or has had
512 its ''pos()'' reset is functionally equivalent to
513 A, which matches at the beginning of the
514 string.
515
516
517 For example, suppose you had a line of text quoted in
518 standard mail and Usenet notation, (that is, with leading
519 characters), and you want change each leading
520 into a corresponding :. You could do
521 so in this way:
522
523
524 s/^(
525 Or, using G, the much simpler (and faster):
526
527
528 s/G
529 A more sophisticated use might involve a tokenizer. The following lex-like example is courtesy of Jeffrey Friedl. It did not work in 5.003 due to bugs in that release, but does work in 5.004 or better. (Note the use of /c, which prevents a failed match with /g from resetting the search position back to the beginning of the string.)
530
531
532 while (
533 Of course, that could have been written as
534
535
536 while (
537 but then you lose the vertical alignment of the regular expressions.
538
539
540 __Are Perl regexes DFAs or NFAs? Are they
541 POSIX compliant?__
542
543
544 While it's true that Perl's regular expressions resemble the
545 DFAs (deterministic finite automata) of the egrep(1)
546 program, they are in fact implemented as NFAs
547 (non-deterministic finite automata) to allow backtracking
548 and backreferencing. And they aren't POSIX-style either,
549 because those guarantee worst-case behavior for all cases.
550 (It seems that some people prefer guarantees of consistency,
551 even when what's guaranteed is slowness.) See the book
552 ``Mastering Regular Expressions'' (from O'Reilly) by Jeffrey
553 Friedl for all the details you could ever hope to know on
554 these matters (a full citation appears in
555 perlfaq2).
556
557
558 __What's wrong with using grep or map in a void
559 context?__
560
561
562 Both grep and map build a return list, regardless of their
563 context. This means you're making Perl go to the trouble of
564 building up a return list that you then just ignore. That's
565 no way to treat a programming language, you insensitive
566 scoundrel!
567
568
569 __How can I match strings with multibyte
570 characters?__
571
572
573 This is hard, and there's no good way. Perl does not
574 directly support wide characters. It pretends that a byte
575 and a character are synonymous. The following set of
576 approaches was offered by Jeffrey Friedl, whose article in
577 issue #5 of The Perl Journal talks about this very
578 matter.
579
580
581 Let's suppose you have some weird Martian encoding where
582 pairs of ASCII uppercase letters encode
583 single Martian letters (i.e. the two bytes ``
584 CV '' make a single Martian letter, as do the
585 two bytes `` SG '', `` VS '',
586 `` XX '', etc.). Other bytes represent single
587 characters, just like ASCII .
588
589
590 So, the string of Martian ``I am CVSGXX !''
591 uses 12 bytes to encode the nine characters 'I', ' ', 'a',
592 'm', ' ', ' CV ', ' SG ', '
593 XX ', '!'.
594
595
596 Now, say you want to search for the single character
597 /GX/. Perl doesn't know about Martian, so it'll
598 find the two bytes `` GX '' in the ``I am
599 CVSGXX !'' string, even though that character
600 isn't there: it just looks like it is because ``
601 SG '' is next to `` XX '', but
602 there's no real `` GX ''. This is a big
603 problem.
604
605
606 Here are a few ways, all painful, to deal with
607 it:
608
609
610 $martian =~ s/([[A-Z][[A-Z])/ $1 /g; # Make sure adjacent ``martian'' bytes
611 # are no longer adjacent.
612 print
613 Or like this:
614
615
616 @chars = $martian =~ m/([[A-Z][[A-Z][[^A-Z])/g;
617 # above is conceptually similar to: @chars = $text =~ m/(.)/g;
618 #
619 foreach $char (@chars) {
620 print
621 Or like this:
622
623
624 while ($martian =~ m/G([[A-Z][[A-Z].)/gs) { # G probably unneeded
625 print
626 Or like this:
627
628
629 die
630 There are many double- (and multi-) byte encodings commonly used these days. Some versions of these have 1-, 2-, 3-, and 4-byte characters, all mixed.
631
632
633 __How do I match a pattern that is supplied by the
634 user?__
635
636
637 Well, if it's really a pattern, then just use
638
639
640 chomp($pattern =
641 Alternatively, since you have no guarantee that your user entered a valid regular expression, trap the exception this way:
642
643
644 if (eval { $line =~ /$pattern/ }) { }
645 If all you really want to search for a string, not a pattern, then you should either use the ''index()'' function, which is made for string searching, or if you can't be disabused of using a pattern match on a non-pattern, then be sure to use Q...E, documented in perlre.
646
647
648 $pattern =
649 open (FILE, $input) or die
650 !!AUTHOR AND COPYRIGHT
651
652
653 Copyright (c) 1997-1999 Tom Christiansen and Nathan
654 Torkington. All rights reserved.
655
656
657 When included as part of the Standard Version of Perl, or as
658 part of its complete documentation whether printed or
659 otherwise, this work may be distributed only under the terms
660 of Perl's Artistic License. Any distribution of this file or
661 derivatives thereof ''outside'' of that package require
662 that special arrangements be made with copyright
663 holder.
664
665
666 Irrespective of its distribution, all code examples in this
667 file are hereby placed into the public domain. You are
668 permitted and encouraged to use this code in your own
669 programs for fun or for profit as you see fit. A simple
670 comment in the code giving credit would be courteous but is
671 not required.
672 ----
This page is a man page (or other imported legacy content). We are unable to automatically determine the license status of this page.