Code Examples
Download the strtok-v2-1.e zip file here.
Download the older strtok-v2.e zip file here.
Back to my Euphoria page here.
** parse()**
parse(string,separator)
Parse will break up a string, breaking on separators you supply. This is most handy for determining words in a sentence and fields in a database. New in v2.1 is "k" for keeping the separators in the parsed nested sequence, and "i" for case insensitivity in comparing alphabetic strings (see the last 2 examples). You can use "k" and "i" at the same time. As for all the token handling routines which parsing is involved, you can supply sequences of separators, shown in examples below. The separator can be an integer, an atom, or a sequence, as you like.
parse("this is a test",32) = {"this","is","a","test"}
parse("123.54.678.987",'.') = {"123","54","678","987"}
parse("this,is.a?test here"," ") = {"this,is.a?test","here"}
parse("www.pell.net/warning/euphoria.html",'/') = {"www.pell.net","warning","euphoria.html"}
parse("this,is.a?test here",",.?") = {"this","is","a","test here"}
parse("nick!ident@a.net","!@") = {"nick","ident","a.net}
parse("nick!ident@a.net",{"k","!@"}) = {"nick","!","ident","@","a.net"}
parse("AbCdEF",{"i",'c'}) = {"Ab","dEF"}
Click here for parse() example code.
** deparse() **
deparse(string,separator)
The reverse of parse(). You can supply only one separator though.
deparse({"this","is","a","test"},'.') = "this.is.a.test"
deparse({"this","is","a","test"},".") = "this.is.a.test"
deparse({"this","is","a","test"},' ') = "this is a test"
Click here for deparse() example code.
** gettok() **
gettok(string,n,separator)
Gets the nth token from the string.
gettok("this is a test",1,32) = "this"
gettok("a.b.c.d.e",3,'.') = "c"
gettok("a b c d e",3,' ') = "c"
gettok("a b c d e",3,32) = "c"
gettok("a.b.c.d.e",9,'.') = ""
gettok({"a","b","c","d","e"},3,"") = "c"
gettok({"a","b","c","d","e"},9,"") = ""
gettok("a.b.c.d.e",{2},'.') = "b.c.d.e" (2nd token onwards)
gettok("a.b.c.d.e",{2,4},'.') = "b.c.d" (tokens 2 thru 4)
gettok("a.b.c.d.e",{4,-4},'.') = "b.c.d" (tokens -4 thru 4)
gettok("a.b.c.d.e",-2,'.') = "d.e" (tokens -2 onwards)
gettok({"a","b","c","d","e"},{2},"") = {"b","c","d","e"} (2nd token onwards)
gettok({"a","b","c","d","e"},{2,4},"") = {"b","c","d"} (tokens 2 thru 4)
Here is an example using gettok in a real life example. Gettok is used to pick a specified item out of the string of months. Naturally, there are different ways to do this, this is just a demo of gettok:
-- url = "http://abcnews.go.com/wire/US/ap20020708_1594.html"
-- directory = "20020708_1594.html"
directory = directory[1..match("_",directory)-1] -- "20020708"
year = directory[1..4] -- "2002"
month = value(directory[5..6]) -- {1,"07"}
monthnum = month[2] -- 7
month = gettok("January February March April May June July August September October November December",monthnum,32) -- "July"
day = directory[7..8] -- "08"
directory = "H:\\" & year & "\\" & month & "\\" & day -- "H:\2002\July\08"
Here is another example, using a real-life example from irc. The point is to use the bold-typed part of this user mask: nick!ident@123.45.67.987 (which (usually) cannot change while connected to the irc server) in a channel flood control script. The script will ban and kick the user who sends a lot of text to a channel. This script is mirc, and there are plenty of other ways to do this.....
on @*:TEXT:*:#:{
set -u0 %mask $gettok($address($nick,5),2,$asc(!))
inc -z20 %flood. [ $+ [ # ] $+ . $+ [ %mask ] ] 2
if ( %flood. [ $+ [ # ] $+ . $+ [ %mask ] ] > 3 ) {
.ban -u120 $chan $nick 3
.timer 1 1 kick $chan $nick Flooding Is Not Allowed In $chan !!!
}
}
** findtok() **
findtok(string,token,n,separator)
Returns the position of the nth equal token in string. If you specify zero for n, it returns the total number of equal tokens. If you provide "i" as the first element in {n}, the match is done case-insensitively (see last three examples right below), otherwise this function is case-sensitive. See also matchntok().
findtok("a.b.c.d","c",1,'.') = 3
findtok("a.b.c.d","e",1,'.') = 0
findtok({"a","b","c","d"},"c",1,"") = 3
findtok({"a","b","c","d"},"e",1,"") = 0
findtok("c.a.c.b.c.d.c",'c',-1,'.') = 1
findtok("c.a.c.b.c.d.c",'c',{1,-1},'.') = {1,7}
findtok("c.a.c.b.c.d.c",'c',{-1,1},'.') = {7,1}
string = "a1 cat1 ba Cat1 da tac2"
findtok(string,"Cat1",{1},32) = 4
findtok(string,"Cat1",1,32) = 4
findtok(string,"Cat1",{"i",1},32) = 2
** revtok()**
revtok(string,separator)
Reverses the tokens in a string, not the characters in the string.
revtok("this is a test",32) = "test a is this"
revtok("123.45.678.3",".") = "3.678.45.123"
revtok({"abc","def"}) = {"def","abc"}
** numtok()**
numtok(string,separator)
Returns the number of tokens in string.
numtok("this is a test",32) = 4
numtok({"this","is","a","test"},"") = 4
** remtok()**
remtok(string,token,n,separator)
Removes the nth matching token from string. If you provide "i" as the first element in {n}, the match is done case-insensitively (see last three examples right below), otherwise this function is case-sensitive.
remtok("a.b.c.d","b",1,'.') = "a.c.d"
remtok("a.b.c.d","e",1,'.') = "a.b.c.d"
remtok("a.c.c.d","c",1,'.') = "a.c.d"
remtok("a.b.a.d","a",2,'.') = "a.b.d"
remtok("a c b c k c d c",'c',{3,-4}," ") = "a b c k d c"
remtok("a c b c k c d c",'c',{3,-4},' ') = "a b c k d c"
remtok("a c b c k c d c",'c',{-4,3},' ') = "a b c k d c"
remtok("a c b c k c d c","c",{-4,3},32) = "a b c k d c"
remtok({"a","c","b","c","k","c","d","c}",'c',{-4,3},"") = {"a","b","c","k","d","c"}
string = "a.A.b.a.A.c"
remtok(string,"a",2,'.') = "a.A.b.A.c" -- 2nd 'a' is gone
remtok(string,"a",{"i",2},'.') = a.b.a.A.c -- 2nd 'a' or 'A' is gone
remtok(string,"a",-2,'.') = "A.b.a.A.c" -- 2nd 'a' from right is gone
remtok(string,"a",{"i",2,-2},'.') = "a.b.A.c" -- 2nd 'a' or 'A' is gone from each end
** deltok()**
deltok(string,n,separator)
Removes the nth token from the string.
deltok("a1 cat1 ba tac1",1,32) = "cat1 ba tac1"
deltok("a1 cat1 ba tac1",-1,32) = "a1 cat1 ba"
deltok("a1 cat1 ba tac1",{1,-1}," ") = "a1 cat1 ba"
** addtok()**
addtok(string,token,separator)
Adds a token to the end of string but only if it's not already in string. It's useful for a sequence storing randomly ordered chunks of data. If you provide "i" as the first element in {token}, the match is done case-insensitively (see last examples right below), otherwise this function is case-sensitive.
addtok("a.b.c","d",'.') = "a.b.c.d"
addtok("a.b.c.d","c",'.') = "a.b.c.d"
addtok({"a","b","c"},"d","") = {"a","b","c","d"}
addtok({"a","b","c","d"},"c","") = {"a","b","c","d"}
string = "a1 cat1 ba"
addtok(string,"Cat1",32) = "a1 cat1 ba Cat1" -- case sensitive
addtok(string,{"i","Cat1"},32) = "a1 cat1 ba" -- case insensitive
** instok()**
instok(string,token,n,separator)
Inserts a token into string and the nth location. The item at the nth location will be bumped to the right.
instok("a.b.d","c",3,'.') = "a.b.c.d" <-- note "d" got bumped!
instok("a.b.d","c",-1,'.') = "a.b.c.d" <-- note "d" got bumped!
instok("a.b.d","c",9,'.') = "a.b.d.c"
instok("a.b.d","c",{2,3},'.') = "a.c.c.b.d"
instok({"a","b","d"},"c",3,"") = {"a","b","c","d"}
instok({"a","b","d"},"c",9,"") = {"a","b","d","c"}
instok({"a","b","d"},"c",{2,3,9},"") = {"a","c","c","b","d","c"}
** puttok()**
** changetok()**
puttok(string,token,n,separator)
Overwrites the nth token in string with a new token.
puttok("a.b.c.d","e",2,'.') = "a.e.c.d"
puttok({"a","b","c","d"},"e",2,"") = {"a","e","c","d"}
puttok("a1 cat1 ba cat2 da tac2","at",{-1,2},32) = "a1 at ba cat2 da at"
** reptok()**
** repltok()**
repltok(string,token1,token2,n,separator)
Different names, same great function! Replaces the nth matching token1 in the string with new token2. If you provide "i" as the first element in {n}, the match is done case-insensitively (see last two examples right below), otherwise this function is case-sensitive.
reptok("a.b.c.d","b","e",1,'.') = "a.e.c.d"
reptok("a.b.c.d","f","e",1,'.') = "a.b.c.d"
reptok("a.b.a.c","a","e",2,'.') = "a.b.e.c"
reptok("a.b.a.c","a","e",{1,2},'.') = "e.b.e.c"
reptok("a.b.a.c","a","e",{-2,-4},".") = "e.b.e.c"
reptok({"a","b","a","c"},"a","e",2,"") = {"a","b","e","c"}
string = "a.A.b.a.A.c"
reptok(string,"a","e",2,'.') = "a.A.b.e.A.c"
reptok(string,"a","e",{"i",2},'.') = "a.e.b.a.A.c"
** istok()**
istok(string,token,separator)
Checks to see if token exists in string. If you provide "i" as the first element in {token}, the match is done case-insensitively (see the last example right below), otherwise this function is case-sensitive.
istok("a.b.c.d","c",'.') = 1 (TRUE)
istok("a.b.c.d","e",'.') = 0 (FALSE)
istok({"a","b","c","d"},"c","") = 1 (TRUE)
istok({"a","b","c","d"},"e","") = 0 (FALSE)
istok("a.b.C.d",{"i","c"},'.') = 1 (TRUE)
example
** matchtok()**
matchtok(string,token,n,separator)
Returns tokens that contain the specified token value. Like wildtok() using *token*. If you supply 0 for n, it returns the total number of matching tokens. If you provide "i" as the first element in {n}, the match is done case-insensitively (see last example right below), otherwise this function is case-sensitive. If you want the indexes of the match, see matchntok().
matchtok("one two three","e",0,' ') = 2
matchtok("one two three","e",2," ") = "three"
matchtok("one.two.three","e",-1,".") = "three"
matchtok({"one","two","three"},"e",0,"") = 2
matchtok({"one","two","three"},"e",2,"") = "three"
matchtok("a1 cat1 ba cat2 da tac2","at",{-1,2},32) = {"cat2","cat2"}
matchtok("a1 cat1 ba cat2 da tac2","AT",{"i",2},32) = {"cat2"}
** matchntok()**
matchntok(string,token,n,separator)
Returns index(es) that contain the specified token value. It's like wildntok() using *token*. If you supply 0 for n, it returns all the indexes of matching tokens. Single matches are returned as integers, multiple matches are returned as a sequence of integers. If you provide "i" as the first element in {n} (for example: {"i",2,3}), the match is done case-insensitively (see last example right below), otherwise this function is case-sensitive.
matchntok("one two three","e",0,' ') = {1,3}
matchntok("one two three","e",2," ") = 3
matchntok("one.two.three","e",-1,".") = 3
matchntok({"one","two","three"},"e",0,"") = 2
matchntok({"one","two","three"},"e",2,"") = 3
matchntok("a1 cat1 ba cat2 da tac2","at",{-1,2},32) = {4,4}
matchntok("a1 cat1 ba cat2 da tac2","AT",{"i",-1,2},32) = {4,4}
** wildtok()**
wildtok(string,token,n,separator)
A more controlled matchtok(). Wildtok no longer matches abc if you ask for *a*, like it did in strtok v1. Note the difference between *a and a* and *a* in the examples. If you specify 0 for n, it returns the total number of matching tokens. If you provide "i" as the first element in {n}, the match is done case-insensitively (see last example right below), otherwise this function is case-sensitive.
given: string = "a1 cat1 ba tac1 k a2 cat2 da tac2"
wildtok(string,"*a*",2," ") = "tac1" -- ("cat1" is wildmatch 1)
wildtok(string,"*a",2," ") = "da" -- ("ba" is wildmatch 1)
wildtok(string,"a*",2," ") = "a2" -- ("a1" is wildmatch 1)
wildtok(string,"a*",{1,2}," ") = {"a1","a2"} -- (match 1 and 2, in that order)
wildtok(string,"a*",{-1,-2}," ") = {"a2","a1"} -- (match -1 and -2, in that order)
wildtok(string,"A*",{"i",-1,-2}," ") = {"a2","a1"} -- (match -1 and -2, in that order)
wildtok("tesT1 test4","t*t*",1," ") = "test4"
** wildntok()**
wildntok(string,token,n,separator)
A more controlled matchntok(). There wasn't a wildntok in strtok v1, but note the difference between *a and a* and *a* in the examples. If you specify 0 for n, it returns all the indexes of matching tokens. Single matches are returned as integers, multiple matches are returned as a sequence of integers. If you provide "i" as the first element in {n} (for example: {"i",2,3}), the match is done case-insensitively (see last example right below), otherwise this function is case-sensitive.
given: string = "a1 cat1 ba tac1 k a2 cat2 da tac2"
wildntok(string,"*a*",2," ") = 4
wildntok(string,"*a",2," ") = 8
wildntok(string,"a*",2," ") = 6
wildntok(string,"a*",{1,2}," ") = {1,6} -- (match 1 and 2, in that order)
wildntok(string,"a*",{-1,-2}," ") = {6,1} -- (match -1 and -2, in that order)
wildntok(string,"A*",{"i",-1,-2}," ") = {6,1} -- (match insensitively -1 and -2, in that order)
wildtok("tesT1 test4","t*t*",1," ") = 2
** sorttok()**
sorttok(string,n,separator)
This will sort the tokens in string according to n. This is a great lil function for mungeing small comma-delimited or semicolon-delimited or whatever-delimited databases. Three examples of using a comma-delimited file, typical of a SOHO gui data, or client list are below.
normal sort("this is a test") = " aehiisssttt"
sorttok("this is a test",0,32) = "a is test this"
Of course, the normal sort isn't useful at all, and that example of sorttok() doesn't produce anything useful either, but look at what happens if you give it a comma delimited string like this:
string = {"Fred,Astaire,45,Alberta,Canada",
"Lucy,Ball,56,Barre,North Dakota",
"Red,Skelton,34,London,England",
"Pierre,du Pont,12,Paris,France"}
string = sorttok(string,3,',')
string = {"Pierre,du Pont,12,Paris,France",
"Red,Skelton,34,London,England",
"Fred,Astaire,45,Alberta,Canada",
"Lucy,Ball,56,Barre,North Dakota"}
Cool, eh? It was sorted on the (fictitious) ages. If i had sorted on -3, the order would be reversed. Lets toss it a curve:
string = {"Fred,Astaire,45,Alberta,Canada
"Lucy,Ball,56,Barre,North Dakota",
"Red,Skelton,45,London,England",
"Pierre,du Pont,56,Paris,France"}
string = sorttok(string,{3,2},',')
string = {"Fred,Astaire,45,Alberta,Canada",
"Red,Skelton,45,London,England",
"Lucy,Ball,56,Barre,North Dakota",
"Pierre,du Pont,56,Paris,France"}
Now it sorted on (fictitious) age first (the 3), and then within the age group it sorted on last name (the 2). You can specify as many of these sort field numbers as you have fields! Now lets really mix it up:
string = {"Red,Skelton,45,London,England",
"Lucy,Ball,56,Barre,North Dakota",
"Fred,Astaire,45,Alberta,Canada",
"Pierre,du Pont,56,Paris,France"}
string = sorttok(string,{3,-2},',')
string = {"Red,Skelton,45,London,England",
"Fred,Astaire,45,Alberta,Canada",
"Pierre,du Pont,56,Paris,France",
"Lucy,Ball,56,Barre,North Dakota"}
You see, it did a ascending sort on (fictitious) age field, then sorted the two last name fields in decending order.
If two lines are identical, they will be returned in the order they appeared in the list. If sorted by sorttok(string,0,','), sorttok will do a normal Euphoria sort, sorting on a line-vs-line basis, starting with the first character of the first token, then the next character, etc.. If you supply a sort pattern that is longer than the number of tokens on a line in string, sorttok() will return "", since while you can sort the same things over and over, why would you throw away cpu time like that?
** sortntok()**
sortntok(string,n,separator)
This will sort the tokens in string according to n, exactly like sorttok() above, but this function returns the line numbers in sorted order, not the string itself:
string = {"Red,Skelton,45,London,England",
"Lucy,Ball,56,Barre,North Dakota",
"Fred,Astaire,45,Alberta,Canada",
"Pierre,du Pont,56,Paris,France"}
string = sortntok(string,{3,-2},',')
string = {1,3,4,2}
** striptok()**
striptok(string,n,separator)
This function is to strip leading or trailing tokens off of a string. You can specify an atom for n or a sequence of two atoms. The count you want removed must be less than the length of the string, or you logically will get nothing back. As usual, you can pass a pre-parsed string to this function.
If you specify using a sequence, there must be two atoms in it, the first atom is the number of leading tokens to strip, the second atom is the number of trailing tokens to strip. The atoms can be negative, but they will be treated as if positive, and used as specified by their place in the sequence, not their sign.
If you specify an atom, it can be positive or negative. A positive one will strip off the leading tokens, and a negative one will strip off the trailing tokens.
striptok("a b c d e",2," ") = "c d e" (first 2 tokens removed)
striptok("a b c d e",-2," ") = "a b c" (last 2 tokens removed)
striptok("a b c d e",{0,-2}," ") = "a b c" (last 2 tokens removed)
striptok("a b c d e",{2,2}," ") = "c" (first and last 2 tokens removed)
striptok("a b c d e",{2,-2}," ") = "c" (first and last 2 tokens removed)
Code Examples
parse() and deparse() examples:
In one news mining program i wrote, i discovered some people title their pages with lines that exceed what my screen width is, and i prefer a shorter title. Short of an Ai deciding what to title the page, the best thing to do is to trim the title down to size, based on word boundries, without chopping words in half to fit into a 80 character limit i set. Well, here is a short lil program which does that:
--== shorten the title on word boundries ==--
if length(title) > 80 then -- if title is too long
parsedtitle = parse(title,32) -- break it on word boundries
titlelen = length(parsedtitle) -- how many words are there?
while length(title) > 80 do -- while title is too long
titlelen -= 1 -- "forget" the last word
title = deparse(parsedtitle[1..titlelen],32) -- drop that last word from title
end while -- until length(title) < 80
end if -- presto!, it's less than 80 chars long, and no words are broken!
Here is a quicky example i use many times a day, using parse and deparse:
-- create_directory("H:\2002\May\23\abc")
procedure create_directory(sequence where)
sequence parseddir, directory
parseddir = parse(where,'\\') -- parsedir = {"H:","2002","May","23","abc"}
for loop = 2 to length(parseddir) do -- start at 2 because we can't make a partition!
directory = deparse(parseddir[1..loop],'\\')
-- if loop = 2 then directory = "H:\2002"
-- if loop = 3 then directory = "H:\2002\23"
-- if loop = 4 then directory = "H:\2002\23\abc"
-- etc
if atom(dir(directory)) then -- if an atom, it's -1, an error, it's not there
system("mkdir " & directory,0) -- so make it for me! I want it!
end if
end for -- loop up, look at the next subdir
end procedure -- all done, it's made!
--------------------------------------------------------------------------------
Revision history:
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
v2.1 - December 17, 2002
Kat Added:
+ striptok() to strip leading and/or trailing tokens
+ "k" to keep the separators in the return in parse()
+ "i" to parse case insensitively in parse()
- all include files
Fixed:
An odd bug in wildcard_match_all2
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
v2
Thanks To:
Andy Serpas for new parse()
made much faster than the original parse()
-- kat modified to support sequences of delimiters
Jordah Ferguson for new deparse()
made faster
Kat Added:
+ ripped sort() from sort.e, and and reverse() from misc.e
(would you believe some people didn't have those includes?!)
+ case insenstitivity in match and find functions
+ a colorized html readme file
+ a text readme file
+ revtok("abc def",32) returns "def abc"
+ matchntok() returns the indexes, not the tokens
+ wildntok() returns the indexes, not the tokens
+ sorttok() returns the the list of lines, sorted
+ sortntok() returns the sort order, not the list
+ wildcard_match_all2() which stops * matching on nothing
+ aliased reptok() with repltok()
- deleted all private types,
all strings are sequences,
everything else is an object
* Modified:
# parse() Almost everything is based on parse(), now you do not need to pass a flat
sequence to the token functions, you can pass a nested sequence like {"a","b","c"}
now, allowing you to save time over reparsing every time you call a token function.
To signify the string is not to be reparsed, you set the separator in all token
functions to "".
# all functions:
* those which match or find tokens can be case-insensitive.
* now accept pre-parsed strings.
* now accept negative indexes, making indexing from the left of the string possible.
* may accept lists of indexes you want returned, mixed in any order.
* accept an atom or sequence or integer as the separator or delimiter.
* accept an atom or sequence or integer as the token.
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------
v1.0 created
original (c) 1999 Gabriel Boehme (gboehme@musicland.com)
Euphoria implementation of mIRC token functions plus some
Thanks to:
Kat for supplying the original idea,
with *excellent* function descriptions and demonstration examples
Hawke' for the find_all() routine
Did I forget you? Send me an e-mail and let me know!
-- or Kat! -- gertie @ pell -dot- net
--------------------------------------------------------------------------------