egghelp/eggheads community

sooo close. Need help with geturl and data functions

2007-09-08T13:59:13-04:00

well I got it to work. As I suspected, when you log in, it redirects you. So I had to grab the meta data for the url to the re-direction.

Full working code:

Code:

package require http####################################Profile Grabber####################################bind pub - !profile profileproc profile {nick host handle chan user} {    set url "http://forums.shooshtime.com/login.php?vb_login_username=zooshbot&vb_login_password=zoosh&s=&do=login&url=/member.php?username=$user"#putserv "PRIVMSG $chan :$url"set http [::http::geturl $url]#putserv "PRIVMSG $chan :$http"###################get meta info###################upvar #0 $http state        array set meta [set ${http}(meta)]        if {![info exist meta(Location)]} {           return $http        }        set redirect $meta(Location)        unset meta#putserv "PRIVMSG $chan :Redirecting to $redirect"#####################Set and cleanup#####################set newhttp [::http::geturl $redirect]set size [::http::size $newhttp]set code [::http::code $newhttp]set data [::http::data $newhttp]::http::cleanup $newhttp################Debug info#################putserv "PRIVMSG $chan :Size of html returned, $size"#putserv "PRIVMSG $chan :Html code returned, $code"#putserv "PRIVMSG $chan :Html data, $data"#regexp {To remove (.*?)you must} $data - notloggedin#regexp {has not registered and therefore (.*?) to view} $data - noprofile#putserv "PRIVMSG $chan :$notloggedin"#putserv "PRIVMSG $chan :$noprofile"#################Search Data#################regexp {Total Posts: (.*?)} $data - tpostsregexp {Join Date: (.*?)} $data - joinedregexp {Referrals: (.*?)} $data - referralsregexp {Birthday:
(.*?)} $data - bdayregexp {User Notes: (.*?)} $data - unotes############Output############putserv "PRIVMSG $chan :User Status for $user"putserv "PRIVMSG $chan :Joined on $joined"putserv "PRIVMSG $chan :$tposts Total Posts"putserv "PRIVMSG $chan :$referrals Total Referrals"putserv "PRIVMSG $chan :$unotes Total User Notes"putserv "PRIVMSG $chan :Was Born On $bday"}

Statistics: Posted by arcanedreams — Sat Sep 08, 2007 1:59 pm

sooo close. Need help with geturl and data functions

2007-09-08T12:58:49-04:00

There are newlines in the output, so you won't see them, which is why I suggested using foreach [split $html \n] to output each line.

Statistics: Posted by rosc2112 — Sat Sep 08, 2007 12:58 pm

sooo close. Need help with geturl and data functions

2007-09-07T02:15:49-04:00

Not sure how to view the putcmdlog, so I just used putlog instead.

only error that showed up was expected. It was an error stating that the $uname variable couldn't be found.

I had it set to spam the channel with all the variables in the entire script.

It will go something like this:

$query: http://forums.shooshtime.com/login.php? ... =&do=login

$http: ::http::5

Then it will stop there and never output the $html variable.

I have tried simply changing the url to just the standard http://forums.shooshitme, and the $html variable will show up like this:

$html: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

So it looks to me like when I use the url that I need to have the bot login to the site, it has trouble at the ::http::data part

If I had to guess I would say the http::data part isn't returning because after logging in, you are redirected to index.php, where the call in the url was to login.php

I also returned the value for ::http::size, and it comes back as 0
For the ::http::code it returns :HTTP/1.1 302 Found

Any idea why, or how to fix it?

Or perhaps an alternative way to get the bot to login to the site and grab info while its logged in?

Statistics: Posted by arcanedreams — Fri Sep 07, 2007 2:15 am

sooo close. Need help with geturl and data functions

2007-09-07T00:18:19-04:00

Best bet is to debug what you are getting in your data var, by looking at it with putcmdlog lines after getting the webpage, like:

Code:

set query "http://somewhere.com/$varwhatever"set page [::http::geturl $query]set html [::http::data $page] set count 0foreach line [split $html \n] {       incr count       putcmdlog "line$count '$line'"}

It's also helpful to [catch] the geturl errors so you can see them, for example (from one of the movie grabber scripts I did):

Code:

set movieurl "http://www.filmspot.com"catch {set page [::http::geturl $movieurl -timeout $::movietimeout]} errorif {[string match -nocase "*couldn't open socket*" $error]} {puthelp "PRIVMSG $chan :Error: couldn't connect to filmspot.com..Try again later."return}if { [::http::status $page] == "timeout" } {puthelp "PRIVMSG $chan :Error: Connection to filmspot.com timed out..Try again later."return}set html [::http::data $page]::http::cleanup $page        # Now we have data to work with and test the contents, etc..if {[regexp $regexm $html match moviedata]} {             # the $regexm var is set earlier in the script depending on              # which commandline was given to the script

Thats a pretty standard template for any url-grabber script.

Statistics: Posted by rosc2112 — Fri Sep 07, 2007 12:18 am

sooo close. Need help with geturl and data functions

2007-09-06T21:40:16-04:00

Here is what I have so far. Dont worry about the username and password..I just made those up for testing purposes.

Code:

bbind pub - !joined joinedproc joined {nick host handle chan text} {    set query "http://forums.shooshtime.com/login.php?vb_login_username=zooshbot&vb_login_password=zoosh&s=&do=login"set http [::http::geturl $query]set html [::http::data $http] regexp {Welcome, (.*?) 
} $html - unameputserv "PRIVMSG $chan :$uname"

That link will log you into the site.

From there, I can't get it to find any info on the page. I'm guessing it has something to do with the way it handles the link...perhaps it is taking the info from the actual login.php page

How do I get around this?

Statistics: Posted by arcanedreams — Thu Sep 06, 2007 9:40 pm