egghelp/eggheads community

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-16T05:16:21-04:00

hi nml!

thanks again for the sound advice...

just to iterate a little...

the "call to the 250 connections" only happens on a specific set of circumstances and I have been watching it "communicate" between the playout box and the eggie and have found this operation completes in under 2 seconds (at most) even if I unplug the box from the network.. the script won't hang.. thats good! but not where the problem was coming from!

I THINK the problem was coming from the search and request system- which did a simular thing- but this time it had to GET the results, and then PARSE the results and then display them in an IM to the party who init'ed it! The problem is that only 10 results would display per call (yes that had to all be parsed and displayed) and then another set of 10, and so on and so on... now here comes johnny-user.. who decides he wants to look up "tainted love" while suzie-user is currently searching for "right here waiting"... now johnny looks up "the" and receives a listing for EVERY song or artist with "the" in the string! being theres 3800+ tracks... OUCH!

Now, of course, I have put in a mean limit of 50 tracks max PER search query AND a time limit to wait between the searches.. well there was still 1000 lines just in the PERSERS alone!

I have completely redid the entire S&L system and its considerably more lean and mean now.. the bot isn't hanging up on all the parsing, well, actually, it does still have to parse (and reparse) up to the max limit of 50 times based on what ppl search for... but its now only 160 lines or so instead of over 1000 ...

but, now I got a new one... I'll start a new thread for this one...

-DjZ-

Statistics: Posted by dj-zath — Mon Nov 16, 2009 5:16 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-11T16:11:34-04:00

Nice to hear you got it working again.

One suggestion for improving your 100-250 url fetches.. Let your tcl-script write a single bash-script to do the large number of fetches, then launch this script into background (using exec). This way, your eggdrop ends up doing a single call to exec, and once the child process is created, it should hand over the control back to the script, leaving the task of spawning 100-250 child processes to the child...

I'm not sure what you are referring to with "remote refresh"..
If you are talking about forcing the client browser to reload the page, there's the meta http-refresh tag:
Another option would be to use some JavaScript with timers. Plenty of examples on how to do this spread all over the web.

You could use PHP to generate the needed tags/javascripts, though in most cases this could just be static text. Keep in mind, that once the page has been generated and the http-transaction has been completed, the PHP-script will terminate, so the PHP-script can't "tell" the browser 1 minute later that it should do this or that..

Statistics: Posted by nml375 — Wed Nov 11, 2009 4:11 pm

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-11T09:05:57-04:00

congraz

Statistics: Posted by raider2k — Wed Nov 11, 2009 9:05 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-11T04:53:06-04:00

UPDATE:

I have rewritten the TCL code... and the bot seems to be as fast as ever.. I have refined some things.. and eliminated about 600 lines of code.. (though I still haven't put the help or AI system back in yet) the Search and request System, along with the icecast/RAC parsers and switching logic has been revamped and some parts were completely rewritten.

Theres still some "minor bugs" in how the switching logic functions handle the graphics aspects.. when a graphic isn't available, it will display a "null" graphic instead of the current DJ's graphic.. this isn't a problem, though, just a different way of doing something... so I'm not worried about it.. (you can see this first-hand on my myspace page http://www.myspace.com/warpradio )

Thanks everyone for all the suggestions and help!

for those who wish to see this thing in action, come see me at irc.warp-radio.com:6667 channel #WARP-Radio and I'll be happy to demonstrate it for you!

oh, heres one for you..

how about a "remote refresh" for pages without using an interval or iFrame? preferbly PHP...

-DjZ-

Statistics: Posted by dj-zath — Wed Nov 11, 2009 4:53 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-08T06:19:39-04:00

hi nml and thanks for your reply!

In reading your last post, I understand what you are saying.. let me see if I can explain why I'm doing all this "in a straight line" and not "multiple processing".

In fact, the things you mentioned above is exactly the reasons I have to run things the way I do...

Orginally, when I started writing this thing, one of the problems I ran into was a case where one proc was running while another proc which the first proc "relied" on was not yet finished.. this resulted in irratic if not errored output.

example:

the "logic" proc running before the "parsers" procs finishing.

This most ALWAYS ended up with either an "online" state reporting to be "offline" or the proc crashing- reporting a "missing or non-existant varable" Although, I have ended up having to run a proc called "SetVars" which sets all the globals to their "offair"/offline state and to have this proc run first thing at startup, to help guarentee that the varable will never be missing in any state, I could not prevent one proc from eventually "catching" another proc in the "constraint" mode between states.. this was the case especially if the connection was "less than perfect" (and we both KNOW how reliable the internet is these days!). In the end, I was ending up with the eggie "freaking out"; setting "broadcast onair" then "broadcast offair" then back "onair" and then "offair", etc,etc,etc... (yes DSL is a wonderful thing... NOT!)
Another problem I had was that the eggie was "slamming" the connection with more than one connection request all at the same time.. being that the "return" path is over DSL.. I found this was causing the "concurrent connection limit" (which is three) to be exceeded- resulting in a dropped connection- and thus a mis-state onair/offair/onair/offair loop.. again results.

To combat this, I decided to run the parsers and THEN run the logic; all in the same proc.. and thus, one-state-at-a-time which helped to reduce the logical "snarfu" of catching one proc in constraint.

another trick was to put in a "change detector" which consisted of a "state read/write and compare" operation:

Code:

proc   update topic {} {     global Cash01 Show MyChan;     if {($Cash01 != $Show)} {          Set Cash01 $Show;          putserv "TOPIC $MyChan :Now Online with $Show";     };     return 0;};

This example will read $Show and compare it to $Cash01.. if it matches, the proc is then ended and the operation is ignored.. this helps in the case where a "missed" connection occurs by requiring TWO cycles to run before a change is detected (beit another one at the start of the proc and another from within the proc itself). The only disadvantage to this approach, however, is I have to generate a global just to "hold" the state of $Cash01 (in fact $Cash01 itself) now imagine a bunch of these amongst the ranks of the procs.. yes- they pile up quickly! (one of the reasons I have so many globals.. because I have a lot of these!)

Now, I DO have multiple procs, of course, but the "main" proc contains the "get info", then the parsers and then the logic. Then, theres the output control which is another proc that contains the topic/song and website updators and server title stream setters.. Then comes the procs for the key binds and control commands and all that... not to mention the login proc and the "setVars" proc.. and finally the search and request procs and other misc procs for greets, stats and other things... and not to forget the run loops... its quite a busy little thing! LOL

-DjZ

Statistics: Posted by dj-zath — Sun Nov 08, 2009 6:19 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-07T15:41:11-04:00

The use of catch makes no difference here, neither that you run the external program in background. The issue is the time it takes for tcl to spawn a new process environment. Using the exec command, irregardless of what you are actually executing, is very time- and resource-consuming. When you are doing this 100 times in a row, this could extend to several seconds...
Since you instruct exec to pipe all outputs to /dev/null and force the application into background, what parameters you send to fetch does not matter at all. The exec command would probably have returned before fetch even have had time enough to parse the URI, let alone try to access it.
I actually time'd a similar loop (using wget to replace fetch) which also launched the external application into background and immediately returning control to tcl.. 100 launches averaged at 3½ seconds.

The big problem is not that your eggdrop has a lot to do, but you are trying to make it do it all at once, and most likely way too often... Once again, I suggest you make use of the time command to profile the time usage of your different procs. Whenever there's a proc taking a second to complete, there's probably room for a lot of improvements, or you'd perhaps should consider taking these actions slower.

A remote example might be a script reading a file of several hundred lines, taking one action (of 500ms) for each line. If you were to do this in a loop (think along "while {![eof $fd]} {gets $fd data; parse $data}"), the application would freeze for minutes.. In this case, having the event engine fire up a proc that just reads one line and parses that line, and then return, would allow the event engine to do other tasks inbetween parsing lines. Doing the whole parsing of the file would probably take a little longer, but your application would still remain responsive during the parsing.

And yes, I do remember your issues with the http-package. That was well-diagnosed to the package using synchronous connections even in the "callback-mode", and would only occur if the connection-attempt is neither accepted or refused. This is something I'd call a bug in the http-package, as the behaviour does not comply with the documentations..

Btw, I'm not sure what you are talking 'bout with "catch" ignoring timing issues... Catch will only trap error return codes from executing the "script" parameter, nothing else.

Statistics: Posted by nml375 — Sat Nov 07, 2009 3:41 pm

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-07T07:31:26-04:00

Hi nml!
thanks for your reply!

I do notice that you've got a loop in your code that iterates 100 times, each time launching an external application that fetches some remote webpage. Due to the single-threaded nature of eggdrop (regardless of whether tcl is threaded or not), your eggdrop will "freeze" until all nested calls (if any) has completed. Especially, launching an external application takes a lot of resources compared to the other code, and doing it this way could very wellcause your eggie to become very sluggish... Do you get any "Timer drift" log-entries?

I used to get LOTS of timer drifts when trying to use an internal command set to call the external link.. however, I also use catch to either "get or no-get" and this seems to have helped emmensely..

You are CORRECTin figuring out what the code does.. you should have also noticed that I don't expect any return of input from the external link.. also, the "catch" basically ignores any fault or timing issue- in turn, this part of the script simply sends out 250 calls to the external link- do or die.. if the external link is ready, all the better, but, if not, no harm done! (the pre-condition elements helps to determine that the external link is "good" before the subroutine is ran anyhow) THAT part of the script runs FLAWLESSLY! To note, this part of the script will only run if a specific set of circumstances exists.. these are:
1.. the studio encoder is online AND streaming. ($DetSA) AND...
2.. The Playout Box is online ($DetOT) AND the Playout Cache is empty($DetPL)
3.. The Mode is set to "Auto Play" ($DetAP)
when these conditions are met, then the script will send 250 (in the latest version) "add track $PlayRand to the Playout cache" commands to the remote link and then "start playing"....

The code is quite ingenious, actually.. its smart enough to accept feedback on some parts (when required) once the commands are sent, the code will now STOP sending the commands since $DetPL will now be "on" and the conditions are no longer met.. the code will be ignored untill the 250 entries are dispatched!

about the timing issues...

NOW you're understanding my delimnea!

For this, I call external commands such as "bot.fch" which is merely the "fetch" command localized just for eggdrop.. the command strings/switches in the call are set up to have very short timeouts and such.. that I have planned the process thread to take 500 ms to complete.. though I give it (up to) 5 seconds to complete.. (I have sucessfully tested it to run at 1 second intervals.. but then a different problem occurs where icecast itself won't close its connections! that problem still exists, however its a matter of 5 to 20 hung connections versus 60 to 500! why this is? I haven't really determined yet! perhaps something to do with my ISP, Fedora and the way their router handles the connections through its network.. I have the eggdrop using an external fetch to get the info I want then to "drop" the connection- if it closes or not.. I let the Fetch command deal with the connection timout/hung issues... I merely had to come up with a soolution! Remember when you had me trying to use the http package and stuff? I could NOT keep eggdrop from hanging on EVERY connection it opened... and some indefinately so, the solution was to let FETCH open the connections- and let eggdrop continue on its merry way.. and that works PERFECTLY! no more hung eggdrop! the Bot doesn't even fall out of the channel or fall off IRC.. EVER.. (though the connections still continue to open and stay open....)

As for "having the bot do too much".. hmmmmm "limit" comes to mind again. I have been running the eggdrop- all this time.. and perhaps I DO have it doing too much.. LOL! You're right that its doing a LOT.. from managing the chatroom to rebooting the server at 6 am. .. from switching in the DJ feeds to generating all the dynamic elements of the website.. to even allowing listeners to search the arichive and to place their requests in the playlist for play. Yeah, that eggie's doing a lot!

I have refined the code once again.. and I have the eggie working well with all the code except the search and request.. I took that out untill I can come up with a better way to retrieve results without a parser entry for every line of text.. its massively NASTY! and I think having 500+ lines of parsers might be too much! was that the culprit? not likely since it WAS working before.. with the code.. the problem came in once I added the image-handling stuff.. and then it broke.. now, the image handler is in place and all is working nicely and fast... I have taken a break from it all- and even took my website down for the time being.. (like you said eariler) for I have been REALLY fustrated- not just with the eggdrop, but a lot of stuff.. but thats going beyond the scope of this thread...

if you're really interested in seeing the whole code, I could pop it up somewhere for you.. though I don't want to release it to the public just yet I'm planning to make a "light" version for the playout system to put on its forums.. but this version now has some "trade secrets" in it.. I have put a lot of time and effort into it.. its not quite hap-hazzardly thrown together.

-DjZ-

Statistics: Posted by dj-zath — Sat Nov 07, 2009 7:31 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-04T13:25:31-04:00

This is just what my guts tell me, from this thread and earlier ones..
You are pretty much trying to do way too much at the same time. I know you have a lot of code that's triggered by 5sec utimers - and I could see the code you pasted taking several seconds to complete under certain conditions. If you're doing alot more like this, I could easily see how your scripts could keep your eggdrop so occupied starting new processes and stacking up more timers, that your eggdrop ends up "unresponsive" - or perhaps "occupied" would be a better word... Frankly, I'm surprised that your eggdrop barely manages to stay online for any period of time...

So, as for limits..
The main limit within eggdrop, is that while it is doing one thing, it will not be able to take any other actions. Timers and triggered bindings will instead be queued and handled as soon as the current task is completed. In general cases - that would mean whenever your proc has completed.
So, if you have a proc that takes 5s to complete, that means during those 5 seconds, your eggdrop will not be responsive. Should you, at the start of calling this proc, create a timer with 5s or shorter duration - that will call this proc again, the new timer would've expired when the first invocation of the proc has completed, and the new one would be instantly executed. In essence, leading up to an infinite loop where your eggdrop hardly does anything else except running this proc over and over...

There's no problems what so ever (perhaps except limitations enforced by the hardware or kernel) to load your eggdrop with script code exceeding 10-15,000 lines. There isn't even any problem having individual procs extending huge number of lines, or nesting deep recursive calls. What you have to be wary of, is runtime. That is, how long will it take for this "entity" of code to complete, and let the control be handed back to eggdrop (sort of).

If you've got a lot of code like that loop doing 100 exec's in your script, then I really suggest you rethink your whole approach, starting from scratch. If you are coding it that way, you'll constantly run into these kind of problems over and over...
I would also recommend that you utilize the tcl "time" command to profile the time usage of your various code snippets.

Statistics: Posted by nml375 — Wed Nov 04, 2009 1:25 pm

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-04T11:30:25-04:00

mh, external calls .. always a nice thing ^^
the only thing popping into my mind right now is:

try to run your script on any other box, see if its the script or the configuration .. cant get any easier than that in my opinion, at least its an easy-to-do option.

Statistics: Posted by raider2k — Wed Nov 04, 2009 11:30 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-04T09:36:25-04:00

zath, in those situations, just sit back, put your mind on other things, and just try to calm down. We all get very frustrated every now and then with things "just not working for no god reason" - usually that means we're overlooking something to trivial to make notice of...

Yuour putlogs, I explicitly said in my last post, that if you use curly brackets instead of quotes, nor variable or command subsitutions will occur. You'd get the very same result if you use them (the beackets) on putserv..
Ever tried something like this?

Code:

set color "blue"putlog "The value of color is $color"

I do notice that you've got a loop in your code that iterates 100 times, each time launching an external application that fetches some remote webpage. Due to the single-threaded nature of eggdrop (regardless of whether tcl is threaded or not), your eggdrop will "freeze" until all nested calls (if any) has completed. Especially, launching an external application takes a lot of resources compared to the other code, and doing it this way could very wellcause your eggie to become very sluggish... Do you get any "Timer drift" log-entries?

Statistics: Posted by nml375 — Wed Nov 04, 2009 9:36 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-04T05:30:43-04:00

hi Arfer:

Sorry for the long delay...

The problem isn't in the compile, I have a working eggdrop which was compiled on the box and OS in which it runs..

The problem is when I exec a seemingly large piece of code, eggdrop gets sluggish/laggy. The more lines I add to the code, the more laggy the eggie gets... untill the other day, at around 3280 lines or so it just stopped alltogether.. once I trimmed off a 100 lines or so it starts up again, but its still very sluggish..

-DjZ-

Statistics: Posted by dj-zath — Wed Nov 04, 2009 5:30 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-04T04:57:07-04:00

compile fine, running eggdrop failed?
or when does missing libs errormessage appear?

Statistics: Posted by raider2k — Wed Nov 04, 2009 4:57 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-04T04:51:41-04:00

Hi arfer!

I recompiled eggdrop fresh on Fedora and BSD the one that ran on the Verio box would not even load at all! error log returned "missing libs" errors..

-DjZ-

Statistics: Posted by dj-zath — Wed Nov 04, 2009 4:51 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-04T04:41:52-04:00

raider2k wrote

another but ... even if you run an eggdrop which has been compiled on a different system it should work .. should .. or might .. most probably it will work - but like I said, Im NOT 100% sure about bsd or any non-debian, non-gentoo distribution.

I would have thought at one time that the above statement was reasonable. I now know that you cannot be sure of this. My shell provider recently changed linux distro from Fedora to CentOS due to support issues. One of my scripts that previously ran flawlessly for many months simply ceased to function, whilst there was seemingly no effect on other scripts.

The script in question was http code and I suspected that it failed on a Tcl 'catch' command.

the first signs of "trouble" appear to be TCL/eggdrop not catching some errors

I can only say suspect because I spent relatively little time trying to prove it. I don't believe in coincidences and I therefore pretty quickly focused on the linux changes and completely recompiled/reinstalled the bot. The script functioned as expected again without any code changes.

Statistics: Posted by arfer — Wed Nov 04, 2009 4:41 am

[SOLVED] Large TCL scripts, RAM and Eggdrop...

2009-11-04T04:45:04-04:00

Hi there nml and Speechless:

first off, thanks for the replies!

I guess I'm speaking in "broken idea" here- which would make things hard to follow from the outside world.. I apologize for that.. I didn't mean to take you on a roller-coaster only to end up either stuck in a tunnel- or to exit the ride dissatisfied.. that wasn't at all my intent.

I guess I was trying to say that I have been experiencing some strange behaviors with eggdrop, my server, my ISP, their routers, TCL, the compile and finally.... my code!

although, things seem to be functioning OKay, on their own, or as individual components.. but, as a set in this particular situation, they aren't operating as I recall they did when I ran a simular configuration on an entirely-different shell/OS.

of all of this, I didn't consider anything else BUT my code being the culprit- however, from the replies I received from this thread, perhaps the problem may be more "involved" that I initially anticipated.

You all probably think I'm some weird or mental dude whos just, plain, out of his mind.. and you probably aren't far from the truth there, but..I have to admit that when I don't fully-understand something, I can go into a sorta "panic" and I become extremely-flustered- expecially when I'm trying to understand the problem and explain it to everyone here.. and I just "freak out" and say things all out of context or exaggerated to a point where its just mangled.. and, reading back, I have done that here a few times allready!

with this all being said, perhaps I need to start over....

First off, I asked a question about there being a "hard limit" of some sort in TCL or the TCL handler (the thing that eggdrop uses to access the TCL libs) and I was told "no, there is no limit, at least within the scope of your problem".

now I'm going to carefully describe what I have experienced..and why I came in here thinking there was a limit..

if I load eggdrop into "debug mode", using -n, with just eggdrop's base configuration code plus some minor functions.. and I purposely put something in that should make eggdrop "bawk" (like using a quote without an escape) I would expect the bot to error- and the config file not to load, instead, return an error and return to prompt.. which it does.. as expected.

my code, on the other hand, won't do that- instead it will continue to load and never "catch" that error.. even if I deliberately put the error in! of course, the code won't work right.. if even at all.. but the bot will load nevertheless.. I understand the idea that perhaps theres another link of code that "cancels out" the inital errors.. but, though, likely for one, not for many in a row.. I have taken my code apart and carefully, line-by-line have repasted it into a new file and looked carefully at each line, making sure its clean and proper.. I'm not saying I'm an expert with TCL, no, not a chance.. and perhaps I could hire someone from here to look it over and give suggestions to what it needs in order to work right... but, before I could do that, I'll need to make sure the eggdrop and TCL compiles are right...

For now, I'll assume the code is at fault.. and, again, take it apart....

As for putlog{}, if I put (example):

Code:

set $color "orange";if {($color != "green")} {     set $color "not green";};putlog {what is color? $color}

what I see in the log is:

"what is color? $color"

and not:

"what is color? not green"

for this, I have to "putserv" it to the channel in order to read the varable as it should be set/shown... why is this? I assume theres something wrong with the compile.. based on what I have read here in this thread. (I DO recall it working properly on the Verio box)

did that help explain things or have I hopelessly started another roller-coaster ride?!

-DjZ-

Statistics: Posted by dj-zath — Wed Nov 04, 2009 4:41 am