The Future of PsychoStats Options

[Stormtrooper]

Alright, it's been awhile since I've announced anything new about PsychoStats so I'll finally give a brief update on what I've been doing.

First things first, I decided to take a good hard look at the present state of where PsychoStats is and where I believe it should be and I've finally come to the conclusion that in order for PsychoStats to stabilize and mature into a better product I needed to take a step back and rethink how I've been developing it. I decided to stop development on the current trunk version 3.2 of PsychoStats. I felt that it simply contains too many bugs and is becoming harder and harder to maintain due to poor programming practices and lazy coding habits. Version v3.2 was starting to feel the effects of "Feature creep" and "Over-engineering" and was hindering its ability to be a good, robust, solid and effective software product for the community.

So, in a nutshell, I stopped working on v3.2 of PsychoStats a couple months ago and have started to develop a new version of PsychoStats: version 4.0. This new version is almost a complete rewrite from the ground up. This has allowed me to fix a lot of lingering issues that the old versions had. Rewritting software tends to take a lot longer than trying to "fix" old software, but the end result will be a better software product overall. And really, this isn't a complete rewrite. I am still reusing several core objects from 3.2, I'm just re-engineering them to be simpler and easier to maintain.

So, what's going to change with PsychoStats v4.0 (PS4)?

How about Real-Time stats? This is the #1 top feature that most of the community has requested time and time again. PS4 can handle multiple streams of logs at the same time. This includes listening on several ports simultaneously and allowing multiple servers on each port at the same time. Each stream can even be a different game.
Multiple games can be tracked in a single installation. This has been another common request and can now finally be realized.

There are a lot of core elements that are being reimplemented that will make the entire core of PS4 stronger and more reliable but aren't important enough to list here.

I've taken a lot of feedback from users and some of the ideas or requests are being implemented into this new version.

Questions? Comments? Let me know.

[Stormtrooper]

I should have mentioned that the trunk in the SVN repository has been updated to the new 4.0 code tree and is NOT in a working state and you shouldn't attempt to check it out and expect it to work yet. The old 3.2 trunk was moved to the tags/3.2 directory in the repository. Use the 3.2 download link on the main page if you want the last good copy of 3.2. The 3.2 'nightly build' will no longer be updated.

[ToXedVirus]

Hi,
I have some questions about the new version of Psychostats.
You mentioned that you are going to implement RealTime stats, so my question is, how will you implement this?
If you run the half life server (the old one, since I still love playing CS1.6) the log is printed in the console. I was playing around with the the idea to write an app in C# (on mono, MonoCurses) which would wrap the console interface, get the output, parse it and display some server information in a more readable way in the console.
Are you doing something similar? Parsing the console output? Or are you forwarding the information somewhere?

[Stormtrooper]

Halflife game servers support streaming log data to a remote UDP port. Which means a remote server can listen on that UDP port and accept the log stream in real-time. Game events are sent as a UDP packets as soon as they occur. PsychoStats v4.0 can now correctly listen for a stream and process the event data w/o having to read batches of log files after-the-fact.

There are limitations of this. For starters, UDP packets are unreliable by nature. So it's possible to drop events and never know it. For the most part this should be fine for most servers, especially those that stream to a local port, or to another server on the same network. Another problem is that if PsychoStats is stopped (or crashes) it'll miss any events streamed until it starts up again. Unlike when reading log files, you never drop a packet and even if PsychoStats crashes it'll start where it left off w/o losing [many] stats.

[BuMHeD]

This news is as exciting as it is disheartening, since it virtually assures a longer wait before we see a fully-complete release. However, that said... I am very glad that you have taken the time to evaluate your coding process in order to arrive at what must certainly have been a difficult decision (a re-write is no small endeavor!) I feel confident that the final product will be a grand improvement upon the piecemeal development that was occurring between 3.1 and 3.2... Also, I would like to volunteer, if I may, to help you build a detailed Knowledgebase (Kb) and FAQ to be released along with version 4.0... This has been another point of contention for users in the past, and I'd like to do my part to mitigate the number of duplicate support threads.

-------------

On the subject of Stream reliability: do you have any ideas that might help limit the possibility of dropped / lost packets, whether due to crash or packet loss?

-------------

Please forgive my lack of in-depth knowledge regarding SQL, but I've been brainstorming a bit, and I had a few thoughts on how you could improve reliability (ideas marked with * are hypothetical, conceptual ideas of which I am uncertain are even possible... ):

1. Allow for the configuration of a fetch-based FTP logsource in tandem with the Stream (already possible in 3.1)
*2. As data is received from the incoming UDP stream, persistently cache a certain # of lines and write them locally to (a) logfile(s)
*3. Periodically (configurable) perform log fetching (per round?) and compare file size, line count, etc of the FTPed log versus Stream-generated.
*4. Analyze and report discrepancies between FTP and Stream data graphically (% loss over time?) to help Admins weigh their options.
------
*5. Maybe it's possible to create a 2nd, parallel set of tables (database?) into which stream data is written in real-time and displayed by front-end...
*6. If #5 is possible, the parallel table data could be compared with an FTPed/parsed log from the same round to account for any missed data...
*7. If #5 and #6 are possible, the parallel table set could then be merged with the primary database (archived) for long-term consumption.
*8. A new parallel table would be created for the new round, and the cycle would continue again with #5

Obviously, #5-8 are based entirely upon a very limited understanding of SQL, so feel free to disregard any superfluous suggestions. In any case, I'm trying to help inspire a way to effectively permit the stats.pl to write stream data to the database to be reported to the frontend in real-time, but allowing for some form of posthumous Quality of Service (QoS) via checksum... something that isn't native to the UDP (unreliable) layer 4 protocol. Because the UDP packets are not being acknowledged by the receiving device, the game server simply sends each log line once, and only once, regardless of what happens to the packets along the way or when they arrive.

---------

In regards to the risk of the stats crashing, perhaps...

*9. Configure some form of network proxy that operates independent of the stats.pl and acts solely to receive and archive incoming, persistent log data in the event that the stats.pl fails.

Although this would add an "extra hop," further degrading the goal of real-time stats, it could serve as a safety net for log data that the server will continue to send, regardless of interruptions on the backend. The proxy would essentially accept the stream data and cache it for a minimal number of lines, then hand it off to the stats.pl for parsing, clearing the cache. The cache would only need to be large enough to allow the proxy to acknowledge with the stats.pl that the log data is being received and parsed (basically confirming, via read-ahead, that the stats.pl process is still alive). In the event that the stats.pl does not acknowledge the data handed off by the proxy, the cache is timestamped, then written to a local log file continuously until the stats.pl is brought back online again. This would also require the proxy and stats.pl to share some system of sequence numbers in order to keep track of where the stats.pl left off when it failed to acknowledge, giving it a reference point from which to proceed.

In order to make this proxy more effective, it may be something that would have to be implemented on the game server, acting as a virtual gateway for the logaddress_add command. I'm not exactly sure how this would (or even could) be done, but having the proxy operate directly on the game server would virtually eliminate the risk of both packet loss across the network and stats crashing... In my imagination, the proxy would receive the UDP transmissions from the game server and, most generously, encapsulate them as they should be (TCP-style), and transmit them to the stats.pl with some built-in QoS (sequence numbers, checksum/hash, and SYN-SYNACK-ACK).

------

Another thought on crash/loss prevention... If all data written to the database in real-time can be managed on a per-round basis, then maybe database changes can be tagged with a cyclic "sequence number" identifying the round in which the data was logged. The sequence number would be tied in with the filename of the log that is being generated on the game server and would be cleared at the end of each round, but only after a new round is begun successfully... This way, if the stats.pl crashes during regular operation, the database changes associated with that round can be rolled back and replaced with the data from the game server's logfile (FTPed). All of this would occur when the stats.pl is restarted, while simultaneously caching any incoming Stream data that is being sent while the stats.pl plays "catch up."

Here's an amateur attempt at illustrating a potential sequence of events:

1. Round 1 Begins (all new data is tagged as sequence 000; associated with file 'L0120000.log'; flagged as 'incomplete')
2. Incoming stream data is received, parsed, and written to the database (or per-round changes could be mirrored to a parallel database)
3. The stats.pl crashes unexpectedly; game server continues to stream data, and log data is lost
4. [On the game server] Round 1 ends; log file 'L0120000.log' is written; Round 2 begins, generating 'L0120001.log'
5. Stats.pl is successfully restarted; all incoming Stream data begins writing to cache until "catch up" is complete
6. Stats.pl checks database for sequenced entries flagged as 'incomplete', notes the associated log file, and rolls back the changes
7. FTP session downloads and parses log file 'L0120000.log' and any other completed log files thereafter (skipping last log; zero-byte file)
7a. Round 2 log file 'L0120001.log' is located, but is the "last log file" (currently being written), thus it is skipped, ending the "catch up"
8. Round 2 is tagged as sequence 001; associated with file 'L0120001.log'; flagged as incomplete
9. All cached data from Round 2 begins parsing until the stats.pl has processed the cache, resuming operations on incoming Stream data
9a. The "caching" must be able to span Rounds in the event that Round 2 ends and Round 3 begins before the cache is fully processed
10. Round 2 ends successfully; 'incomplete' flag is cleared and sequence number is incremented pending the start of Round 3...

--------------

Okay, that's enough brain-storming for one night. Forgive me if I went a little overboard, lol.

*~BuM~*

[Stormtrooper]

Thanks for the feedback/brainstorming Bum...

You're over-engineering this quite a bit. Dropped UDP packets shouldn't account for more than 2% of all packets sent under normal circumstances (2% can be considered 'OK'). And most people won't drop anything. And in the case of stats.pl crashing the simple solution is to wrap stats.pl in a looping script that restarts it automatically if it crashes after a few seconds. This would only result in the loss of a few seconds of data. If an admin is having that much trouble with a UDP stream then they have other network issues to deal with and can use a FTP stream instead. Or, another possibility is to have a proxy running on the game server that accepts the UDP stream and writes it to local log files to be processed in batches with stats.pl; This prevents issues with crashing but then you lose the 'real time' updates. This is exactly what the old "PsychoStreamer" script did.

Also, you've made a mistake on how the log files are written by the game server. Halflife writes to a single log file until the map changes and then increments the 3 digit number in the log file name. It's not a round based number. Also, you forget about some games that might not have rounds.

The idea of having the stats stream "verified" by also downloading a log file and comparing the last X events to the file is something I've considered a long time ago, but again it goes into the over-engineering bucket. I've also considered on writing a 'proxy' daemon that runs on the game server and streams log data via TCP, this is possible and is still on the table of useful ideas to possibly implement in the future. Not only would it provide reliability but it could also provide security and less bandwidth by encrypting and/or compressing the log data.

[BuMHeD]

The idea of having the stats stream "verified" by also downloading a log file and comparing the last X events to the file is something I've considered a long time ago, but again it goes into the over-engineering bucket. I've also considered on writing a 'proxy' daemon that runs on the game server and streams log data via TCP, this is possible and is still on the table of useful ideas to possibly implement in the future. Not only would it provide reliability but it could also provide security and less bandwidth by encrypting and/or compressing the log data.

Thank you, ST, for taking the time to reply thoughtfully to my post. This has been a very good learning exercise for me. I certainly agree on the over-engineering sentiment, except regarding "verification." I don't know how difficult it would be to implement such a system, but I think the minority of incredibly anal server admins may appreciate the ability to "Process FTP logs for verification," even if it ran as a separate batch altogether. Anybody who happens to notice a discrepancy in their Stream stats may find it a useful tool to rule out network connectivity / packet integrity. Besides, as you said... less than 2% of the UDP stream should be lost, which would make this tool a great metric to make sure that UDP loss is kept low (that is, if UDP is still on the table, pending consideration of the TCP daemon that you mentioned). But, I imagine that you've thought it through, and it would only make it in the 'over-engineering bucket' if implementation cost outweighed the potential benefits... I just know that it would serve as a great piece of mind to have a statistic telling me how effectively my Stream is performing rather than living with the uncertainty.

If nothing else, I'm glad that my TCP "proxy" suggestion fell into the category of past-and/or-present consideration. I'll do my best to come up with some good wish list / suggestion items to further encourage your firing neurons.

Cheers!

*~BuM~*

PS. While we're hot on the subject, I would like to refer you to http://www.hlsw.net, which provides a Server Management Tool (HLSW) with which you may or may not be familiar. On the site, they also provide a Metamod plugin called "Logmod," which is designed to utilize the logaddress_add function in tandem with HLSW to optimize log streaming to their tool. Consequently, I fear that use of LogMod could (and potentially already does) interfere with your real-time stats objectives. Consequently, you may want to look into it, or have a skilled metamod coder look at the source, just in case. A heads up, if nothing else.

[Stormtrooper]

but I think the minority of incredibly anal server admins may appreciate the ability to "Process FTP logs for verification,"

And, I, would be one of those people most likely (coming from a networking background myself). It would be nice to visually see network performance metrics but this will have to be something to consider after everything else as it doesn't help to get the new product released.

[BuMHeD]

And, I, would be one of those people most likely (coming from a networking background myself). It would be nice to visually see network performance metrics but this will have to be something to consider after everything else as it doesn't help to get the new product released.

Haha. Ya know, I'm perfectly comfortable with that answer. Thank you. I couldn't agree more! Carry on, sir.

PM, btw.

*~BuM~*

[BuMHeD]

Edit: Regarding the LogMod heads-up... it may end up being more of a blessing that a burden, according to the self-published intentions:

LogMod is a Add-On for your HL Server by using MetaMod. The HL-Server has only limited possibilities to let you receive Log-Files via an external Program or even Real-Time Stats. LogMod extends these limitations. With LogMod your Real-Time Stats continue working even if you redirect the Logs with HLSW or another program to your personal computer.
With HLSW it is possible that up to 25 Clients can receive the Log at the same time. An extra plus for LogMod is the circumstance that you need no further configuration if you want to receive the Log behind a Router or Gateway. Therefore LogMod is easy to use by everybody.

It may simply involve replacing "logaddress_add" with the convenience and added functionality of 'logmod add'... I'm not sure how keen you are on collaboration, but HLSW is under active development and stays very current. Either way, here are more details on installing and utilizing Logmod: LogMod Readme

*~BuM~*

[Stormtrooper]

But "LogMod" is simply an attempt to enhance the UDP logging feature within HL (I'm not sure what limitations they're talking about). It still uses UDP to transport log events and still suffers from the same reliability issues. It appears to offer some network statistics, but really that's not helping anyone much, except those that want to closely monitor their network metrics.

[BuMHeD]

But "LogMod" is simply an attempt to enhance the UDP logging feature within HL (I'm not sure what limitations they're talking about). It still uses UDP to transport log events and still suffers from the same reliability issues. It appears to offer some network statistics, but really that's not helping anyone much, except those that want to closely monitor their network metrics.

I haven't had a chance to test this, but I suspect it has been causing me problems for a while without realizing it. Because my fellow server admins and I have been using HLSW for years now, each time we attempt to capture logs using the "Get Log" feature, we're opening a new log stream (UDP) to our HLSW client. I intend to verify this as soon as I can... but by using the "logaddress_add" function in the server.cfg to establish the Stats stream, I think it is possible that the use of HLSW to create new streams can disrupt the stream that is being used for the stats already. Whether this is happening by sheer accident (say, attempting to stream using the same outgoing port)... or is related to some other function, I don't know..

Basically, all I seek to determine (by experimentation) is if LogMod has the potential to create compatibility issues for PsychoStats. If so, I want to see if using LogMod's "logaddress management" is preferable over traditional logaddress_add, keeping it separate from separate user inquiries.

Disregard for now, and I'll get back to you when I've played with it a bit.

*~BuM~*

[Guardia Republic...]

Good to know that PS will have a next version.

Of course real-time stats should be included and I think it is a big feature.

I am particulary interested in dods. Last year the game had a big update adding maps and new events like domination and revenge.
Awards for bombs by default for dods? Awards for new events?

Also data related to detonation maps should be included in Team /Action Profile section (only flags are shown currently).
I would like to have a link with detailed bonus information to know how bonus points were calculated.
For instance I have added bonus points for domination and revenge evens but I do not know if they are working because I have no detailed information.

I have using PS 3.2 for a while. The only issue I found was related to players using special chars in names.
It has a random behaviour and I do not know why. Some days names are shown OK, sometimes wrong.

http://www.dayofdefeatsource.com.ar/stats/player.php?id=37

Installing PS 3.1 will solve this issue?

Regards,
Pablo.

[[Arnold]]

Good to hear that PsychoStats is going to stay and improve! Will you provide a "final" version of 3.2 until 4.0 is ready for public use? As far as I could see, the nightly build is still sort of 3.1.x.

[Stormtrooper]

3.2 was already tagged in SVN and the nightly build as you see it is the final version for 3.2.

[flintridge]

I haven't had a chance to test this, but I suspect it has been causing me problems for a while without realizing it. Because my fellow server admins and I have been using HLSW for years now, each time we attempt to capture logs using the "Get Log" feature, we're opening a new log stream (UDP) to our HLSW client. I intend to verify this as soon as I can... but by using the "logaddress_add" function in the server.cfg to establish the Stats stream, I think it is possible that the use of HLSW to create new streams can disrupt the stream that is being used for the stats already. Whether this is happening by sheer accident (say, attempting to stream using the same outgoing port)... or is related to some other function, I don't know..

Basically, all I seek to determine (by experimentation) is if LogMod has the potential to create compatibility issues for PsychoStats. If so, I want to see if using LogMod's "logaddress management" is preferable over traditional logaddress_add, keeping it separate from separate user inquiries.

Disregard for now, and I'll get back to you when I've played with it a bit.

*~BuM~*

I run both Pyshco Stats (FTP) and HLstats (Stream). On log streaming from HLSW i had many issues with hlstats working correctly until i installed LogMod from HLSW. I have a few of us that use HLSW to run the server and when we logged in several times hlstats would stop logging data. I have no idea if there were other issues, but once i did install LogMod that issue went away, our logging in no longer seemed to effect HLStats. I think he is right in saying to keep that in mind.

I love both Psycho Stats and HLstats, both are simple to install and very user friend and free. ^^ How can you not luv free and well supported.

I am looking forward to the update and will always run Psycho Stats. I do not think it would hurt though to look into the methods at which both HLstats and LogMod (Especially LogMod) stream the data. I use to have port permission issues with HLSW also until i installed LogMod and all that went away.

[skyride]

Just another suggestion for verification. Psychostats currently listens on a single port for data. Well It is unlikely that 2 packets would be dropped so I guess it could be possible to simply send 2 UDP streams on seperate ports. Then stats.pl would verify all data. If the same data came through on both, it would be considered "good" data and log it, and if it only recieved the data on 1 port then it would still write the data but flag it in a log of some form.

[Stormtrooper]

Just another suggestion for verification. Psychostats currently listens on a single port for data. Well It is unlikely that 2 packets would be dropped so I guess it could be possible to simply send 2 UDP streams on seperate ports. Then stats.pl would verify all data. If the same data came through on both, it would be considered "good" data and log it, and if it only recieved the data on 1 port then it would still write the data but flag it in a log of some form.

That's an interesting idea, but not w/o its own set of problems. For instance, you've just doubled the amount of traffic between the game server and where PS is listening. Also, a network interface can only accept so many packets per second before it starts dropping them locally (nothing to do with the network itself). So for a very busy server that is collecting streams from multiple servers would quickly degrade in performance.

You also have the problem of synchronizing the 2 streams. UDP packets are not guaranteed to arrive in the same order they were sent. Generally this wouldn't realistically be a problem, but it is something that would have to be considered.

Also, to clarify one of your uncertainties... PS4 can listen on multiple interfaces and multiple ports per interface for log streams. There's virtually no limitation (I haven't fully load tested this yet) in how many streams a server could handle before the network interface starts dropping packets. BTW, each stream can be an entirely different game from the others.

@flintridge
If LogMod somehow fixes the problems with HLSW streaming data then that's good, but chances are it doesn't actually change the way the data is sent to the stream. I'd have to test to verify; but I doubt it would. Which means installing LogMod should be a transparent change to the server and PsychoStats.

[Arjen]

Im sorry to ask but when do you think this might be ready!
Will it take a very long time, or not at all ?

And i would really like to help you with testing.
Ive got a CSS server running on my second PC (at home)
And ive got a rented website AND an site at my second pc (at home)
So, if i can help you, say it!

[Stormtrooper]

Anyone can help test it, once its in a workable state. Using SVN anyone can download the current trunk and play with the software. But right now it wouldn't work if you tried. I will eventually announce when the trunk version is at least in a beta stage for people to try.

And no, I don't really have a hard date for when this will be released. I'm making a lot of progress in the backend, fixing countless bugs and adding enhancements. Once that's done I will rework the front-end.