PDA

View Full Version : Compression and log files



Aelfin
01-30-2008, 03:10 PM
i've been overhauling a lot of Tango's code to add some new stuff.
part of the overhaul included looking into compression for different things. for one, i can now optionally cache log lines in a compressed format, seems to work good and may open the doors to other things down the road.

i am looking into adding the ability for tango to open compressed log files. i think it would be nice. would anyone else find it useful?

if so, i have to determine what to support. zip, gzip, bzip, can do them all i believe. the next hard part is making an assumption: that only ONE log is held within the compressed file. is that reasonable?

Gamanern
01-30-2008, 05:01 PM
When you mentioned compression, I had a little think about it myself, but chose not to act on it at this time, but here are some thoughts that may help you.

Logging absolutely everything will use about 1GB every 6 - 12 months, current hard drives are at least 160GB or larger. Without ever deleting, you will likely upgrade your computer before you would ever run out of space, unless you have a lot of movies, music and other media taking up an enormous amount of space. This alone was the reason I didn't go any further.

Instead, I made the ability to export individual fights as small, named and dated log files, so people could easily store the fights they want, allowing them to delete their main log files while keeping just what they wanted. This saves a tonne of space, not quite as much as compression, but enough that I was happy with the results.

Your compression suggestion would be well suited doing something similar, where you produce small logfiles, and then append them to the 'storage file'. Then you just need to make Tango able to pull individual files from an archive. It is also much faster to read in a single fight than parse a super file.

As for format, assume 90%+ use Windows XP and Vista, and from their the OS directly supports ZIP. As your program is a Windows App, that means ZIP would be your best bet.

Aelfin
01-30-2008, 06:44 PM
try tango sometime. it has a split option. it runs through a log file and splits it into new files by date. i then zip those files so they take even less room. that's just me =) been coding since timex sinclair days, i like small files (easier to email and ftp, too). and ya, i get pressed for space sometimes... digital video recorder makes lotsa files of the baby girl. they collect until i decide to archive them off to the file server.

since tango can now store all the lines from a log in a pretty frugal fashion, spitting out complete fight files is a non-issue now.

other uses for the compression capability: the xml and html tango produces. xml is... verbose. done right, IE7 can decompress a bz2 html stream on the fly.

EDIT: i've thrown the functionality in now regardless. /shrug, pretty trivial, maybe useful.