I have written a new module, that allows unlimited size strings to be read/written in the config.ini file.

I am submitting it here for peer review, and comment.

Sorry it lost all the indentation during the copy/paste.

Edit: Dec 10, 2013. Please note: The recommended method of data storage is SystemNotes, for all data types that are appropriate for that method. If the data you are storing will only change when a specific character is online, then please use the SystemNotes method to store that data. Please only use lkini to store data that may change when no character is logged in, or data that is not character specific, and may change when any character is logged in. Example: Character specific options (options that may differ from one character to the next) should be stored using SystemNotes. Global options (options that operate across all characters) should be stored using lkini or gkini.

Edit: Dec 10, 2013: -- lkini version 1.1 -- blanks out excess keys (if any) on write.
Edit: Dec 27, 2013: -- lkini version 1.2 -- tighter code.
I have replaced tabs with --[[]]. It will run this way. Or you can replace back to tabs.
---- Recommendation: This code does not check the size of the data you are writing. It writes it, using as many keys as needed. I suggest that you (the plugin writer) check the size of your data before you write it. If you know that your data should always be less 5k, have your code check that the data is less than that before writing it. Do not write a megabyte of garbage to the config.ini. Writing hundreds or thousands of keys will freeze the game, and it could corrupt the config.ini file. At ~0.75k per key, 5k is only about a half dozen keys, and if only few plugins use that much, it should be ok.

Note: You do not need the lkini:Test() function for production use. I only list it here for people doing peer review.

Edit: Dec 27, 2013: -- lkini version 1.3 -- Added a size limit argument to lkini.Write()
ex: lkini.Write("%plugins%", "akey", "123456", 5) -- will not write anything, will return false.

-- lkini version 1.3 -- added sizelimit argument to lkini.write()
-- lkini version 1.2 -- tighter code.
-- lkini version 1.1 -- blanks out excess keys (if any) on write.
-- lkini.Read(section, key, default)
-- lkini.Write(section, key, string, sizeLimit)
-- config.ini long string read and write. Extends gkini.ReadString and gkini.WriteString to very large strings.
-- adds a percent sign and an index number to each key, and splits the data across indexed keys as needed.
-- low overhead. short strings will only generate a single key.
-- console_print("Loading: lkini.lua")
declare("lkini", {} )
local baseindex = 1000000 -- some big number starting with a number other than zero
local maxLineLen = 766 -- Maximum Line length including all elements (key, data, and equal sign).
function lkini.Write(section, key, data, sizeLimit)
--[[]]if sizeLimit and ( string.len(data) > sizeLimit ) then return false end
--[[]]local ix = baseindex
--[[]]local maxDataLen = maxLineLen - string.len(key.."%"..baseindex.."=")
--[[]]while data ~= "" do
--[[]]--[[]]gkini.WriteString(section, key.."%"..ix, string.sub(data,1,maxDataLen))
--[[]]--[[]]data = string.sub(data, maxDataLen+1)
--[[]]--[[]]ix = ix + 1
--[[]]end
--[[]]while gkini.ReadString(section, key.."%"..ix, "") ~= "" do -- blank old keys
--[[]]--[[]]gkini.WriteString(section, key.."%"..ix, "")
--[[]]--[[]]ix = ix + 1
--[[]]end
--[[]]return true
end
function lkini.Read(section, key, err)
--[[]]local res, r, ix = "", "", baseindex
--[[]]repeat
--[[]]--[[]]r = gkini.ReadString(section, key.."%"..ix, "")
--[[]]--[[]]res = res .. r
--[[]]--[[]]ix = ix + 1
--[[]]until r == "" -- read until we find an empty element.
--[[]]if res == "" then
--[[]]--[[]]return err
--[[]]else
--[[]]--[[]]return res
--[[]]end
end

-- for development only --
function lkini.Test()
--[[]]local sec, testkey, data = "%plugins%", "AnExpendableTestKey", "abcdefghijklmnopqrstuvwxyz"
--[[]]local keysize = string.len(testkey.."%"..baseindex.."=")
--[[]]local maxDataLen = maxLineLen - string.len(testkey.."%"..baseindex.."=")
--[[]]console_print("Test: lkini: Write/Read:")
--[[]]console_print(
--[[]]"Max line: "..maxLineLen..", keysize: "..keysize..", data size: "..maxDataLen.."+/-2")
--[[]]while string.len(data) < (maxDataLen + 2) do
--[[]]--[[]]data = data .. data -- expand test pattern to larger than needed.
--[[]]end
--[[]]for i = 2, -2, -1 do -- test data sizes across max size boundry, down shift.
--[[]]--[[]]data = string.sub(data,1,maxDataLen + i)
--[[]]--[[]]lkini.Write(sec, testkey, data)
--[[]]--[[]]if data ~= lkini.Read(sec, testkey, "false") then return false end
--[[]]--[[]]console_print(i,".."..string.sub(data,-5-i))
--[[]]end
--[[]]for i = -1, 2 do -- test data sizes across max size boundry, up shift.
--[[]]--[[]]data = data..(i+2)
--[[]]--[[]]lkini.Write(sec, testkey, data, 999)
--[[]]--[[]]if data ~= lkini.Read(sec, testkey, "false") then return false end
--[[]]--[[]]console_print(i,".."..string.sub(data,-5-i))
--[[]]end
--[[]]lkini.Write(sec, testkey, "") -- blank data, and check second key
--[[]]if gkini.ReadString(sec, testkey.."%"..baseindex + 1, "") ~= "" then return false end
--[[]]if lkini.Write(sec, testkey, data, 1) ~= false then return false end
--[[]]console_print("lkini: Write/Read passed. Tested across boundry of lines 1 and 2.")
--[[]]return true
end

It should be noted that entries in the config.ini cannot be deleted programmatically. You must manually edit the config.ini to delete lines from it.

Basically this is why everyone uses systemnotes to store anything more involved than really simple switches. gkini sucks :(

SystemNotes is perfect for some types of data. Data that will only change when a specific character is online is a perfect example of data that is suitable to be stored using SystemNotes.

But some data is not, or should not be, character specific, and SystemNotes is not the appropriate method to store that data.

If all the data that is appropriate for SystemNotes is stored by that method, that should leave a relatively light load for gkini.

I am seeing a lot of plugins limited themselves to offering only character specific configuration options, because they refuse to use gkini. If you are only storing a few dozen bytes, there is nothing wrong with gkini.

Further, you are limiting the usefulness of SystemNotes, by refusing to use gkini where it is appropriate.

For example: If I want to make a plugin that will display my inventory across all characters, then I should store each character's inventory using SystemNotes, in a format where it can be read using dofile, and I should store a list of character names in config.ini. I can then use the character names to build the path names to read the SystemNotes files. SystemNotes carries the majority of the data load, but only for the data it is suited for. config.ini carries the data load that only it can properly service.

The config.ini files should not be overloaded. But the current trend is significant under utilization of config.ini. I wish to correct this trend.

SystemNotes should be the primary method for all data for which it is appropriately suited. config.ini should be reserved for data for which SystemNotes is not the appropriate method.

These two methods properly combined in concert are far more versatile that being limited to a single method.

05:10AM draugath
It should be noted that entries in the config.ini cannot be deleted programmatically. You must manually edit the config.ini to delete lines from it.

draugath,

I feel that config.ini file is central to the operation of the vo client, and should not be unnecessarily overloaded with data that could be serviced otherwise. However, I find that your argument that entries can not be deleted as reason to premote the SystemNotes method to be somewhat flawed. I am not aware that a plugin can delete SystemNotes files. At best, you have moved the problem away from the more critical file. My argument is that your argument is too extreme. I ask you to modify your argument to: "config.ini should not be used for data for which SystemNotes can properly service."

@arf: The only thing that really sucks about gkini, is the data length problem, for which I am presenting this solution.

Thank both of you for your input. I have added a note in my original post that I hope will address everyone's concerns about this issue. Please review it and let me know if it should be refined or worded otherwise.

Read my post again. I made no argument against using config.ini for storing data. I was merely pointing out one of the problems with using config.ini, especially if you start saving large amounts of data and the possibly clearing it later.

After actually reading the code, I will now make an argument for fixing the code. Currently it appears that your code will split long strings and read them back, but it doesn't appear to make any allowance for deleting that data. Obviously you.can't actually delete entries, but it should be possible to easily replace such split string keys with the empty string.

Further, since config.ini is where all of the game settings are saved, I view anything that pollutes the config.ini in this manner to be unnecessay. All it does is increase the chances of corruption, not mention making it difficult to find legitimate config settings.

Targetless, a good example of a plugin that would benefit from a shared storage location, can easily generate hundreds of kilobytes of data, if not megabytes. Targetless is also a good example of a plugin that completely defeats the games memory storage by forcing a ReloadInterface() on logout. If it didn't do this, you'd be able to share data structures between characters.

I agree there should be a better way to share plugin data between characters, but quite frankly I think this is irresponsible.if Ray ever gets around to reviewing the plugin loader I wrote, it can support safe plugin-specific saved data.

draugath, per your suggestion:

Posted version has been updated:
-- lkini version 1.1 -- blanks out excess keys (if any) on write.

As far as sharing data structures between characters, when you log out, write all your data using SystemNotes, and update a config.ini key to point to that one character. After the interface reload, reload all the data you shoved to SystemNotes. In this case, you do not have to limited the SystemNotes method to data unique to that one character. You can actually shove all data to the SystemNotes of whatever character is being logged out, as long as you update the config.ini key to point to the latest character. It will leave an obsolete copy of the SystemNotes file on every character logged before. But who cares? Those files will be recycled next time those characters get logged on/off. The only real limitation to this method, is that you can only write the data when a character is online.

bump.

version 1.2

I am preparing to work on a SystemNotes module, and I will be using this code to support a directory structure, where config.ini holds character names, which is what you need to find SystemNotes files when the character is not logged in.

Edit: Dec 27, 2013: -- lkini version 1.3 -- Added a size limit argument to lkini.Write()
ex: lkini.Write("%plugins%", "akey", "123456", 5) -- will not write anything, will return false.

After reading this post, I started thinking a little more on the topic of SystemNotes, and maybe what I found is what you were referring to.

It turns out you can dofile() or loadfile() a systemnotes file from other characters. I'm not sure how this got missed all these years, or maybe it was just something that was discovered and kept secret. You still can't, to my knowledge, write to other character systemnotes. If you store the name of the last character to store data for a plugin, and craft the stored data to be valid lua (ie have it return a string), you can effectively have a client-wide data store that doesn't require using gkini.

It's still not an optimal solution, since it can leave multiple copies of data around in varying states of completeness. Also, if the config.ini gets hosed (which has happened to me before) you won't necessarily know which character to load from to get the most recent data. Though, you could always just iterate over the characters on the current account, using a datestamp to identify the latest data.

If i am logged onto the character "DE-1409y", and at the console I type

/lua SaveSystemNotes( "return 'test data'" , 1.44444444444e+069 )

which will write "return 'test data'" to file "settings/DE-1409y/system1.44444444444e+069notes.txt"
then later even when logged onto some other character, or no character at all i can type

/lua console_print(dofile("settings/DE-1409y/system1.44444444444e+069notes.txt"))

which will print "test data" (without the quotes) in the console.

you can only write the file when a specific character is logged on,
but if you format it as a dofile, you can read it anytime.

but you need to know the path, and the path requires you know the character name.
so i wish to use config.ini to store character names.

it is not a perfect solution, for data that may change independent of a given character.

it is a perfect solution for data that will only change when a specific character is online. character stats and inventory are perfect examples of data that literally can not change, except when the relevant character is logged in.

if i wish a consolidated inventory of all characters, then i will need to record this data in a SystemNotes file, per character, where each file only contains the data for that character. I will then need to merge all the files in memory, to form the consolidated list. If any character data changes, only that character's file needs to be updated. I will have to merge the data again to update the consolidated information.

using SystemNotes to store non character specific data can be done, but does become more complicated. I leave that for future work.

My first priority is to maximize the use of SystemNotes for the type of data it is best at. This will require the use of config.ini

Right. To extend that idea, as I suggested, if you include a time-stamp in the data that's stored in system notes, you can iterate over the characters if the last character is not stored in config.ini. Assuming that the data could be updated by more than one character.

@draugath: I have considered that, and other methods, to extend the use of SystemNotes. At some point, I may be interested in receiving input on the subject. Currently, I am putting that subject aside, while I develop the infrastructure to pursue it. Thus far, I have considered our conversations here reasonably pertinent to this thread. But at this point, I would like this thread to revert to being about the use of config.ini in general, and peer review of my code specifically, with the topic of using SystemNotes to branch out to some other thread. If you start such a thread, please include cross links from here to there, and there to here. Thank you for your continued input.