Achieve uber-speed! Convert to binary fil
Printed From: Mirage Source
Category: Tutorials
Forum Name: Approved Tutorials
Forum Discription: All tutorials shown to actually work with MSE are moved here.
URL: http://ms.shannaracorp.com/backup-forums/forum_posts.asp?TID=46
Printed Date: 20 December 2006 at 6:01pm
Software Version: Web Wiz Forums 8.01 - http://www.webwizforums.com
Topic: Achieve uber-speed! Convert to binary fil
Posted By: Sync
Subject: Achieve uber-speed! Convert to binary fil
Date Posted: 07 February 2006 at 6:17pm
|
Originally posted by Dave Well I was so frusturated by not being able to post in a lot of forums that I made a second account... People complain that their server is slow, they wait 15 minutes for it to boot before being able to play, they send all those slow packets, they wait for saving all players, sometimes locking their computers and everyone elses for two or three minutes on end! Why live through this when you can fix it?! The reason saving players (among everything else) takes so long is because: (drum roll please) You're writing an INI file, all text, and its about 15 KB of file to write. Text is already slow in VB, even when working with small ammounts of it. If you havent noticed, you also need to convert all those values INTO text before you can save them, and then you need to convert them back to numbers again when you load them... LAME! (also slow-lisious) What's even more lame? About 1.2 KB of that 15 KB is actually being used by the server when you load the account, compared to the 15 KB that you write when you save... WASTE! So, you may ask, how can I make this less lame? The easy answer would be to not use INI files... right? Right... right. Really, GetVar and PutVar are wonderful to use! THEY REALLY ARE! They're awesome for storing and saving... but they're just way too slow to use them here, where we're writing a lot of data and need it done very quickly. They're good to use when it's a one-time load... not when you're loading and saving in a loop for all eternity! (eternity does, infact, last longer when you use getvar ;)) So... we move on to alternitives. Ok, we want to store data but not use the lovely (and convinient) GetVar and PutVar thingies. What alternitives are there? Text file, Binary File, or database... that I have researched ;) I'm sure there are other ways to do it. Anyways, lets learn a little bit about each... Text files would be nice, they store the data and are still easily edited. Still, however, we have some of the major problems with GetVar and PutVar, we need to write a string, and we need to convert the string back to a number when we load it in. That = slow. Also, all that header information is stored, like Name= Sprite=... that stuff is lame... and takes up space that we REALLY don't need to use. That's why the INI files are so huge in the first place... Binary files are great, you gain the ability to store numbers and strings in the same file without changing format, however you need to store an extra peice of information with the strings(the length, specificly, however that's not an issue with our accounts, don't worry about it, I'll explain later.) Tell me, how many strings are stored per account? Well... 5. Login, Password, Name 1, 2, and 3. Not much, considering all the other numbers (what 700 of them?) that are stored. Well worth the little hastle that you get with those five strings, don't you think? :) TELL ME NOW WHY YOU CONVERT 700 NUMBERS INTO STRINGS?! ITS LAME! DONT DO IT OR I WILL EAT YOUR CHILDREN!! The fact of the matter is, when you get into big numbers with lots of repetitions, even the fastest functions take time... The Val() function returns the value of a string as a... integer? ITS SLOW! and you eliminate 700 calls to it if you store files as binary... per account load... even more if you convert the other data files! :) Databases are more complicated, but they are the fastest of all choices. You load the data straight from RAM instead of getting it off your hard drive. Other than that, they're pretty similar and share all the advantages to and with binary files that I know of. The other advantage (or disadvantage in some cases ;)) to using binary and databases is that you can't just open up a text editor and use it to edit the account (or other file) You need to use either a hex editor, or write up your own custom application to use as an account editor on your custom file format data (wow, isn't that professional! :)) Knowing this, what file type would you choose to use? INI, Text, Binary, or use a Database? We choose Binary (and if you didn't, too bad!) Following this thread I will post a tutorial to convert ALL of Mirage Source Engine (Build 1) to use binary files. This includes Classes, Items, NPCs, Shops, Spells, Accounts, even the MOTD! In otherwords, everything but the banlist... and I might even convert that later down the line. When you convert these files into binary format, keep in mind that you will no longer be easily able to edit these files. You will need to create your own application to edit them, and I will not aid in the creation of those :) For this reason, do not attempt this tutorial if you don't have the skills (or patience, it's a lot of repetition) to write your own editor. To give you an estimate on how fast this will speed up your server, my server boots up in about half a second, loading 1000 maps and the default ammount of everything else. The only thing that takes any time is the maps, because you're constantly opening and closing files... this can be fixed by putting all the maps into one file, that of which I will not cover here (mainly because I havent done it yet, myself ;)) Are you ready to undertake the, perhaps, most frusturating thing I have ever done in my entire life? Oooh yeah, I'm making you do it yourself, dude(ette), I'm not handing this crap away... you must know the pain it caused me! MUAHAHAHA! I hope to present it in a manner that you can achieve this yourself with only half the headaches that it caused me :) Get ready to cripple your development, ladies and gentlemen! After you complete this part of the tutorial, you will be pretty-much unable to test the rest of the stuff until you create an account editor. The reason for this is to edit any of the data files in game, you need to have access... how do you add access if you're the only account and you dont have it? You write an account editor :) Why do I do accounts first? Believe it or not, they were easy (er) to convert than the other files, and it supplys perhaps the best gain in preformance (saving players). Again I must stress... DO NOT ATTEMPT THIS TUTORIAL UNLESS YOU ARE COMPETANT IN VISUAL BASIC! Being the first conversion I am assisting you with, I will give you most of the code in an easy to understand way, the other files will not be handed away so easily. For the record, if anyone asks where this code goes, I will eat your children because that means you didnt read where I say right here: it's all server side, and almost entirely in modDatabase. Ready? OK! Lets start with the SavePlayer sub. First, you need to add one more variable to the list of defines. It is an integer called nFileNum and it is used to store the file number of the account that we're editing. Next, you see where FileName = "ABunchOfCrap" & ".ini" Change that .ini to something less lame, like .bin or .dat or how about you get the two letter abbreviation for your engine and then add an A at the end standing for account! Yeah, that would be cool! (Example for Playerworlds: Account.pwa) Or I guess you could do something lame, like .porn or something... then if someone made the account "ILove" it would be stored in a file name called "ILove.porn" haha, could be funny :) Anyways, make the extention practical. Well now we go back to a binary lesson. We don't have those nice PutVar subs anymore. That took care of opening the file for us... so lets open the file so our program can use it, shall we? Note: You can leave this step out, but your server wont work... Ah, before I forget, comment out every line below FileName =. That stuff is un-needed anymore, and we'll write it all again later. Well before we start writing to this file, we needa get rid of it... The reason we do this is becaue binary files, when writing, you pretty much 'pile on' the data. So we do this: If FileExist(FileName, True) Then Kill FileName Next, we need to make it so our file number, nFileNum is actually open to use. To do this we use: nFileNum = FreeFile FreeFile means new file... and if it doesn't, I don't care because it works! =D We're ready to open our file, so now we can tell the program to open it, as what type, and what file number to open it with... Open FileName For Binary As #nFileNum Self-explanitory, right? We're opening the file to use as a binary file with this number. Coo :P. So now our files open! YAY! Let's write! Put #nFileNum, , Player(index).Login WAIT! That's a string! Remember what I said about strings?! Here's where I explain it to you. If you right click on Login and click definition, you can see the declare for it: Login As String * NAME_LENGTH It's fixed-length, that string is NAME_LENGTH characters long, no matter what... so we don't need to worry about storing the length because VB already knows it is 20 characters long! ITS MAGIC! Anyways, if this string was not fixed length, we'd have to set it up a little different. I'll show you how when we get the MOTD. What else haven't I explained? Oh, right the code I just posted :) That Put says that we're 'putting' information into the file, the #nFileNum tells us that we're writing to file number nFileNum. The # is is there because it is... don't ask questions. The next value we can pass into the function is blank in our case, that value is the starting byte of where to write. When left blank, it states to start writing where we stopped writing last... because we've never written anything, it's 0. You do not need to worry about this either, for accounts, I will show you later when we get the password. :) We're leaving this blank. The next value is what we're writing. This can be of any variable type and no matter what it is, other than string, it will automaticly pull out the right ammount of bytes! Coolio! It does it all for us, a lot like the putvar function! Anyways, put the password after that, and then we move onto the loop for characters... For i = 1 To MAX_CHARS Follow the patturn just like the PutVars took, now that you're all set up, it's just that easy. So we get to the bottom, these loops will work in there too, so don't think you need to change them.
After all our writing is done, we need to close the file, done simply with one line of code :) Close #nFileNum Guess what! SERIOUSLY GUESS! That's all for that sub... now how do we load this behemoth pile of numbers that we created? (we literally just created a pile of numbers, with no rhyme or reason... to the average noob... there are no labels in there at all, saving TONS of space! :)) Now we move on to loading! hoo-rah! (loadplayer sub) The thing to know with binary files is that you need to load them in the same order you wrote them, which shouldnt be a problem... First you need to declare a new variable at the top of the sub... can you guess what it is? We'll see because Im not telling you! HAHAHA! Call ClearPlayer(index) Wow... so far our changes are EXACTLY like they were in teh saving sub, oh where could the difference possibly be? Get #nFileNum, , Player(index).Login It works the exact same way as Put... but instead of putting, guess what, it gets! woo! Follow that patturn again, complete the sub yourself. Remember to close the file at the end. Close #nFileNum. Onto AccountExist... change that file extention to your cool file extention. PasswordOK... we need to break out that file again... sigh... time to learn something a little more complicated... slowly :) Two declares at the top this time, one to hold the password we get out of the file and one to hold the file number. Dim RightPassword As String * NAME_LENGTH Notice that it's a constant length? This prevents us from needed to store the length of it with us. Convinient :) nFileNum = FreeFile Open that bad boy up, inside the If AccountExist statement.. Remember to change your file extention to your cool file extention, because .ini is lame! Now we get to the slightly new part. Get #nFileNum, 20, RightPassword This takes the string of length NAME_LENGTH (defined by the length of RightPassword) from byte 20 and stores it into RightPassword... .got that? It's tough... How do I know to start at byte 20? I counted. I know that the password is stored after the login, which is also a constant length string of length 20 bytes. It uses bytes 0-19 so the first character of the password must be stored on byte 20. The function automaticly does the rest :) Close that file, check to make sure RightPassword = Password and then get out of here. AddChar sub, this is beside the point, but comment out that SavePlayer() call there... it's not needed. Find where AddChar( is called, and you'll see that right after you call it, it saves. You save the player twice... effectively doubling time time it takes to save a new character... because... you do... it twice... Hey dudes, guess what... you're done! All your accounts are now stored in binary format. They load quicker, they save quicker, and they're much smaller... what could get any better? -laughs- Go write yourself an account editor. Next we're doing classes! This gets a little strange, but is easier to do than the accounts. Only two subs we need to modify in this quick and easy change. Start by finding LoadClasses Change the FileName extention to something cool... like... an example for PW would be Classes.pwc Now we need to open the file, remember how to do that? It's this code: nFileNum = FreeFile Then instead of GetVar, replace it with the Get statement, Get #nFileNum, ,Max_Classes Replace all the other GetVar stuff with the Get stuff. Close the file at the end, Close #nFileNum That's all, next sub, saveclasses Change the file extention, Check to see if the file exists, and destroy it if it is... If FileExist(FileName, True) Then Kill FileName Open the file. Now, here we do a little check just to be safe. Max_Classes, if it is 0, will crash our server when we boot. Therefor I threw in a check, If Max_Classes < 1 Then Max_Classes = 1 Change all those PutVars to put statements, then close the file. Go into CheckClasses and change the file extention, also add a call to ClearClasses() before you save them. You can then get rid of the clearclasses() call that's in the LoadClasses sub. That's all, your classes are saved binary now. Congratulations! Go write yourself up a class editor :) I'm going to use the oprotunity of the short tutorial here to have you do some other htings, as well. this is mostly a global replace thing. NONE OF THIS IS NECESSARY, HOWEVER IT BOOSTS PREFORMANCE SLIGHTLY! First, press control F and find all cases of "". (two quotation marks). Replace them with vbNullString. There are a few that you want to keep "". Don't do a global replace. Find stuff like Mid() function calls, Left(), Right(), Space(), Chr(), Trim(), STR(), LCase(), UCase() and stuff, almost anything that works with a string, and replace them with Mid$(), Left$(), Right$(), Space$(), Chr$(), Trim$(), STR$(), LCase$(), UCase$() Before, all those functions were returning a varient data. That takes a little mroe memory and is just a little slower. Adding that dollar sign made them only return a string, and sped the function a bit. Good luck with your other MS programming stuffs, next in like for converting is ITEMS! -Dave |
Replies:
Posted By: Sync
Date Posted: 07 February 2006 at 6:18pm
|
Originally posted by Dave Before we begin and go any further, I'd like to affirm the idea that you can only do one of these and it will still work. That's why im splitting it up. Each post I make should be considered an independant tutorial that builds on the previous one. You can do one and not the next, you can skip one and move on, and you can not do any, for all I care. ----------------------------------------------- Let's look at how items are saved, exactly. There's the SaveItems() Sub. This is a sub that loops, caling SaveItem() on each item. There's the SaveItem() sub. This sub actually saves the items, but does only one item per call. There's LoadItems() Sub. This loads all the items into memory. Lastly, there's CheckItems(). This makes sure the file exists and creates it if necessary. Before, in the other conversions, we wrote the entire file in one shot. Now, we need to be able to start writing an overwrite data in the middle of the file. Before now, you wouldnt be able to do this! Well, the only way we can possibly know where to start writing is if we know the SIZE of each item, in bytes. Let's start in the SaveItem() sub. Add two declarations to the top, StartByte as Long and nFileNum as Integer Change the file extention. Set nFileNum = FreeFile Open the file to write. Change the putvar crap to Put crap :) Close the file, Close #nFileNum. Now, we havent used StartByte yet. We're going to add that now! Remember the second spot in the put function that we were always leaving blank? Well, thats the top secret part that lets us write (and read, mind you) in the middle of a file. All we need to know is the byte number of where to start writing, and we can start writing there! Sweet! So now the puzzle comes in, DO we know the byte? Of cource we do! =) Every item is exactly the same size, lets go look at the ItemRec and all those variable declares.
In the rec, we have 4 integers, 1 byte, and a fixed-length string of 20 characters. Longs are 4 bytes big Integers are 2 bytes big, Bytes are 1 byte big, Strings are 10 bytes, plus string length. Fixed length strings are their length :) Knowing this, I can tell you each shop is only 29 bytes big. Multiply that by (ItemNum - 1) Add 1 to the rest, Reason being you need to start on byte 1 if you're using shop 1. Do a little check, if item number is 1, do you get 1? If itemnum is 2, what do you get? 30? Make sure it works out! Add StartByte into the FIRST call of Put, after that you no longer need it.
From now on, you need to calculate StartByte by yourself :) LoadItems time. Change the file extention, open the file, get the file, close the file, ba-dabing you're done. No messy crap here :) CheckItems you need to change the file extention and add a call to ClearItems() before you check if it exists. That's all, folks. Next will be shops, the hardest one of them all! Alright, let's do shops. SaveShop sub, Change your file extention, Open the file, remember to set nFileNum = FreeFile first. Calculate your shop size, remember how we did that? Byte = 1 byte Integer = 2 bytes Long = 4 bytes String = string length + 10 Fixed length string = string length. Count 'em up  This equation gave me the most trouble, I made it way harder than it needed to be, in the end my equation was two lines long. Set your StartingByte number, then convert all the PutVars into Puts like always. Remember to put the StartByte in only the first call of Put. Yay! Move onto LoadShops, do all the same stuff. Change file extention, open, convert to gets, close and that's all. In CheckShops put a call to ClearShops() before you check if the file exists. Change the file extention NPC time! Save NPC sub, you gotta change the extention, open up that file, and calculate the starting byte. Then you Put all your data, just like we've been doing, and then close the file. Remember to put the startbyte number in only the first put call. Close the file. LoadNPC sub, change the extention, open the file, get the stuff, and close Sure is getting easier, eh?! Actually, as you learn more about it, I'm giving you less and less information. You've noticed im sure.Look, I'm typing this up and I dont even get a scroll bar on this one! In CheckNPCs change hte file extention and add a call to clearNPCs before you check if it exists. |
Posted By: Sync
Date Posted: 07 February 2006 at 6:20pm
|
Originally posted by Dave So im kinda getting lazy, and bored of doing the same thing over and over. Let's spice it up. MOTD time! Onto the SendWelcome sub, it's on modServerTCP. Sub SendWelcome(ByVal index As Long) Ok I got rid of the GetVar there, and I made a function called GetMOTD in ModDatabase... I figured we should keep all the saving and loading to one module :) Look how fun, and how much less code there is. horay! So let's write the GetMOTD function... how? Well lets go back into modDatabse, and make a new function. It's a function, it accepts no values and it returns a string. We need a few declarations to start off with Dim FileName As String Uhoh... something new... msgLen... interesting, what coudl that be for... Lets set FileName right off the bat, make it like... motd.bin or something. Next, I checked to make sure the file existed. If it didnt, I said that they should type /motd to set htier own... like so. If Not FileExist(FileName, True) Then Alright, after that little check, we can open the file... Now you'll learn this later, but when we save the file, we're going to save the length of the MOTD as an integer, and then we're going to save the MOTD itself. This is the way you have to save variable-length strings. So first, we get msgLen, then we make it so MOTD is msgLen big, with the Space$() funtion, then get MOTD.
After that, do a check to make sure the MOTD is actually somehting, and if it isnt, make it so. If MOTD = vbNullString Then --------------------------- SaveMOTD sub
Only one thing new here, msgLen = Len(NewMessage), gets the length of newmessage... so we can save it. Open the file, put msgLen, then put NewMessage and close the sub. Find where the MOTD is set, search for MOTD. It's in the modHandleData. Instead of PutVar, call SaveMOTD instead. That's all folks. This concludes the tutorial, however tomorrow I will include some 'bonus material' tomorrow, saving the maps into only one file. Enjoy! Please let me know of errors in my code, problems with my wording and stuff like that, hope you like the changes! -DAve Since I'm bored, crabby, a little pissed off, and tired, I'll be nice :) HEre's all the code for the maps-in-one file stuff. You should understand it all by now, and you should realize suddenly that you're stupid because you didn't figure it out yourself :) Sub SaveMap(ByVal MapNum As Long) Sub LoadMaps()
That's all for this extremely long winded tutorial! Enjoy, DAve |
Posted By: Sync
Date Posted: 07 February 2006 at 6:20pm
|
Approved .... I think ... |
Posted By: Stillborn
Date Posted: 07 February 2006 at 10:41pm
|
Good stuff Even though you didnt spell out everything I would say you explained everything good enough that any n00b could get this done and possibly understand it well enough to make an editor O.O Still waitin for those headaches =/ |
Posted By: Stillborn
Date Posted: 07 February 2006 at 10:48pm
| oh and I forgot I think freefile means next available file number :) |
Posted By: Dave
Date Posted: 07 February 2006 at 10:52pm
| It's been a long time since I wrote this tutorial, it could use an update :P I know a lot more now. |
Posted By: Stillborn
Date Posted: 09 February 2006 at 9:56am
| I'm working on the editor now which when its finished I will most likely release the source for so everyone can use it. One thing to remember for people new to VB is that you will still need a good ammount of experience to use this because you are going to have to still modify it to fit your needs. But if you understood this tut it shouldn't be a problem. :D |
Posted By: funkynut
Date Posted: 09 February 2006 at 10:59am
| heh, if your learning vb and wanna learn file i/o to save stuff then this is pretty much the only method your really gonna need. Dont bother with inis, personally, I think their more hassle then their worth, only benefit is that someone can edit the inis in notepad or something. |
Posted By: diablo
Date Posted: 24 February 2006 at 3:20pm
|
wait...if you need an account editor then wouldnt that make it a hassle adding in to a project?? You would have to send the variables containins info to your server which sends it to the editor which makes the account..... wierd.. ------------- I am... the new Diablo98188 Past Names: Diablo98188 |
Posted By: Misunderstood
Date Posted: 24 February 2006 at 3:38pm
| no, thats only if you want to edit them by hand you need an account editor. |
Posted By: diablo
Date Posted: 25 February 2006 at 5:28am
|
Ohhh ok thank you, that cleared it up ALOT. Now im not so hesitant about doing this, i just need an editor lol. Thanks MIs and thanks Dave for writing this, and thanks to Sync for posting it here ^_^ ------------- I am... the new Diablo98188 Past Names: Diablo98188 |
Posted By: Misunderstood
Date Posted: 25 February 2006 at 1:43pm
|
to make an editor you just need to load it(which this tut does) then put each value into an appropriate textbox that you make for it, then to save it put each value you set in a textbox to the rec and save it(like this tut does) this would be for the separate program if you dont wanna do it clientside, you might be able to just rip out the editor in the client, and the EditorInitiate sub relating to the area(like the itemeditorsub) and modify it a little to load the values into all the areas. Get what im saying? |