Print Page | Close Window

Optimize your packets!

Printed From: Mirage Source
Category: Tutorials
Forum Name: Approved Tutorials
Forum Discription: All tutorials shown to actually work with MSE are moved here.
URL: http://ms.shannaracorp.com/backup-forums/forum_posts.asp?TID=49
Printed Date: 20 December 2006 at 6:01pm
Software Version: Web Wiz Forums 8.01 - http://www.webwizforums.com


Topic: Optimize your packets!
Posted By: Sync
Subject: Optimize your packets!
Date Posted: 07 February 2006 at 6:26pm
Originally posted by Verrigan

Difficulty Hard 5/5

Packet Lag.. I've spoken about this before.. I submitted a tutorial on how to decrease the lag caused by the server sending large packets.. Today, I am going to try to explain how you can optimize your packets. (Dave, and anyone else for that matter, you are welcome to add to this tutorial if you like, but I will be placing the final tutorial in this original thread.)

Let's get down to business. Packets are the most important part of any online application. If you didn't know this, consider that there has to be communication between the server and its clients before your application can be online. :) If you don't have communication between the two, you defeat the purpose of creating an online application. This communication can happen in several ways.. Here are a few:
    [*]MySQL - Allow server and client both access to a MySQL database to communicate with each other.. Both will have to read the database periodically to check for new messages.. (Not a very secure way to do business)
    [*]Message Files - On a LAN, you could have a file (or files) on a shared drive to store messages.. Both server and client will need to access the file periodically to check for new messages.. (Also, not very secure)
    [*]Sockets - You can send messages via sockets either in TCP (checks the receipt of the sent messages) or UDP (Doesn't check the receipt of sent messages). Server/Client wait for messages to arrive, and processes them. (Can be unsecure, but there are ways to keep good security..)


Since this is a tutorial for MSE, we will discuss how to use TCP to send messages back and forth. Currently, both server and client send strings (text) back and forth as their packets. We split the string using a separator character, and then check the packet type by comparing the first string. (The text before the first separator character) That string can be any length, and the number of bytes it uses is the length of the string..

The Current Way

For this example, I'm going to use the "MAPDATA" (MD) packet. So, we have the MD packet, and what do we know about it?

Well... With a base MSE, the MD packet can be from 2,677 bytes and up. It actually has a maximum size, but we're not worried about that right now. :P The size of the MD packet varies with what tiles/NPCs/etc it has on it.

Okay.. Why?

Think about it.. If you send a tile as "1" it uses 1 byte. But you need to think about the SEP_CHAR. So that's 2 bytes. Okay.. What if that tile is 10? Now you're using 2 bytes to send a byte's worth of data.. Add the SEP_CHAR, and it becomes 3. Now consider tile #200.. (All the way up to 255) You send 4 bytes of data to show 1 byte's worth of data. If your number is above 255, then you're sending an integer's worth of data.. (2 bytes), but it's still represented by 4 bytes+ in a string.

As Dave mentioned, each character in a string is a byte, so a 10 character string is equal to 10 bytes. This is how the packet is sent across the internet. You can pick it up on the other end as a string or as a dynamic byte array. Dynamic byte arrays work differently than static byte arrays.. See my prior tutorial on this for a description of how to declare dynamic arrays.

However, now we get to our memory issues. See, VB doesn't store strings as 1 byte per character. So when you receive the packet as a string, you're (behind the scenes) converting that data to UniCode. What this means is.. You are using 2 bytes per character (in memory) for each string.. So a 10 character string is using 20 bytes of memory. (Confused yet?)

So.. that 2,677 byte string you received is using up 5,354 bytes.. (Double what it needs. :P) Great.. Twice as much memory is being used than is necessary. And don't forget the extra 10 bytes used by variable-length strings.. (Yes, the MSDN library apparently lies about this data type's size.. It's actually almost twice the size it claims) Do you think that might be why it takes longer to compare strings in VB? (I don't know, but it looks like a start.)

Before I move on to the next section, I want to assure you all that UniCode has its place in VB. Strings are not bad. They are very useful. But, if you don't need to use UniCode, it is better to use a byte array. :)

A Better Way?
You be the judge. I am just here to provide you with the information you can use to optimize your packets. It's tough to make the changes, and very time consuming. (It took me around 2 weeks to finish just the server-side packets.. I will gladly make those changes available as a download, once I finish this tutorial, for those who would like an example of how I did it, if enough people ask, and Shannara does not mind.)

For the most part, VB takes longer to compare strings than it does to compare numbers. Why? One reason could be because there is more bytes of memory to compare.. I won't kid you.. I don't know every aspect of why VB takes longer to compare strings.. I just know it does. If someone wants to elaborate on this, please feel free.

So, since strings take longer to compare, it makes sense to compare numbers, right? Okay.. How do we do that?

This is where the byte array comes into play. Luckily Winsock is on our side here. When you receive a packet with winsock, you can choose to either receive that packet as a string, or as a byte array. If you choose to receive that packet as a string, VB automatically converts that string to unicode for you. If you receive it as a byte array, it requires no change.. (Ooo.. saving a little time already. :) Don't go crazy.. You're only saving about .001 milliseconds for a 10 character string.. Yes, that was a guestimate. :P)

Okay, great. So we can receive the data as a byte array. Why would I want to receive it as a byte array? Here's the short answer: You will save bandwidth (tons of it over time) using byte arrays.

Okay... I am going to make the long answer as simplified as I can.

Let's recap. Strings use twice as much memory (in bytes) as the number of characters in them. As Shannara pointed out in his reply, "1" has a Len() of 1, but a LenB() of 2. Packets are sent across the internet as a list of bytes, whether you want them to be or not. If you send a string "1" over winsock, it is transmitted as asc("1"), or the byte value 49. When you catch that packet on the other side, you can either grab it as a byte, or as a string. If you catch it as a string, it is automatically converted to unicode. That's just a fancy way of saying it adds another byte value of 0 to it. You might ask yourself why that is.. And that's fine. Go visit msdn.microsoft.com for an answer, cause I'm not here to explain unicode to ya.

Now, when you send data back and forth between a server and client, you need to consider a couple of things. The first thing you need to consider is that packets are not always received exactly as sent. Oh, the order is still the same, but it might be broken up, making your 1 packet into 2 or more packets, so we have to have some way of being able to tell when our packet has ended. Another thing you need to consider, especially in a multi-user server/client environment, is how much data is being passed back and forth. The less data sent, the better. More data = lag for both client and server.

How do you tell when a packet ends? If you use a string, you can end each packet with a special character, or series of characters. But then, you also need to have some way of separating multiple pieces of data in those packets.. So your string is likely to end up being larger than necessary. If you use a byte array to send your data, you can have the first few bytes of that byte array tell the receiver how many bytes (after those) to wait for before processing the packet. In my method, I use 4 bytes, which is the size of a long, which would allow for a packet size of over 4 billion bytes.. (A little over-kill.. You could easily use 2 bytes, which allows for a packet size of over 65,000 bytes.)

Now, onto the next consideration.. The size of the data. If you use a string, you know that you have to send your numbers in the string, and as described above, you would end up using more bytes to transfer values that actually take up less space, so the smallest way to send data would be to copy the information you'd like to send into a byte array.

So now we know how to tell when the packet ends, and how we should send the packet. For this example, I will use the USEITEM packet. The old way, we send the packet as:
"USEITEM" & SEP_CHAR & InvNum & SEP_CHAR & END_CHAR

This way uses 11-12 bytes to send the packet.

With my way, I create a byte array called Buffer, and add the necessary data to it. In this case, the USEITEM packet would be: 3, 0, 0, 0, 15, 0, 11 (Breaking this down, the first 4 bytes show a long value of 3. The next 2 bytes show a packet type of 15, an integer value, for USEITEM, and the inventory number, which is 11 in this case.) A total of 7 bytes.. Saving 4-5 bytes. :) Now, you could easily use an integer (2 bytes) to show the length of the packet, and just 1 byte to show the type of the packet, which would drop the size of the packet to 4, saving 8-9 bytes out of that 11 we used in the strings.

So, when the server receives the packet, it can then use CopyMemory to move the first 4 (or 2) bytes over into a long (or an integer). Then, it can copy the next 2 bytes (or 1) into an integer (or byte). And finally, copy the last byte into a byte variable, and handle that information as it used to. :)

Below is a link to my modified copy of MSE. This MSE has the following things done to it:
    [*]The server-side packets have been changed to byte arrays.
    [*]IOCP has been added to the server via the JetByte COMSocketServer class DLL. (This was the same code I used to setup the IOCP tutorial, but the WinSock control works on the same principle, so you do not have to have IOCP to convert your packets to byte arrays.)
    [*]I have implemented a "call-by-address" method, and separated each packet into a separate packet handling function on the server-side. This gets rid of the if-thens in HandleData.
    [*]I have added modBuffer.bas to handle all the buffer (byte array) manipulation.. (AddByteToBuffer(), AddIntegerToBuffer(), GetByteFromBuffer(), etc.) This file is used on both server and client-side.
    [*]I have added modMessages.bas to enumerate all the server-side messages, and for future placement of the enumerated client-side messages. This file is used on both server and client-side.


***** YOU WILL NEED TO REGISTER THE COMSOCKETSERVER.DLL FILE TO BE ABLE TO USE THIS EXAMPLE! Do this by going to Start/Run, and typing: regsvr32 <PATH>\COMSocketServer.dll

Get your copy of my modified MSE http://www.verrigan.net/downloads/MSE-Verrigan.zip - here . :)

[edit]
Forgot to mention that I changed the packet length to an integer, and the packet type to a byte in the download. :P




Replies:
Posted By: Sync
Date Posted: 07 February 2006 at 6:26pm
Approved.




Print Page | Close Window

Bulletin Board Software by Web Wiz Forums version 8.01 - http://www.webwizforums.com
Copyright ©2001-2006 Web Wiz Guide - http://www.webwizguide.info