Optimizing performance when having a large datalist

Hi,
I am running a very extensive and expensive db search that returns (and it should return) a large set of results, let say around 1000 records. I splited the result page into two pages, one containing records grouped by certain criteria (a) and the other showing results inside one particular group (b).
I want to achieve two things. First is that I want to store this large set of results as either javascript array or an xml and second is that I want to split pages into frames. Basically results will be stored in a parent page, and there would be one child page in a frame that will store either page a or b previously mentioned. I will need to use paging for a and b and with this solution I believe I would not have to make repeated searches to retreieve results and I would not have to re-render this large chunk of data each time page refreshes (paging!). At the moment I have only one obstacle, and that is I am not able to access data located in a parent page from a child page using server side. I could only copy the entire result set inside a html element and then access it using server side, but this solution makes this entire process pointless since I will duplicate the entire result set.
Any tips?

You could try to store the results in Cache (or maybe even SessionState). I don't know how much memory that will take up ... but it will save you from having to hit the database multiple times, and you will be able to access it from either page. You could compensate for the amount of memory this would take up by setting the expiration and priority of the Cache element. This approach might even alleviate your need for frames all together. When the results are returned fill them in a DataSet, because it can be serialized and stored in Cache or SessionState. You could probably even code the results to be a property of the page like this:
public DataSet myResults
{
get
{
string CacheStringKey = "myResults";
if(Cache[CacheStringKey] != null)
{
// The data was already in cache ... just return it from there
return (DataSet)Cache[CacheStringKey];
}
else
{
// The item wasn't in cache, so get it from the DB, save it to cache, and return it

// TODO: Get the data from the DB here and use it to fill a DataSet

// Cache the item for 20 minutes, with a low priority. NOTE: DBResults is the DataSet with the data from
Cache.Add(CacheStringKey, DBResults, null, DateTime.Now.AddMinutes(20), TimeSpan.Zero, CacheItemPriority.Low, null);
}
}
}
That would make it easy to refer to the DataSet from inside the page, and would keep all this logic encapsulated.

It makes sence, but will still have to test for high traffic issues. Session is using server memory. What about Cache? Does it also use server memory or server disk space?
If you use cache, doesnt that mean each user who hits the search will get the cache from the previous user? If someone does hit search and is paging in the datalist then another users hits search with different criteria will that change the cache to the new users search?
Cache is availabe to all the users if the cache key for each user is a static value. If you assign different cache key for each user, than each user will access only a part of cache that was assign to him (unique cache key). The only bad thing about using cache or session is a memory overload if cache is not being cleaned frequently or to many users are trying to access/create cache.

Cache uses server memory, and depending on how much memory the server has and the virtual memory settings, it could also use server disk space. But, when the "results will be stored in a parent page" that is also stored in server memory.
In response to drock22's post: It would totally depend on how you implemented the search, and what the "results" were that you stored in cache. I would need to know a little more about what the search does, the criteria the user is able to set, and more about what this very extensive and expensive db search actually does and what it returns. mko, can you give us a little a more detail on that stuff?

Well, the search loops through entire catalog of items and returns an itemid and price for a given item. Search is based on startdate and enddate when services is being offered, price, etc. The most importand filter is date, and it is very difficult to believe that two or more users will start (at the same time or in 20 min difference) a search based on same dates! So I believe caching results would be pointless in this situation. I am developing solution using IFrame and array in javascript and I find it more appropriate since all the results would be stored on client.
The other thing that is important in here is would it be possible for user to access the results of the search in offline mode? I believe client has to be contected to a server in order to get results from cache, or i.explorer would retreive page stored in history that has already been rendered without having to access cache.
Sounds like you have it figured out. I didn't know if you were joining a lot of tables or using a cursor in order to generate the values that you are returning (which is what I think of as an "expensive" database search). Caching may not be as benefitial in this situation, but it is still an option. The only benefit here would come from cutting down on trips to the database. Example, if there are 20 people that perform searches ... your implementation would connect to the database 20 times. Using cache you would connect once, get the data that is common to all users (probably the whole table), and then filter it on a per user basis. When an individual user performs a search you could filter that table like this:
string strExpr = "Date > '1/1/05'";
string strSort = Price DESC";

// Use the Select method to find all rows matching strExpr, and order them by strSort
DataRow[] foundRows = myTable.Select(strExpr, strSort);
Your idea to use JavaScript and iFrames may work great, and especially if being able to access the results offline is a priority. I am not for sure that will work with JavaScript (but for for all I know it might), but I know the client would have to connect to the server to access its cache. Good luck, and let us know what you decide to go with.