« Return to Thread: read test data from file or database
Re: read test data from file or database
by Philip Aston-3
::
Rate this Message:
Marc Van Giel wrote:
> Thomas,
>
> Filebased is the way to go for cross-process data sharing. You could
> lock the data row with a combination of the processNumber and the
> threadNumber, to make sure that each row is used uniquely for each test.
>
> Another random thought, since this feature comes up frequently:
>
> Would the console be able to distribute test data? It already has a
> communication layer between the agents. Maybe a new feature could
> allow the console to manage the data (from files or database) and
> distribute it to the agents (in subsets by agent/process, on request,
> etc.). Distribute subsets initially on start of the tests would be
> best I presume, "on request" might have significant impact on the
> sampling. Just a random thought.
This should work already. Just put the files in the distribution
directory, they'll be shipped out the the agent caches. There's no
support for sending specific files to individual agents, but I think
that would overcomplicate things anyway.
There's no specific API to get the location of the cache directory, but
the worker process can read the files by knowing that the cache
directory is the first entry in the system path. E.g. (example modified
from Cal's FAQ entry):
import sys.path
users = []
infile = open("%s/users.txt" % sys.path[0], "r")
for line in infile.readlines():
users.append(line.split(","))
infile.close()
>
>
> best regards,
> Marc.
>
> On Thu, Dec 4, 2008 at 3:31 PM, Thomas Falkenberg
> <thomas.falkenberg@...
> <mailto:thomas.falkenberg@...>> wrote:
>
>
> Hi everybody,
> before I try to reinvent the wheel I'm asking you guys because I'm
> sure you've been there. I'm looking for an efficient solution to
> get test data (like "user_id") from a file or a database in order
> to use them in my grinder EJB-calls and without slowing down the
> agent, impacting the throughput. I guess a file would be the way
> to go. It should be made sure that each id from the file is only
> used once by each thread. So I'm thinking about reading the file
> in the memory(Hash? Linked List?), partitioning it depending on
> the number of rows and the number of threads and then serving the
> data to each worker thread. But what if I use several processes or
> agents? I could create data files for each process, but that's not
> very comfortable. So if anyone has a working solution, I would
> greatly appreciate it.
>
> That would also be a good candidate for the script gallery.
>
> Thanks and regards,
> Thomas
>
>
------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use
> Thomas,
>
> Filebased is the way to go for cross-process data sharing. You could
> lock the data row with a combination of the processNumber and the
> threadNumber, to make sure that each row is used uniquely for each test.
>
> Another random thought, since this feature comes up frequently:
>
> Would the console be able to distribute test data? It already has a
> communication layer between the agents. Maybe a new feature could
> allow the console to manage the data (from files or database) and
> distribute it to the agents (in subsets by agent/process, on request,
> etc.). Distribute subsets initially on start of the tests would be
> best I presume, "on request" might have significant impact on the
> sampling. Just a random thought.
This should work already. Just put the files in the distribution
directory, they'll be shipped out the the agent caches. There's no
support for sending specific files to individual agents, but I think
that would overcomplicate things anyway.
There's no specific API to get the location of the cache directory, but
the worker process can read the files by knowing that the cache
directory is the first entry in the system path. E.g. (example modified
from Cal's FAQ entry):
import sys.path
users = []
infile = open("%s/users.txt" % sys.path[0], "r")
for line in infile.readlines():
users.append(line.split(","))
infile.close()
>
>
> best regards,
> Marc.
>
> On Thu, Dec 4, 2008 at 3:31 PM, Thomas Falkenberg
> <thomas.falkenberg@...
> <mailto:thomas.falkenberg@...>> wrote:
>
>
> Hi everybody,
> before I try to reinvent the wheel I'm asking you guys because I'm
> sure you've been there. I'm looking for an efficient solution to
> get test data (like "user_id") from a file or a database in order
> to use them in my grinder EJB-calls and without slowing down the
> agent, impacting the throughput. I guess a file would be the way
> to go. It should be made sure that each id from the file is only
> used once by each thread. So I'm thinking about reading the file
> in the memory(Hash? Linked List?), partitioning it depending on
> the number of rows and the number of threads and then serving the
> data to each worker thread. But what if I use several processes or
> agents? I could create data files for each process, but that's not
> very comfortable. So if anyone has a working solution, I would
> greatly appreciate it.
>
> That would also be a good candidate for the script gallery.
>
> Thanks and regards,
> Thomas
>
>
------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
grinder-use mailing list
grinder-use@...
https://lists.sourceforge.net/lists/listinfo/grinder-use
« Return to Thread: read test data from file or database
| Free embeddable forum powered by Nabble | Forum Help |
