Re: [Condor-users] condor, sqlite, python

[Mag Gam <magawake@xxxxxxxxx>]

Brian,

Really depends on what how many times you are doing this. If you need
to do this several million times then condor maybe the right tool for
you.

I also suggest you moving away from sqlite when working with "large"
data because its really not efficient. Perhaps have your data in a tab
 or ccommand seperated values which will make data manipulation much
easier.

You should also look into the UNIX join command it can do many things
to avoid a database.


If you can show us some same data it would be easy to create a sample
script or even a condor job...

hope this helps!




On Thu, Apr 1, 2010 at 12:21 PM, Geary, Brian W. <bgeary@xxxxxxxxx> wrote:
> Hello.
>
>
>
> We are beginning a research project which will by joining 2 million rows
> against 250,000 rows three times.
>
> We can either do the 3 joins in one query or do three separate queries with
> 1 join.
>
>
>
> We are using sqlite and python to execute the sql queries and there is a bit
> of logic in the python as well.
>
> We can use Windows or *nix for this project or both.
>
>
>
> Can someone explain if Condor would be appropriate for this research
> project? Would we just submit the
>
> php script as a job or some other scenario.
>
>
>
> Thanks..
>
>
>
>
>
> --
>
> Brian
>
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
>