multithreading - Python Socket and Thread pooling, how to get more performance? -



multithreading - Python Socket and Thread pooling, how to get more performance? -

i trying implement basic lib issue http get requests. target receive info through socket connections - minimalistic design improve performance - usage threads, thread pool(s).

i have bunch of links grouping hostnames, here's simple demonstration of input urls:

hostname1.com - 500 links hostname2.org - 350 links hostname3.co.uk - 100 links ...

i intend utilize sockets because of performance issues. intend utilize number of sockets keeps connected (if possible , is) , issue http requests. thought came urllib low performance on continuous requests, met urllib3, realized uses httplib , decided seek sockets. here's accomplished till now:

getsocket class, socketpool class, threadpool , worker classes

getsocket class minified, "http only" version of python's httplib.

so, utilize these classes that:

sp = comm.socketpool(host,size=self.poolsize, timeout=5) link in linklist: pool.add_task(self.__get_url_by_sp, self.count, sp, link, results) self.count += 1 pool.wait_completion() pass

__get_url_by_sp function wrapper calls sp.urlopen , saves result results list. using pool of 5 threads has socket pool of 5 getsocket classes.

what wonder is, there other possible way can improve performance of system?

i've read asyncore here, couldn't figure out how utilize same socket connection class httpclient(asyncore.dispatcher) provided.

another point, don't know if i'm using blocking or non-blocking socket, improve performance or how implement one.

please specific experiences, i don't intend import library http want code own tiny library.

any help appreciated, thanks.

do this.

use multiprocessing. http://docs.python.org/library/multiprocessing.html.

write worker process puts of url's queue.

write worker process gets url queue , get, saving file , putting file info queue. you'll want multiple copies of process. you'll have experiment find how many right number.

write worker process reads file info queue , whatever you're trying do.

python multithreading sockets threadpool http-get

Comments

Post a Comment

Popular posts from this blog

iphone - Dismissing a UIAlertView -

iphone - Dismissing a UIAlertView - i set in app purchase app, , when user taps button, purchase started. basically, tap button, , depending on speed on net connection, waiting 10 seconds until new alert view comes asking if purchase product. user tap button multiple times since nil came up, , multiple purchase alert views come up. additionally, maybe seen user app bug. in end, problem. i want alert view come spinning wheel says "loading..." when users taps purchase button. problem is, how dismiss when new alert view comes asking user if want purchase product? if ([uialertview alloc] says: @"whatever apple's alert view says") { //dismiss "loading..." alert view here } i uncertainty work, input appreciated. thanks! you need have access alertview. can this. create alertview instance var in app delegate , when want show loading initialize instance var assign property , when want dismiss phone call [alertviewinstance d...
Read more

intellij idea - Update external libraries with intelij and java -

intellij idea - Update external libraries with intelij and java - simple plenty really. i'm using apache jars, i'll create changes jar's every often. these jars listed external libs in intelij (i.e classpath looking install dir of jar's). when create changes intelij doesn't seem know new implementation. have remove jar external library , re-ad it. does know have intelij picks changes automatically ? i've done clean , rebuild project had little effect. you can go preferences -> build, execution, deployment -> build tools -> maven -> importing , check box says import maven projects automatically . in mac, can command + shift + a , come in action reimport , click on reimport maven projects . intellij-idea
Read more

javascript - send data from a new window to previous window in php -

javascript - send data from a new window to previous window in php - i have image in main.php file : <img src="..\images\image.gif" class="cur" onclick="imgwin();"> when click on image below function has called : function imgwin() { imagewin=window.open("../pages/img.php","imagewin","toolbar=no,width=500,height=200,directories=no,menubar=no,scrollbars=yes"); } this function opens img.php in new window img.php file : <html> <head> <title>upload image</title> </head> <body> <form name="imgform" method="get" action="#"> address : <input type="file" name="imgurl" size="50"> description : <input type="text" name="description" size="50"> <input type="submit" value="sumbit...
Read more