Python - getting multiple URLs -

-

i'm coding api endpoint, can think aggregator of other apis. send parameters , translate parameters underlying api's parameters , makes calls. ideally wouldn't want make these calls in serial manner.

what best way this?

edit: suspected, threads or processes isn't way go when thing you're trying getting urls. because of time spent waiting network reply, want change manage changing between tasks waiting tasks doing requests. because of this, think answers exist similar question bad answers.

after research, far can tell single-threaded asynchronous code better answer threads specific case of getting several urls, , case of many many urls:

there's twisted, framework: http://twistedmatrix.com/documents/14.0.1/web/howto/client.html

and gevent, library: http://sdiehl.github.io/gevent-tutorial/

simple example http://mauveweb.co.uk/posts/2014/07/gevent-asynchronous-io-made-easy.html, doing 100 calls using pool of 20:

from gevent import monkey monkey.patch_all()  import urllib2 gevent.pool import pool   def download(url):     return urllib2.urlopen(url).read()   if __name__ == '__main__':     urls = ['http://httpbin.org/get'] * 100     pool = pool(20)     print pool.map(download, urls)

Comments

Post a Comment

Popular posts from this blog

java - HTTP Status 500 - An exception occurred processing JSP page /second.jsp at line 15 -

-
this question has answer here: what nullpointerexception, , how fix it? 12 answers i have 4 pages on jsp web project student showing details db.. have used tomcat web server. index.html <body> <h1> student search page</h1> <form action="second.jsp"> <h2> enter id search details</h2> <input type=text name=id1 /> <br/> <input type=submit value=search /> </form> </body> second.jsp <body> <% string id =(request.getparameter("id1")); studentdao std=new studentdao(); student s=std.searchinfo("id"); %> <h1> <%=s.showstudent()%></h1> </body> student.java import java.io.*; public class student implements serializable { string name; string id1,phone,clas; ...
Read more

SQL Server - MyCTE query based on 24 hour period (next day) -

-
i have bit of code: ;with mycte ( select *, row_number() over(partition carduser order cardtableid) newvariation cardchecker ) update mycte set status = newvariation which updates status column, want happen on 24 hour period, status starts again next day @ 1, , counts again based on carduser specified above: current data , happens: 2 aaa 1 2015-06-25 08:00:00.000 123 1 null 3 ccc 1 2015-06-25 00:00:00.000 124 1 null 4 aaa 1 2015-06-25 17:30:00.000 125 2 null 5 aaa 1 2015-06-26 17:30:00.000 125 *3* null what want happen: 2 aaa 1 2015-06-25 08:00:00.000 123 1 null 3 ccc 1 2015-06-25 00:00:00.000 124 1 null 4 aaa 1 2015-06-25 17:30:00.000 125 2 null 5 aaa 1 2015-06-26 17:30:00.000 125 *1* null im not quite sure how add above query possible point me in right direction? the main problem eventtime field contains both date , time, adding partition means status 1 based on time parameter of field thanks ...
Read more

php - Using str_replace to translate a MySQL Table in html -

-
i have website contains pricelist of cs:go items. now, how did this: mysql db w/ prices --> select db x ( item name ) --> echo table ( works fine ) now want translation of item names. how planned it: mysql db --> $result= ( signature character ) --> $translate = str_replace (signature character $result) however, there's issue code or str_replace think. $translate returns $result no filtering. <?php require 'lang.php'; $servername = "localhost"; $username = ""; $dbname = ""; $password = ""; // create connection $conn = new mysqli($servername, $username, $password, $dbname); // check connection if ($conn->connect_error) { die("connection failed: " . $conn->connect_error); } // filter name $sql = "select name, price utf name 'k%'"; $result = $conn->query($sql); // translate $tr = str_replace('k','karambit','$result'); // ech...
Read more