java - Exclude Some URL from getting crawled -



java - Exclude Some URL from getting crawled -

i writing crawler , in crawler not want crawl page(exclude link not crawl). wrote exclusions page. wrong code.. http://www.host.com/technology/ url getting called despite writing exclusions.. not want url starts url http://www.host.com/technology/ crawled..

public class mycrawler extends webcrawler { pattern filters = pattern.compile(".*(\\.(css|js|bmp|gif|jpe?g" + "|png|tiff?|mid|mp2|mp3|mp4" + "|wav|avi|mov|mpeg|ram|m4v|pdf" + "|rm|smil|wmv|swf|wma|zip|rar|gz))$"); list<string> exclusions; public mycrawler() { exclusions = new arraylist<string>(); //add here exclusions //i not want url crawled.. exclusions.add("http://www.host.com/technology/"); } public boolean shouldvisit(weburl url) { string href = url.geturl().tolowercase(); system.out.println(href); if (filters.matcher(href).matches()) { system.out.println("noooo"); homecoming false; } if (exclusions.contains(href)) {//why loop not working?? system.out.println("yes2"); homecoming false; } if (href.startswith("http://www.host.com/")) { system.out.println("yes1"); homecoming true; } system.out.println("no"); homecoming false; } public void visit(page page) { int docid = page.getweburl().getdocid(); string url = page.getweburl().geturl(); string text = page.gettext(); list<weburl> links = page.geturls(); int parentdocid = page.getweburl().getparentdocid(); system.out.println("============="); system.out.println("docid: " + docid); system.out.println("url: " + url); system.out.println("text length: " + text.length()); system.out.println("number of links: " + links.size()); system.out.println("docid of parent page: " + parentdocid); system.out.println("============="); } }

if don't want crawl url starts exclusions, you'd have this:

for(string exclusion : exclusions){ if(href.startswith(exclusion)){ homecoming false; } }

also, if statement not loop.

java web-crawler

Comments

Post a Comment

Popular posts from this blog

iphone - Dismissing a UIAlertView -

iphone - Dismissing a UIAlertView - i set in app purchase app, , when user taps button, purchase started. basically, tap button, , depending on speed on net connection, waiting 10 seconds until new alert view comes asking if purchase product. user tap button multiple times since nil came up, , multiple purchase alert views come up. additionally, maybe seen user app bug. in end, problem. i want alert view come spinning wheel says "loading..." when users taps purchase button. problem is, how dismiss when new alert view comes asking user if want purchase product? if ([uialertview alloc] says: @"whatever apple's alert view says") { //dismiss "loading..." alert view here } i uncertainty work, input appreciated. thanks! you need have access alertview. can this. create alertview instance var in app delegate , when want show loading initialize instance var assign property , when want dismiss phone call [alertviewinstance d...
Read more

c# - Can ProtoBuf-Net deserialize to a flat class? -

c# - Can ProtoBuf-Net deserialize to a flat class? - protobuf-net uses nested protobuf constructs back upwards inheritance. however, can made force properties flat target class has same properties inherited "serialized" version? see test illustration below. needless result of flat namespace null both properties. possible solution: re-create info flat.b first on property property basis. note: not prefered option. using system; namespace hierarchy { using protobuf; [protocontract] public class { [protomember(1)] public string prop1 { get; set; } } [protocontract] public class b : { public b() { } [protomember(1)] public string prop2 { get; set; } public override string tostring() { homecoming "prop1=" + prop1 + ", prop2=" + prop2; } } } namespace flat { using protobuf; [protocontract] p...
Read more

javascript - Change element in each JQuery tab to dynamically generated colors -

javascript - Change element in each JQuery tab to dynamically generated colors - i'm using jquery ui tabs widget , have button add together additional tabs. i'm tying each tab markers on map coordinating randomly generated color using function gen_color() { homecoming '#'+(math.random()*0xffffff<<0).tostring(16); } so each tab has color corresponds number of markers of same color on map. i'm trying add together 15x15 block of color within tab's tab left of it's label. right i'm using tabtemplate alternative , trying alter when new tab added: $tabs = $("#tabs").tabs({ tabtemplate: "<li><span class='ui-icon-color' style='background-color:#000000;'></span><a href='#{href}'>#{label}</a> <span class='ui-icon ui-icon-close'>remove tab</span></li>", selected: 0, add: funct...
Read more