• Register
0 votes

Problem :

Currently I am trying to write the screen scraper script in python with module 'mechanize' also I would like to use the mechanize.click_link() function on the link that has the javascript:__doPostBack in the href. I guess the page I want to parse is using AJAX.

Please Note: mech is the mechanize.Browser()

Please find below the page source for your reference:

>>> next_link.__class__.__name__
>>> next_link
Link(base_url='http://www.citius.mj.pt/Portal/consultas/ConsultasDistribuicao.aspx', url="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext','')", text='2', tag='a', attrs=[('id', 'ctl00_ContentPlaceHolder1_Pager1_lnkNext'), ('title', 'P\xc3\xa1gina seguinte: 2'), ('href', "javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext','')")])
>>> req = mech.click_link(next_link)
>>> req
<urllib2.Request instance at 0x025BEE40>
>>> req.has_data()

I am trying to retrieve the page source after clicking the link.

6 5 3
7,540 points

Please log in or register to answer this question.

1 Answer

0 votes

Solution :

I don't really use the mechanize, but I do a lot of web scraping myself with python.

When I run into the javascript function like __doPostBack, I do the following:
I access the web site in the Firefox, and use the HttpFox extension to make sure that I see the parameters of the POST request the browser have sent to the web server when clicking the required link.
After that I build the same request in python using the urllib.parse.urlencode to build the query strings and POST the data which I need.
Sometimes the website uses the cookies as well, so I will just use python's http.cookiejar.

I have used this technique very successfully many times.

9 7 4
38,600 points

Related questions

0 votes
1 answer 4 views
Problem: Trying to develop a shooter and ships game with the SFML library, I have the problem that I can't get the textures of the enemy ships to appear. The problem appeared after adding a vector of bullets to the game, from there the vector of enemy ships was no longer drawn ... getPosition().x + 15), (aircraft.getPosition().y + 40))); } What could be causing the problem and how do I solve it?
asked 2 days ago sasha 6.4k points
0 votes
1 answer 15 views
I want a program that will help me to copy one file to another file in Java.
asked Sep 22, 2020 Daniel Anderson 4k points
0 votes
1 answer 7 views
I am facing the problem in my website. Can someone explain me why this is happening and how to solve this issue?
asked Sep 14, 2020 Daniel Anderson 4k points
0 votes
1 answer 6 views
0 votes
1 answer 28 views
0 votes
1 answer 462 views
0 votes
1 answer 235 views