On daft.ie you can view a map of properties in your search result, but it doesn't work properly and never has because it shows properties that have already been let/sold rendering it useless. So it could be nice to add this functionality to this module.

I have played with folium to do this. Here is some example code (ignore the function definition it will go when the fetchall argument is added to .search()

import folium
from daftlistings import Daft, RentType

#---------------------------------
def get_all_listings(daft):
    listings = daft.search()
    tot_listings = listings
    offset = 0
    
    while len(listings) == 20:
        offset = offset + 20
        daft.set_offset(offset)
        listings = daft.search()
        tot_listings = tot_listings + listings
    return tot_listings
#---------------------------------

daft = Daft()

daft.set_county("Dublin City")
daft.set_listing_type(RentType.APARTMENTS)
daft.set_min_price(1000)
daft.set_max_price(1500)

listings = get_all_listings(daft)

# Center the map on first listing
m = folium.Map(location=[float(listings[0].latitude),
                         float(listings[0].longitude)],
               zoom_start=13)

# Add a marker for each listing.
# On clicking the marker show price and a weblink to the listing
for l in listings:
    folium.Marker([float(l.latitude),
                   float(l.longitude)],
                  popup=l.price + ' <a href="' + l.daft_link + '">(link)</a>').add_to(m)
m

One issue is with folium is it seems to be slow generating the map.

After e.g.

from daftlistings import Daft, RentType

daft = Daft()

daft.set_county("Dublin City")
daft.set_listing_type(RentType.APARTMENTS)
daft.set_min_price(1000)
daft.set_max_price(1500)

listings = daft.search()

The output is

Found 20 items

But after checking daft.get_url() the search actually found 84 results, the 20 refers to the number of results on the first page, gotten by counting <div>s in the url. Is there a way to get all results? As far as I can see there isn't but I guess it's possible either 1.) use daft.offset() to iterate through pages until you get less than 20 <div>s on a page, or 2.) regex'ing for "Found XXX properties" in the returned url. Not sure which is preferable here, I guess the former.

In several of the Listing object methods, we make a request to the listings page to get specific data i.e. property features, facilities, listing photos. We should save the page content if it hasn't already been saved, otherwise use the existing saved content. This will reduce the number of requests to Daft and improve the library performance.

Hey,

I was writing a script today to siphon off all of the houses in Cork into a Google MyMaps (just find it easier that way)

I noticed that the script returned None for both Latitude and Longitude... :(

I installed the module via pip install daftlistings

Example link to error case:

https://www.daft.ie/cork/studio-apartments-for-rent/kinsale/10-hillcre-kinsale-cork-1962386/

Exerpt from the page showing the data does exist

$(function() {
            if (typeof trackingParam === 'undefined') {
                var trackingParam = {"environment":"prod","platform":"desktop","page_name":"PropertyDetails","area":"Kinsale","county":"Cork","longitude":-8.5363374,"latitude":51.7095033,"seller_type":"private","property_category":"rental","ad_id":"21962386","property_title":"10 hillcre, Kinsale, Co. Cork","published_date":"2019-09-02","no_of_photos":8,"currency":"\u20ac","price_frequency":"monthly","price":1000,"bathrooms":0,"beds":0,"facility":"Parking,Central Heating,Washing Machine,Dryer,Microwave,Internet,Garden \/ Patio \/ Balcony,Serviced Property","open_viewing":"no","available_from":"2019-10-01","property_type":"studio","ad_ids":"21962227,21951003,21955972,21960146,21957792,21935650,21957202,21961035","available_for":"6","furnished":"yes","lease_units":"months"};
                tealiumTrack(577634, 4, 'PropertyDetails', trackingParam);
            }
        });

Minor snippet of code I used..

print(listing.latitude)
print(listing.longitude)

I had a look at the code and I see there is a regex in use; is it possible the page's code has changed?

Lots of stuff missing from documentation!!

daft now hide the numbers on a property's page:

<button data-tealium="daft-show-number" class="phone2 phone-number" data-p="MDg2MjcyOTMyMA==">
Show Number 2
</button>

data-p is a base64 encoded version of the number.

>>> import base64
>>> coded  = "MDg2MjcyOTMyMA=="
>>> base64.b64decode(coded)
'0862729320'

also, because of the beautifulSoup find call: https://github.com/AnthonyBloomer/daftlistings/blob/master/daftlistings/listing.py#L286

more time is wasted looking for the non existent div with class phone-number. this is causing a pretty significant hit to performance when calling as_dict

Need logic to compare PropertyType and SearchType, empirically seems can only use APARTMENT, STUDIO_APARTMENT or HOUSE (or no searchtype string) with SearchType.RESIDENTIAL_RENT but any option is okay with the buy type options. This apparently matches the selectable options on the website.

Just review the code and it seems to look for div with class PropertyCardContainer__container which is no longer available on the Daft UI.

Do you have plan to update the library?

This would be a useful field to add for my own use-case, at least. The value is present on a listing, although I don't believe there is a queryParam for it.

The method set_added_since in the Daft class is ineffective.
To fix it, it's enough to replace:
self._query_params + str(QueryParam.DAYS_OLD) + str(added)
with:
self._query_params += str(QueryParam.DAYS_OLD) + str(added)

Related to (#101) the following code:

from daftlistings.daft import Daft
from daftlistings.enums import SearchType, PropertyType, Ber
from daftlistings.location import Location

daft = Daft()
daft.set_search_type(SearchType.RESIDENTIAL_RENT)
daft.set_property_type(PropertyType.APARTMENT)
daft.set_property_type(PropertyType.HOUSE)
daft.set_location("Galway")
daft.set_min_beds(1)
daft.set_max_beds(2)
daft.set_min_price(1500)
daft.set_max_price(2000)

listings = daft.search(max_pages=4)
print(len(listings))

Usually gives output:

Got 1477 results.
200

but annoyingly sometimes:

Got 1477 results.
177

The scaper no longer works for property-for-sale urls due to the page theme changing. The scraper currently searches using soup.find_all("div", {"class": "box"}), this needs to be updated to soup.find_all("div", {"class": "PropertyCardContainer__container"}) and the listings object subsequently parsing that string needs to be updated.

Hey there!

First, I immensely appreciate the work you put into this :). I just wanted to mention something as somebody might have my same "problem".

In my app I followed this example to go through all the results.

What I found is that I kept getting duplicate listing objects all over the place. After some trial and error I figured that the manual offset is no longer necessary as the search() function now has the fetch_all=True option and automatically fetches everything in one swoop.

This is just to avoid somebody else the headache :).

Thanks and happy Christmas 🎄

right now both functions get self.daft_link and then search the resulting page for an element with a matching id using beautifulSoup find. A second find_all call is made to gather the list elements.

Two possible improvements are:

• use SoupStrainer to only search on the part of daft_link needed

• even better would be to reduce the two calls made to get the list elements into one:

list_items = self._ad_page_content.find_all('li', {''id': 'features'})
# or with css selectors
list_items = self._ad_page_content.select("#features li")

I'm suggesting this because I've found while doing some profiling of as_dict these two calls are each taking around half a second on my machine.
Right now its taking me 2 seconds to convert each listing to dict, hoping this will reduce it a bit

When we find grouped listings in a search, the price is returned as "from x" where x is the lowest price in the development. Similarly in such cases the number of bedrooms we get back contains all those in the development e.g. "1, 2, 3 & 4 bed", and the number of bathrooms is not obtained at all. For example if we run the following example with a minimum price of 7500:

import pandas as pd
from daftlistings import Daft, Location, SearchType, PropertyType, SortType, MapVisualization
 
daft = Daft()
daft.set_location(Location.DUBLIN)
daft.set_search_type(SearchType.RESIDENTIAL_RENT)
daft.set_property_type(PropertyType.APARTMENT)
daft.set_sort_type(SortType.PRICE_DESC)
daft.set_min_price(7500)

listings = daft.search()

# cache the listings in the local file
with open("result.txt", "w") as fp:
    fp.writelines("%s\n" % listing.as_dict_for_mapping() for listing in listings)

# read from the local file
with open("result.txt") as fp:
  lines = fp.readlines()

properties = []
for line in lines:
  properties.append(eval(line))

df = pd.DataFrame(properties)
print(df)

The results look like:

i.e. We are mostly finding prices not satisfying our search criteria.

If we go to the daft link for the second result we find:

i.e. The reason is that there are properties satisfying the search requirements in the development, but the entire development is returned as a single listing so we don't distinguish between these properties leading to incorrect results.

I have a fix ready for this that expands such listings so that each property within the development becomes a separate listing and I'll be ready to pull request it soon.

thanks for replying to my other requests! really cool api you made. Planning to use it in a demo mobile app ( which will probably remain a demo due to Daft's TOS 😬 )

I wanted to make a server which would basically act as a rest api for daft using your api. When jsonifying the listing objects, I need to convert to as_dict first. A big performance boost here would be if as_dict resolved each value from daft asynchronously.

i guess you could use python 3.5.+ await/async syntax, or to keep it python 2/3 agnostic gevent seems to work

see https://github.com/AnthonyBloomer/daftlistings/blob/master/daftlistings/listing.py#L431

I believe this should be:

'facilities': self.facilities,

I've been trying to use the module from the dev version and it seems that my search parameters are not actually being applied. My search is as below:

from daftlistings.daft import Daft
from daftlistings.enums import SearchType, PropertyType
from daftlistings.location import Location

daft = Daft()
daft.set_search_type(SearchType.RESIDENTIAL_RENT)
daft.set_property_type(PropertyType.APARTMENT)
daft.set_location(Location.GRAND_CANAL_DOCK_DUBLIN)
daft.set_location(Location.IFSC_DUBLIN)
daft.set_min_beds(2)
daft.set_max_beds(2)
daft.set_min_price(1500)
daft.set_max_price(3000)

and the issue can be seen clearly by adding:

import pandas as pd

cols = ['title', 'price', 'point', 'seoFriendlyPath', 'category', 'propertyType', 'numBedrooms']
data = [[vars(item)["_result"][col] for col in cols] for item in listings]
data = pd.DataFrame(data, columns = ['Title', 'Price', 'Coordinates', 'Link', 'Type', 'Property', 'Beds'])

print('\nSearch Types:', data.Type.unique())
print('\nProperty Types:', data.Property.unique())
print('\nNumbers of Beds:', data.Beds.unique())

The result seems to be including rental and sale properties, houses and apartments, any range of prices and number of bedrooms. From looking at the other examples in the repo I can't see an issue with my implementation.

Hey, first off, love this library and this isn't an issue.

Daft updated their site structure today and I noticed that their individual ad pages now output all the details as JSON at the end of the page and figured you might be interested in that info :) Enjoy!

You can parse it with

script_text = self._ad_page_content.find('script', {'id': '__NEXT_DATA__'})
json.loads(script_text.string)

Great project! Thanks for sharing. Regardless of the query I currently only see the url for the REST api and "Fetched 0 listings". Any thoughts on why this might be?

My current guesses are:

• Silliness on my part
• A daft.ie API change
• some authentication required that I don't know about

Any help would be much appreciated.

An example of what I am running (with a dev install from master):

from daftlistings import Daft, SortOrder, SortType, SaleType, MapVisualization
import pandas as pd

daft = Daft()
daft.set_county("Dublin")
daft.set_sort_by(SortType.PRICE)
# must sort by price in asending order, MapVisualization class will take care of the weekly/monthly value mess
daft.set_min_price(400000)
daft.set_max_price(800000)
listings = daft.search(fetch_all=True)

The facility options available for filtering are heavily dependent on the SearchType. In my local version I have facilities filtering added and functioning but without any checking related to this. I was planning to resolve this issue before pushing it. It will probably make sense to solve this in the same way as whatever ends up being done for the PropertyType SearchType combos issue (#108 (comment)), although there are more differences in the available facilities across SearchType than there is in PropertyType. The facilities values by SearchType are as follows:

Buying Facilities:

"value":"alarm"
"value":"gas-fired-central-heating"
"value":"oil-fired-central-heating"
"value":"parking"
"value":"wheelchair-access"
"value":"wired-for-cable-television"

Renting Facilities:

"value":"alarm"
"value":"cable-television"
"value":"dishwasher"
"value":"garden-patio-balcony"
"value":"central-heating"
"value":"internet"
"value":"microwave"
"value":"parking"
"value":"pets-allowed"
"value":"smoking"
"value":"serviced-property"
"value":"dryer"
"value":"washing-machine"
"value":"wheelchair-access"

Share Facilities:

"value":"alarm"
"value":"cable-television"
"value":"dishwasher"
"value":"ensuite"
"value":"garden-patio-balcony"
"value":"central-heating"
"value":"internet"
"value":"microwave"
"value":"parking"
"value":"pets-allowed"
"value":"serviced-property"
"value":"smoking"
"value":"dryer"
"value":"washing-machine"
"value":"wheelchair-access"

New Homes Facilities:

None

Commerical Facilities:

"value":"alarm"
"value":"cat-5-cabling"
"value":"cat-6-data-cabling"
"value":"kitchen-area"
"value":"meeting-rooms"
"value":"reception"
"value":"parking"
"value":"phone-lines"
"value":"toilets"

Referring to end of #61.

Hi Anthony

First of thanks for creating such a package. I am new to coding and more to a marketing guy. I was trying to use your code and export the result into excel. Can you please suggest what is the best way to do that?

I am trying to compare listings that are posted on rent as well as on sale. Do you know what is the best way to do that?

from daftlistings import Daft, SortOrder, SortType, RentType, University, StudentAccommodationType

daft = Daft()
daft.set_listing_type(RentType.STUDENT_ACCOMMODATION)
daft.set_university(University.UCD)
daft.set_student_accommodation_type(StudentAccommodationType.ROOMS_TO_SHARE)
daft.set_min_price(850)
daft.set_max_price(1000)
daft.set_sort_by(SortType.PRICE)
daft.set_sort_order(SortOrder.ASCENDING)
daft.set_offset(offset)
listings = daft.search()
import csv
import pandas as pd
writer = pd.ExcelWriter("daftdatanip.xlsx")
for listing in listings:
print(listing.price)
print(listing.formalised_address)
print(listing.daft_link)
listing.to_excel(writer, index=False,header=True)
save(writer)

http://www.daft.ie/searchteamup.daft

Does this make sense to add? Would be handier than having to extract the integer from the string every time.

Nice script.

Some filters produce listings but searching for houses does not, e.g.

daft = Daft()
daft.set_result_url("https://www.daft.ie/cavan/houses-for-sale/?ad_type=sale&advanced=1&s%5Bmnb%5D=4&s%5Bmnbt%5D=3&s%5Badvanced%5D=1&searchSource=sale")

listings = daft.search()

Additionally, no listings are returned when using SaleType.HOUSES instead of set_result_url.

The get_listing_image function should return multiple images if there's more than one image. The ul attribute with the class smi-gallery-list contains each image for a listing.

When I try the example:

from daftlistings import Daft

daft = Daft()
listings = daft.search()

for listing in listings:
    print(listing.formalised_address)
    print(listing.daft_link)
    print(listing.price)

I get no listings:

http://www.daft.ie/ireland/property-for-sale/?offset=0
Fetched 0 listings.

Is the scraper broken? (I'm on daftlistings-1.8.1-py2.py3-none-any.whl)]]

Thanks

Hi,
I noticed that I cannot query latitude and longitude field of the daft rental listings. It leads to the failure of mapping functions.

To reproduce the process

import pandas as pd
from daftlistings import Daft, RentType

daft = Daft()

daft.set_county("Dublin City")
daft.set_listing_type(RentType.APARTMENTS)
daft.set_min_price(1000)
daft.set_max_price(1250)

listings = daft.search(fetch_all=False)

properties = []
for listing in listings:
    properties.append(listing.as_dict())

df = pd.DataFrame(properties)
print(df)

Here is the printout of df

search_type agent_id       id    price price_change  ... transport_routes latitude longitude ber_code commercial_area_size
0       rental        2  2085334  1021.00         None  ...              N/A     None      None     None                  N/A        
1       rental           2085235  1200.00         None  ...              N/A     None      None     None                  N/A        
2       rental           2085151  1150.00         None  ...              N/A     None      None     None                  N/A        
3       rental           2084854  1200.00         None  ...              N/A     None      None     None                  N/A        
4       rental           2084844  1000.00         None  ...              N/A     None      None     None                  N/A        
5       rental           2084714   999.35         None  ...              N/A     None      None     None                  N/A        
6       rental           2084460  1100.00         None  ...              N/A     None      None     None                  N/A        
7       rental           2084382  1250.00         None  ...              N/A     None      None     None                  N/A        
8         None     None     None  1200.00         None  ...              N/A     None      None     None                  N/A        
9       rental     2946  2084033  1250.00         None  ...              N/A     None      None     None                  N/A        
10      rental           2083780  1250.00         None  ...              N/A     None      None     None                  N/A        
11      rental     8357  2083396  1200.00         None  ...              N/A     None      None     None                  N/A        
12      rental     5469  2083390  1223.00         None  ...              N/A     None      None     None                  N/A        
13      rental      776  2083307  1250.00         None  ...              N/A     None      None     None                  N/A        
14      rental           2081711  1250.00         None  ...              N/A     None      None     None                  N/A        
15      rental           2081156  1200.00         None  ...              N/A     None      None     None                  N/A        
16      rental           2081144  1200.00         None  ...              N/A     None      None     None                  N/A        
17      rental    10994  2080533  1100.00         None  ...              N/A     None      None     None                  N/A        
18      rental      362  2080500  1000.00         None  ...              N/A     None      None     None                  N/A        
19      rental           2079985  1100.00         None  ...              N/A     None      None     None                  N/A        

[20 rows x 31 columns]

trying to import as following which gives the error for MapVisualization:
from daftlistings import Daft, RentType, SortOrder, SortType, MapVisualization

Mac jupyter notebook: /anaconda3/lib/python3.6/site-packages (1.8.0)

I was trying to make the map visulization function work with the new daft API implementation, which depends on price in asecending order.

The code below outputs prices in asecending order, but some weekly rent are mixed with the monthly rent.

from daftlistings import Daft, Location, SearchType, PropertyType, SortType

daft = Daft()
daft.set_location(Location.DUBLIN)
daft.set_search_type(SearchType.RESIDENTIAL_RENT)
daft.set_sort_type(SortType.PRICE_ASC)

listings = daft.search()

for listing in listings:
    print(listing.abbreviated_price)
    print(listing.daft_link)

Scrape estate agent information. Example estate agent page: https://www.daft.ie/dublin/estate-agents/dublin-2/herlihy/

For example, test_property_for_rent, test_property_for_sale.

http://www.daft.ie/searchroommates.daft

• Fix Daft and add any new methods.
• Fix enums.
• Fix Listing and add any new methods.
  • contact_advertiser method and related.
  • distance related functions
• Fix unit tests.
• Deprecate methods.
• Update docs and examples.
• Update readme.

This may be a feature request but is this is standard on the daft site. I'm assuming contact advertiser does the same thing as email correct? And that there is a flag for getting a copy of the email sent? - To confirm it was sent correctly.

When I run

from daftlistings import Daft, SaleType

daft = Daft()
daft.set_county('Dublin City')
daft.set_area('Dublin 15')
daft.set_sale_agreed(True)
daft.set_min_beds(2)
daft.set_max_beds(2)
daft.set_added_since(15)

listings = daft.search()

I can see 1 listing is returned. That one listing has posted_since value of None, despite the query params being successful.

Looking at the code, it looks like the class date_entered you used to identify this field is not longer present.

Looking at some other fields, there appears to be one or two others as None which probably shouldn't be. ber_code, url ? Perhaps they refreshed some of their css recently.

hi,
New to python but your daft listings has been a great help. Only thing is that i'm struggling with is to get the list results into a dataframe. Any help greatly received.

Thanks, Alex
daft.xlsx

Hi Anthony,

I was using your crawler to build a renting properties visualision. And I can't tell the difference between weekly and monthly prices of the crawled listings. After adding print("calling property for sale") to PropertyForSale class's price method, I found out that this method was called rather than PropertyForRent's price method.

Below is a simple script to run after you adding the print statement:

`from daftlistings import Daft, RentType, Listing

daft = Daft()
daft.set_county("Dublin City")
daft.set_listing_type(RentType.ANY)
daft.set_min_price(330)
daft.set_max_price(4000)
daft.set_min_beds(4)
daft.set_max_beds(4)

listings = daft.search()

for listing in listings:
listing.price`

Thanks,
Yifei