#StackBounty: #c# #web-scraping Return different types from web scraper method

Bounty: 50

Application Logic

The software take odds data from an internet site and store them inside my own database, this application is basically a scraper.

There are different types of odds, eg:

  1. FullTime
  2. Under / Over
  3. Double Chance

My goal

I want avoid code redundancy, for doing so, I created a method called GetOddsRow which returns a type called GenericOdds, this is the code:

private async Task<List<GenericOdds>> GetOddsRows(Uri fixtureLink, string endpoint, string[] filter)
    HtmlDocument doc = new HtmlDocument();
    List<GenericOdds> gos = new List<GenericOdds>();
    int i = 0;

    foreach (string newEndpoint in filter)
        //Make a new endpoint to get the odds type.
        fixtureLink = new Uri(fixtureLink, endpoint + newEndpoint);

        string html = await NetworkHelper.LoadAndWaitForSelector(fixtureLink, _oddsTable);

        //Get only the containers that contains data.
        var containers = doc.DocumentNode.SelectNodes("//div[@class='table-container' and not(@style)]");

        //The odds are not available for this container.
        if (containers.ElementAtOrDefault(i) == null)

        GenericOdds go = new GenericOdds();
        HtmlNode container = containers[i];

        //Get the table which contains the odds.
        HtmlNode table = container.SelectSingleNode(".//table[@class='" + _oddsTableClass + "']");

        //Get the odds description type.
        go.OddsType = container.SelectSingleNode(".//strong[1]//a").InnerText;

        //Get all the rows (odds available).
        go.Odds = table.SelectNodes(".//tbody//tr[normalize-space()]");


    return gos;

This method have three parameters:

  1. fixtureLink: is the link of the event that contains the odds
  2. endpoint: is the odds type
  3. filter: specify the category of the odds



The method logic is really simple, I iterate through all the filters available and then join this filter to the endpoint to create a new url, that url will be used to get the odds data.

I store each list of rows (of a specific category) in a new GenericOdds which have the following implementation:

public class GenericOdds
    public HtmlNodeCollection Odds { get; set; }
    public string OddsType { get; set; }

This allow me to organize the odds rows, so I can easily navigate to that list and see the type of the rows, eg:

     HtmlNodeList => 10 elements
     OddsType => Over 0.5
     HtmlNodeList => 5 elements
     OddsType => Over 1.5
     HtmlNodeList => 3 elements
     OddsType => Over 3.5

Now I have 10 methods that call GetOddsRows, for example of Over / Under:

public async Task<List<OverUnder>> GetOverUnder(Uri fixtureLink, OddsCategory cat)
    List<OverUnder> overUnders = new List<OverUnder>();

    string endpoint = "?r=1#over-under;2;";

    if (cat == OddsCategory.Firsthalf)
        endpoint = "?r=1#over-under;3";
    else if (cat == OddsCategory.Secondhalf)
        endpoint = "?r=1#over-under;4";

    //Odds filters.
    string[] filter = { "0.50;0", "1.50;0", "2.50;0", "3.50;0", "4.50;0", "5.50;0", "6.50;0" };

    List<GenericOdds> gos = await GetOddsRows(fixtureLink, endpoint, filter);

    gos.ForEach(od =>
        foreach(HtmlNode tr in od.Odds)
            OverUnder overUnder = new OverUnder();

            overUnder.Book = tr.SelectSingleNode(".//td//div//a[2]").InnerText;
            overUnder.Description = od.OddsType;

            //Get event odds
            HtmlNode odds = tr.SelectSingleNode(".//td[contains(@class, 'odds')][1]");
            overUnder.Over = OddsUtility.ParseOdds(odds);

            odds = tr.SelectSingleNode(".//td[contains(@class, 'odds')][2]");
            overUnder.Under = OddsUtility.ParseOdds(odds);
            overUnder.Payout = OddsUtility.GetPayout(tr);


    return overUnders;

as you can see I have an endpoint for the specific type of odds which is Under / Over, this type is defined as below:

public class OverUnder
    public string Book { get; set; }
    public string Description { get; set; }
    public KeyValuePair<decimal?, int> Over { get; set; }
    public KeyValuePair<decimal?, int> Under { get; set; }
    public decimal? Payout { get; set; }

there is also OddsCategory that is an enum that allow me to understand which types of odds need to be downloaded.

as I said before, there are also others odds method eg: GetFullTime, GetDoubleChance, GetHalfTime, etc…

all of these methods vary only by:

  1. endpoint
  2. filter
  3. return type

So my goal is to avoid code redundancy, honestly I don’t like the solution of GenericOdds, I’m looking for a way to return different types from GetOddsRow, something like an interface.

Additional methods:

The site is based on ajax request, for handle that I’m using Puppeteer Sharp, and I create the following method:

public static async Task<string> LoadAndWaitForSelector(Uri url, string selector)
    var browser = await Puppeteer.LaunchAsync(new LaunchOptions
        Headless = true,
        ExecutablePath = Environment.GetEnvironmentVariable("CHROME_PATH"),
    using (Page page = await browser.NewPageAsync())
        await page.GoToAsync(url.ToString());
        await page.WaitForSelectorAsync(selector);
        return await page.GetContentAsync();

this method wait for the selector requested by the caller and return the odds table generated by the ajax request.

Also, if you are looking for another odds type:

public async Task<List<HalfTime>> GetHalfTime(Uri fixtureLink)
    List<HalfTime> hfts = new List<HalfTime>();

    string endpoint = "?r=1#ht-ft;2;";

    //Odds filters.
    string[] filter = { "27", 30", "33" };

    List<GenericOdds> gos = await GetOddsRows(fixtureLink, endpoint, filter);

    gos.ForEach(od =>
        foreach(HtmlNode tr in od.Odds)
            HalfTimeFullTime hf = new HalfTimeFullTime();

            hf.Book = tr.SelectSingleNode(".//td//div//a[2]").InnerText;
            hf.Description = header.InnerText;

            //Get odds.
            HtmlNode odds = tr.SelectSingleNode(".//td[contains(@class, 'odds')][1]");
            hf.Odds = OddsUtility.ParseOdds(odds);


    return hfts;

and type:

public class HalfTime
    public string Book { get; set; }
    public string Description { get; set; }
    public KeyValuePair<decimal?, int> Odds { get; set; }

Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.