How SeLoger uses Dareboost to monitor short-lived URLs

Today, we open the doors of our blog to Antonio Gomes Rodrigues and Benjamin Tournache of SeLoger.com, a French group specializing in the publication of websites and newspapers for real estate classified ads. Antonio is in charge of the initiatives related to web performance, and as such he handles very specific business and technical requirements.

At Groupe SeLoger, we use Dareboost for many things: compare URLs, monitor the performance of our websites, finding optimizations and validate them, etc.

We are increasingly industrializing the usage of Dareboost. For example, we send an event to Dareboost at each deployment in production/AB testing campaign/SEO crawl, updating the URLs monitored for our real estate, using CircleCI, AWS Lambda, AWS CloudWatch Events and GitHub.

In this blog post, we’ll dive in our process to explain how we built it.

Groupe SeLoger requirements

Groupe SeLoger faces many challenges regarding the usage of Synthetic Monitoring tools:

  • Some of our pages are short-lived, so we must be able to constantly determine the relevant pages to be tested.
  • All teams have specific timetables, with feature releases at different frequencies, according to their own sprint duration
  • Some teams manage several websites and many features per website
  • The performance team is too small and cannot monitor all this activity alone

In order to achieve our goals, we needed to easily and automatically update the monitored URLs.

Since our pages are ephemeral, we must regularly ensure, via automation, that a tested page is related to an available real estate. We are able to determine the URL of such a page, but we also need to update our monitor consequently.  
Without automation, we would get errors or worse, bad statistics, because our measures would be computed on different kind of pages randomly (available real estate vs. expired real estate).

We also needed to ensure that teams are autonomous regarding Web Performance monitoring of their short-lived pages.

With our previous tool, the performance team had to maintain all the monitors on its own. It was time-consuming and error-prone.

Teams have a better understanding of their deployment process and code, which is why they are most relevant to maintain their monitors (only the specific part).

We also needed to share code across all feature teams, but with different parameters.

In our previous tool, performance measurements were initiated by executing a code file at regular intervals. As each team had specific needs, each file was composed of a common part completed by a team-specific part.

To avoid maintaining duplicated AWS code, we needed to combine the core code into a single file and keep individual files only for the specific settings of each Feature Team.

Finally, since updating URLs is a Production process, we need to be able to monitor it.

How we organized ourselves for the POC

First, we checked if Dareboost had an API endpoint allowing us to change our monitors’ URLs. As they didn’t, we consulted them, and explained our needs. Shortly, they deployed a new API version that met our needs.

Then, we formed a motivated and multidisciplinary team.

Our toolbox

Since we had the API endpoint, the team and the requirements, our next task was to decide which tools we would use.

One of the objectives was to select tools among those already used at Groupe SeLoger. Here are the chosen tools/product/services:

  • To get the new URL and use the Dareboost API to update the monitor: an AWS Lambda.
  • To host/share/version the code and team-specific parameters of the lambda: GitHub.
  • To deploy the lambda: CircleCI.
  • To schedule the execution of the lambda: CloudWatch Events.

How we have made the POC

This is what our Lambda does:

  1. The AWS Lambda executes the search on the monitored website
  2. Results are answered
  3. The Lambda focuses on the interesting HTML element from the result, using document.querySelector, according to a CSS selector.
  4. Once the element is found, the Lambda extracts the URL from an attribute.
  5. The Lambda updates the matching Dareboost monitor
  6. The Dareboost monitors is modified.

We needed to have a generic lambda and to externalize in a file all the parameters.

The parameters are:

  • The search URL of the monitored website
  • A CSS selector (cssSelector) and an HTML attribute (htmlAttribute)  to find the URL of the real estate in the HTML response
  • The Dareboost monitoring id (monitoringId) to update only the right monitor
  • The Dareboost API Key (token) to communicate with Dareboost

Here’s an extract of the Lambda code.

[…]

function getPageDetailUrl(source, cssSelector, htmlAttribute) {
  const { document } = new JSDOM(source).window;

  try {
    var url = document.querySelector(cssSelector).getAttribute(htmlAttribute);
  } catch (err) {
    throw "Cannot obtain an url with this selector and attribute";
  }

  if (!url.length) {
    throw "No url retrieved";
  }

  return url;
}

function sendToDareboost(pageDetailUrl, monitoringId) {
  const dareUrl = "https://www.dareboost.com/api/******************";

  const body = JSON.stringify({
    token: token,
    monitoringId: monitoringId,
    url: pageDetailUrl
  });

  return fetch(dareUrl, {
    headers: {
      "Content-Type": "application/json"
    },
    body: body,
    method: "POST"
  })
    .then(response => {
      if (response.ok) {
        console.log("Dareboost API responding");
        console.log("MonitoringId:" + monitoringId);

[…]

Once the lambda is in place, we can test it on the AWS console for some websites.

Here are the parameters:

To monitor the lambda, we decided to use CloudWatch metrics and logs.

How we have industrialized it

Before adding the Lambda source code in GitHub, we – of course – removed the Dareboost API key and put it in the vault of our CircleCI account.

Now that all the blocks are in place, let’s take a look at the complete workflow.

Step1

A developer from the feature team prepares the parameter file to configure the CloudWatch Event (defining the frequency of the Lambda execution) and the input parameters for the Lambda.

We have one file by Dareboost monitor.
Here an example of parameter file for the Belles Demeures website.

Step2

The file is pushed to a Github repository shared by all Team Features, which automatically generates a Pull Request (PR) between the main branch and a temporary branch automatically created by the platform to host this contribution.

Here the organization of our repository.

Step3

The PR is validated, provoking a call to CircleCI.

Step4

If the file is commited for the first time (new file), a CloudWatch Event is created by CircleCI.

If not, CircleCI updates it (for example the CSS selector for scraping the website has changed).

Here are three CloudWatch Events for three Dareboost Monitor.

Here we can view the three CloudWatch Events attached to our Lambda.

Step5

At this step, the CloudWatch deployment is done.
The CloudWatch Event triggers the lambda with the updated parameters.

Step6

The Lambda updates the Page Monitor in Dareboost.

What next?

The migration of an in-house tool to Dareboost (build versus buy) allowed us to:

  • Free up time (except for the implementation of this process)
  • Gain in reactivity and give more responsibilities to the feature teams
  • Fix past mistakes

Continuous improvement is important for us and we will:

  • Working in “PR specific monitor parts” automation
  • Improve the Lambda to reduce the randomness when choosing the real estate

Antonio Gomes Rodrigues is responsible for the application performance of the SeLoger Group sites. He assists developers in the implementation of monitoring tools, load testing, methodology and performance optimization.

Benjamin Tournache is a member of the DevOps team, in charge of the SeLoger Group’s subsidiary sites (BellesDemeures, SeLoger Neuf…) and BtoB. His role is to support Feature Teams in their use of AWS cloud, and to ensure the application of DevOps “best practices”.


About Boris Schapira

Customer Success Manager at Dareboost, I'm also a developer, an advisor, a strategy consultant, a teacher… come to me with your performance or governance issues and I will help you tackle them. Twitter: @boostmarks |

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.