Feeds

What is a feed?

A "feed" is a term used to describe the data that can be taken from an online shop and accessed over the internet. It can come in different forms, like XML or JSON files, which contain details about the shop's products.

In our system, each "feed" is linked to a specific website and set up with a connector type that you choose.

A feed in our system is therefore a configuration containing information about scheduling, data permissions, and their relationship to a website. Additionally, the feed is equipped with a specific type of connector, which determines how it requests the feed and how it processes the resulting data, so we can ingest multiple types of feeds. Some feeds are XML data, others have a unique integration to the web shop platform, while others may paginate data in a unique way. We've built a system that can accommodate these differences.

What is a connector?

Feed connectors are configurations for how to read the product- or content feed. It can be tailor-made for certain web shops or more generally to fetch a particular kind of data. They can also be customized using things like HTTP basic authentication, special HTTP headers, or additional URL parameters.

We have already created connectors for popular platforms like Shopify and Magento 2. And we're always adding more connectors to our collection to meet our clients' needs and cover as many setups as possible.

How we request a feed

Once you've set up a feed in our system, there are two ways we can ask for the latest data:

You can log in and start a feed run manually, or you can set it up to run automatically on a schedule.

How we process a feed

When a feed is running, it goes through a few stages:

First, it waits in a queue to be processed.
Then, it gets downloaded.
Finally, it's parsed to find potential products.
The time it takes for each stage can vary depending on how big your feed is.

Feed Synchronization

The feed system enables synchronization of data between an external system and ours with high accuracy. There are three types of synchronization that our system is capable of handling:

Creation of new data: When new data is received from the feed, our system will, if permitted, create it.
Updating existing data: Our system can detect changes in the data received from the feed and update the corresponding parts in our existing data.
Deletion of data: If data that was previously received is no longer present in the feed, our system will, if permitted, mark it for deletion.

By integrating these three core functionalities into our feed system, we have developed a robust platform that can effectively integrate your data into our system. Our system provides the flexibility to configure each feed to allow or disallow creation or deletion of data. Therefore, all feeds can per definition update existing data, while some feeds can create or delete data, or a combination of them.

You can read about how to allow/disallow (in other words alter a feed's permission to handle data) in the sections "Allow a feed to create" and "Allow a feed to delete".

How the feed queue work

When a "feed" needs to be executed, it is assigned a new ticket on a system that tracks all the active feeds. A considerable number of queue workers are available to pick up the ticket and initiate the processing of the corresponding feed. This approach enables us to regulate the number of feeds being processed at any given time, ensuring that your web shop is not overloaded. The queue ticket also stores important information about the status and other metadata related to monitoring the feed's progress or its outcome.

There are a few rules that a feed must follow to be queued:

Feed queue ticket limit: Each feed is allowed only one active/processing feed queue ticket and one pending ticket at a time. If a feed is automatically scheduled or manually queued and there is already one pending ticket, the system will continue with only one pending ticket.
Feed cool down: If a feed is already running or has recently been processed, the system may decide to activate a "cool down" period for the next pending queue ticket. This effectively parks the ticket for a brief time to prevent potential bottlenecks from high-frequency feeds. This safeguard ensures stability for both your server and our system, promoting reliability across the feed service for all clients.

Requirements for a feed

To establish your feed in our system, it must satisfy a set of basic prerequisites:

The feed must be accessible over the internet. Hence, it is essential to have a URL that points to the feed.
The feed must return either JSON or JSONL data. To some degree we support XML, please read the section "The XML format" in Data formats.
Each item (a product, order, content, or permission) in the feed cannot exceed the size of 256kb.
As a safeguard the total time it takes for a feed to be processed (downloading + processing time) cannot take longer than 6h. However, usually, feeds never go above this time limit and the part of the process that takes the longest is the downloading of the actual feed.
Other requirements may apply to the individual type of feed and/or connector.

Best practices

Our feed system primarily supports JSON and JSONL formats. However, we also support XML due to the prevalence of shop platforms and plugins that historically exported in XML. When an XML feed is passed into our system, it is internally converted to JSON.

It's important to note that the conversion process from XML to JSON may result in unclear data artifacts or incorrect data inference for some field values. For instance, it may not be entirely clear for the converter if a value should be interpreted as text or a number. Therefore, we strongly recommend producing new feeds in JSON or JSONL formats to avoid potential data inconsistencies.

Delta feeds

An ideal approach for data ingestion is to utilize a feed that supports delta data. Delta data, aka. "delta feeds," enables our system to receive the latest product modifications frequently without burdening your webshop server. Our system will keep track of the most recent "delta token" retrieved from the feed and utilize it for the next delta request, thereby, again, obtaining the changes that have occurred since the last delta request.

This process significantly reduces the load on your server, as it no longer needs to send us the entire product catalogue every time. As a result, we can schedule the feed more frequently, ensuring that your data is always up-to-date.

Please note: Once every 24 hours a delta feed will be run fully, obtaining the full catalogue and not only the delta changes. This ensures a complete synchronization between your shop and our system.

Pagination

Feeds that can be paginated provide an additional benefit by considerably reducing the memory allocation required by both our system and yours to process the feed. The feed requests are divided into smaller chunks of data fetches, allowing for more efficient processing. Ultimately, our system handles the feed as a single, large feed, but practically split into multiple requests.

Scheduling

A feed can be initiated manually, or it can be configured with a schedule, allowing the feed system to automatically run the feed at set intervals. The schedule interval denotes how often the feed is run, and subsequently, the freshness of your data. The lower the interval, the more frequently the feed is queued for processing.

The size of the feed can impact your web shop's performance, so some clients prefer running the feed once a day, while others have the resources to run it more frequently. The optimal frequency depends on the frequency of product modifications and the desired freshness of your data in our system.

Incorporating multiple product feeds can also be useful. You can use a lightweight feed that runs more frequently and a heavier feed that runs once a day or week.

We see, that in most cases our clients can run the feed twice a day, around noon and midnight, without experiencing any issues.