How to Build Robust & Explainable Hybrid Recommender Systems

Recommendation systems are an essential part of our digital lives, guiding us through vast amounts of content based on our unique preferences. Between Instagram, Spotify, Netflix, YouTube, Peleton, and Amazon, we’re constantly being fed recommendations. If you’re like me, you probably wonder why you’re getting them. Pro tip: they sometimes tell you why!

And indeed, a great recommendation system doesn’t just predict what we might like — it also helps us understand why a particular choice was made. In this project, I developed a hybrid movie recommendation system that combines two popular methods and provides explanations for each recommendation. This not only helps users connect to the “why” behind the recommendations, but it also opens doors to improving the model by bridging insights with user expectations.

The Dataset: MovieLens Small

To create this system, I used the classic MovieLens Small dataset by GroupLens, widely used in recommendation system research. It includes:

• 100,000 ratings and 3,600 tags
• Details for 9,000 movies, rated by 600 users

The dataset’s balance of user interactions and movie metadata makes it ideal for building a hybrid recommendation model that is both accurate and explainable.

How do Recommender Systems Work? Which Hybrid Approach Should I Use?

Recommendation systems typically rely on either collaborative filtering (using user behavior) or content-based filtering (leveraging item attributes). Each approach has its own strengths & weaknesses, and data is often incomplete in the real world, so my goal was to integrate both in this ‘hybrid’ model, enabling the system to capture a user’s preferences while providing recommendations based on movie attributes.

Hybrid models can be implemented in several ways, depending on the data. I went with a weighted approach, which assigns weights to each method (using a tuneable alpha value) and blends their scores. Another related model you could consider for your use case is the model blending approach, which uses separate models to make predictions then simply combines their outputs (using the “set union”) for the final recommendations.

Lastly, with sparse data, it’s usually best to tend towards a switching approach, which alternates based on available data (e.g., using content-based filtering when little user data is available). Or, feature augmentation methods which enhance the input for one model using outputs from another (e.g., collaborative filtering identifies a user’s broad preference, say, genre, which can then be used as input for a content based filter).

Collaborative Filtering with SVD

For the collaborative filtering component, I applied Singular Value Decomposition (SVD) from the Surprise library. SVD is the go-to choice in collaborative recommendation systems, as it captures hidden relationships between users and items and can learn a user’s taste from the data.

Content-Based Filtering with Cosine Similarity

On the content-based side, I focused on the movie attributes, such as genres and user-generated tags, to identify similarities between movies.

1. Data Cleaning: Movie genres and tags were processed and joined into a single text feature for each movie.

2. Feature Extraction: Using CountVectorizer from scikit-learn, I transformed the text features into numerical vectors.

3. Similarity Calculation: With cosine similarity, I calculated similarity scores between movies, creating a matrix that enables content-based recommendations.

Creating the Hybrid Recommender

First, I generated candidates. For a given movie, the system will find content-similar films using the cosine similarity.

Then, each candidate movie is assigned a hybrid score, which balances the collaborative and content-based scores. The alpha (α) parameter can be tuned to control the balance. Finally, the top recommendations are ranked!

Adding Explainability to Recommendations

A key goal was to make the recommendations transparent. Here’s how I provided explanations for each sub-model, if it contributed to the output:

• Content-Based Explanation: For each recommendation, the model compares shared genres and tags between the recommended movie and the user’s favorite, showing why certain movies align with the user’s tastes.
• Collaborative Filtering Explanation: The model also tells the user if the suggestion was made based on broader viewing patterns, if other users with similar tastes rated the movie highly.

You could also use SHAP! SHAP would further enhance transparency by quantifying each factor’s impact on a recommendation, showing users exactly how certain features influenced their suggestions. I figured that it would hurt the explainability for this particular use case, though.

Sample Output — Recommendations with Explanations

Let’s say that User 1 rated “Toy Story (1995)” as their favorite movie. Here’s how what the model output for them:

1. Monsters, Inc. (2001)
   Because it shares genres animation, fantasy, children, adventure, comedy, and users with similar preferences also liked it.
2. Toy Story 2 (1999)
   Because it shares genres animation, fantasy, children, adventure, comedy, and has similar themes like pixar, and users with similar preferences also liked it.
3. The Emperor’s New Groove (2000)
   Because it shares genres animation, fantasy, children, adventure, comedy, and users with similar preferences also liked it.

Each recommendation is explained through connections in genres and themes, alongside the preferences of similar users. Of course, the particular wording or indication provided to the users could be tweaked. For example, Netflix varies its reasons for different recommendations, with phrasing that highlights viewing history (“Because you watched…”) and genre categories (“Action”), making the experience feel tailored without being repetitive.

What Building This Hybrid Recommender Taught Me

And some takeaways that could apply to any similar project.

Blending Two Approaches

Hybrid approaches leverage both collaborative and content-based filtering, giving recommendations that feel both unique to the user and widely relevant.

Adding Explainability to Build Trust & Engagement

Recommendations are more valuable when users understand why they’re receiving them. When users feel their preferences are understood — and can clearly see why — it drives engagement and improves the overall experience. Adding explanations helps foster trust, and could even be used to help train & improve the model if implemented in the right way.

Scaling to Your Data

The dataset used here was manageable, but these methods are mostly computationally cheap to compute and scalable with the right optimizations. Hybrid models can be adapted across industries and scaled up to larger datasets by tuning parameters, refining the blending balance, and ensuring that the model’s underlying structure supports diverse data inputs.

Next Steps & Future Improvements

This project opens doors to more advanced developments in hybrid recommendation systems. Here are some options for expansion:

• Time-Informed Recommendations: User tastes aren’t static. Capturing how preferences shift over time could help keep up with relevance to the user’s shifting tastes, as well as the current popular options.
• Interactive Interface: Adding a user-friendly frontend would let users interact directly with recommendations, opening a feedback loop that could further refine the model and make it more adaptive to user input.
• More Advanced ML Modeling: Neural collaborative filtering could help the system capture even deeper relationships between users and items.

In short, this project showed me that combining algorithms with a strong focus on user experience can create recommendations that go beyond data-driven suggestions — they offer an intuitive, explainable, and personalized user experience. This approach is just dipping a toe into the pool of robust, scalable, and explainable recommendation technology.

If you’re interested in more details, feel free to explore the code on GitHub or reach out! Hybrid recommendation systems are rapidly evolving, and this project reflects just one example of what’s possible when we bridge insights with user expectations.