Large List support is something that many organizations struggle with in SharePoint. If you're wondering what I'm talking about, here's an overview of what Large List support is and why it is important.
What are Large List?
When we start out using SharePoint, our intentions are fairly modest. The last thing we're focused on is having SharePoint tell us we've exceeded the number of list items or documents in a library. In fact, Microsoft tell us in their marketing of SharePoint that it support millions of list items and documents. Strictly speaking it can; however, it's never straightforward and there are boundaries set by Microsoft to ensure the performance of SharePoint is not impacted by users not understanding the impact of having very large lists. As IT folks, it happens all the time with SharePoint. We provision the site, then we provision the list, then we provision the document library, and then we hand it off to the user with very little governance. Sometime down the track, we get the phone call: the user has uploaded the entire file share and suddenly SharePoint stops working. Sounds familiar?
Typically, when the user gets the error that they have exceeded the maximum number of list items, it usually sounds alarm bells around the organization because the list is rendered useless using the out-of-box SharePoint list view. SharePoint restricts the number of items that can be retrieved for a list or library. The reason it does this is performance. Imagine SharePoint returning 10's of thousands of items without this consideration – it may cause SharePoint to perform slowly. To combat the issue of having lists views too large, Microsoft implemented list threshold settings. The setting that manages the size of the list is called "list throttling" and its purpose is to ensure that large lists view request don't give the impression that SharePoint is slow or not working.
Plan for it
There are some out-of-the-box activities within SharePoint to manage the large lists and these are normally used after-the-fact. The point of this article is to raise awareness regarding the planning of list/library content and to critically ask yourself whether you've done an appropriate amount of planning of your SharePoint content. There are many good articles from Microsoft and others on some of the pitfalls and remediation steps with supporting large lists in SharePoint. Try and plan for this now if you haven't already and if you have, ensure your strategy for large list support is sound and meets the requirements of users. When I talk about strategies, what I'm really getting at here is "Are the solutions outlined in this article (under the heading 'Rendering Large List') really going to suit your users?"
There are many out-of-the-box solutions to the issue of managing large list in SharePoint and there may be one that suits your situation. In most cases (from my experience), customers just increase the list threshold settings temporarily to give them a bit of breathing space whilst they figure out the best long term solution. I do support this approach as a short-term tactical solution but long term, increasing the list threshold is not sustainable and performance will be an issue very quickly. The reason I say this is because temporarily increasing the threshold limits is not addressing the problem; the real problem relates to the viewing of the data.
Why are we having this discussion at all?
The reason that threshold limits even exist in SharePoint is related to querying and rendering the list/library results set. Microsoft don't want you thinking SharePoint is slow or broken just because you're trying to query and render 10,000 items.
When we think about what's involved in solving the issue of large list, there's really two things to think about.
- Limits - how do we set acceptable limits or do we remove the limit all-together? - and more importantly,
- Rendering - how do we render the results that allow us to do our job effectively?
Setting List Threshold Limits
So if increasing the limit to tie you over is just delaying the issue (albeit an attractive solution as it fixes an immediate problem), what should the limit be set to? How much is enough and when will you hit this limit again?
So think about this, is increasing it by a 1,000 going to be enough, or 10,000? Why not just remove the limit? If we consider removing the limit for lists and libraries, it means we need to have alternate and more performant ways of displaying the results from SharePoint lists and libraries. If a list has 10,000 items, it's just not practical to bring back the entire set and expect SharePoint to be lightning fast.
Rendering Large List
We need to find alternate ways to limit our results sets or just render what we care about, a page at a time. Fixing the rendering issues caused by supporting large lists is the key to solving this problem. We know that SharePoint can support 50 million items; and if you've chosen to use it to manage tens-of-thousands of items, that's fine, however, you'll need to find alternate way to search, filter, find and display the items you're looking for.
There are some simple methods within SharePoint that will give you some immediate solutions; these include:
- Create SharePoint Indexed Columns
- Create Filtered Views Based on Column Indexes
- Organizing Items into Folders
- Controlling the display of data on the page
- Using offline synchronization
- Using Search to find list or library items
- Using indexed and fallback queries
- Using External Data
I'm not looking to re-purpose the main reference for this article where it explains these in detail, however, what I would recommend is to keep it simple.
I support most of these options, however there's three that really resinate with me:
- Organising items into Folders: This is a simple solution and one that can be implemented without much pain. There are a few things to consider with folders but if your business requirements support it, it's a good option.
- Using Search to find list or library items: This is not as simple as the folder solution but is very powerful and extremely quick to return results. Search indexes are built for performance so retrieving information is faster using Search. Plus, search does not suffer from list/library threshold limits.
- Controlling the display of data on the page: Controlling what the user sees and the number of results being queried is probably the best way of solving this issue. Having a way of only returning back a page at a time is going to improve the viewing experience. Grouping, filtering and paging the results is the best way to manage the displaying of large lists (coupled with indexing). This can be achieved through out-of-the-box web part and custom web part solutions, however, it requires some advanced configuration.
Conclusion - Qualitem's Approach
As a SharePoint vendor, we endeavour to make the management and use of SharePoint easier for customers. Qualitem has taken a solution approach to this problem, and through our solution, we allow administrators to more easily manage lists thresholds (per list) and solve the displaying of the content issue.
We tackle the issue in two ways:
- Remove the list threshold (hence creating what I call a large list).
- Provide a web parts to display and have paging capabilities across large lists.
Comments