Difficulty loading complete websites is commonly due to the fact that modern webpages pull in resources from several different domains on the web. HTML, Javascript, and CSS resources are often loaded from other domains or Content Distribution Networks (CDNs). Sometimes the main website is listed in the Allow list, but these other resources are not, which causes only part of the page to display. This is especially true of Social Media platforms and Streaming Media sites.
There are a few steps you can take to research and unblock these resources, so that the page fully loads:
1. Ensure the domain is allowed
The first step is to ensure that the Fully Qualified Domain Name (FQDN) of the website is added to the policy Allow list. It is best practice to add the bare domain (e.g. microsoft.com) if possible, because this will allow the bare domain as well as the WWW subdomain. If a domain is Allowed, it will always be allowed, irregardless of if it is blocked in a category. This is because in any policy conflict Allow list always wins.
Avoid Entering IPs
Avoid entering IP addresses (e.g. 172.34.83.238) or URLs (http://website.com/homepage.html) because these can change. It is always better to enter FQDNs (website.com)
2. Check the Query Log
The next step is to use the Query Log to check for any other blocked resources that the page is requiring to load. A user-friendly way to do this is to check the query log.
First, attempt to visit the site. It may take a long time to load, and in the end not all of the elements may be present. The picture below is an example of Facebook in a policy where Social Media is blocked, and “facebook.com” is Allowed.
Wait about 5 minutes, then login to the Smart Content Filter Dashboard and check the Query Log for that location. You will see the domain that is Allowed, and near it there may be other domains that were blocked. Continuing the Facebook example in the image below, we see that “facebook.com” was allowed because it was on the Allow list. However, “connect.facebook.com” and “static.xx.fbcdn.net” were both denied because they were in the Social Networking category, which is blocked by this policy.
These additional domains will need to be Allowed as well for the page to load.
3. Use browser DevTools
The Query Tools step above will usually resolve most issues. However, another way to view similar information is to use Chrome Developer Tools.
For this step, you will need to use a computer that has unfiltered access to the internet. One way to do this is to visit a nearby WiFi on a computer with no roaming agent. This computer’s network adapter should have unrestricted DNS settings (such as Google’s 8.8.8.8).
Start the Chrome Web Browser. Open up the Dev Tools panel (Ctrl-Shift-I in Windows, ⌥-⌘-I in Mac). Select the “Sources” tab. Now visit the desired webpage and the information for page resources will be shown. In the image below “facebook.com”, “pixel.facebook.com”, “scontent-atl3-1.xx.fbcdn.net”, “static.xx.fbcdn.net”, and “fbsbx.com” are listed as page resources. Several of these domains overlap with what was found with the Query Log in the last step.
It is suggested, especially for Streaming or Social Media sites, that you record what the Dev Tools shows on the Login page of a website, and also after you login. This is because different resources are used for a homepage than for a web application. If you are Allowing for your organization, you’ll want to capture every possible resource so that you have full functionality.
The last step is to add this list of domains to the Allow list in your policy. Changes will take effect on our servers instantly and your users will have access when their browser cache refreshes (1-15 minutes).
