Using a Reverse Proxy to Automatically Force External Lync Meeting Guests to Use Silverlight Client
https://lync.contoso.com/meet/username/EJHFSN and you are not a part of the Contoso organization and you do not have federation set up or do not allow automatic discovery of federated partners, it will fail with a useless numeric error code that means absolutely nothing. Since the desktop client does not allow you log on anonymously, it will never fallback to guest logon, even if the meeting organizer has it enabled for the meeting. TechNet to the rescue! All you have to do is append “sl=1” to the end of the query string of the URL, so that you visit https://lync.contoso.com/meet/username/EJHFSN?sl=1 and then it will force the Silverlight client, which will allow you to log on anonymously. In this scenario, Lync meetings then behave basically like WebEx or GotoMeeting, where external participants need a browser plugin to connect to the meeting. Perfect. That’s exactly what I want. Again, one problem. Imagine trying to get your entire staff to always remember to append that to the meeting link when they set up external meetings. Despite best efforts, it’s just not going to happen. Your CFO has better things to do and she will forget, because that is human nature. And, really, this is Microsoft’s shortsightedness here. You can read my comment at the TechNet article linked above.
Thanks for the “?sl=1” trick. That did the trick for me. But explaining this to my users is going to be a pain. Imagine me in the CFO’s office after months of extolling the virtues of Lync and how we even got rid of our WebEx subscription because, heck, Lync does meetings too! But suddenly, a meeting participant is also using Lync at his company but we have no federated relationship with each other, so when we click on each other’s meeting links it just fails with a terrible numerical error. “I thought this thing could replace WebEx,” the CFO bellows, scowling at me in disdain. “Oh, it can,” I reply, “just make sure you modify every meeting invitation so that the URL has ?sl=1 at the end of it!” Yea, that will go over well.
Thankfully, there is a workaround. And due to the way Lync is designed, it’s really not difficult to set up. When you set up your Lync websites, it creates an internal and external site. The external site by default uses the non-standard ports 8080 and 4443. The Lync best practice is to use a Reverse Proxy or firewall port forwarding rules to send traffic destined for the normal web ports to the Lync alternate ports. Your internal users, on the other hand, use ports 80 and 443 as normal, directly communicating with the Lync server. Reverse proxies can also be set up to modify URLs before the connection is sent to the backend. This is known as URL Rewriting. In this case, you want a URL rewrite rule that will modify connections to /meet/ such that ?sl=1 is always added to the end. I found from trial and error that you get the best results by only modifying the /meet/ part of the above URL (assuming you are using Simple URLs like that). So I set up my topology so that 8080 and 4443 were exposed directly to the outside so I have an option to bypass the reverse proxy once the connection is established. This is all completely secure and transparent to the end user. We’re not bypassing the firewall, just the reverse proxy’s URL rewriting when it is not needed. So the final topology looks like this. (The Lync Front End is either your Edge server or your single server depending on the size of your deployment.) From outside my firewall, ports 80, 443, 8080, and 4443 are all open. If you connect to 80 or 443, you are sent to the reverse proxy. If you go to 8080 or 4443, you are sent directly to Lync. To prepare Lync for this configuration, I first edited the topology so that the published ports are assigned the same as the internal (8080 and 4443) as this will allow us to bypass the reverse proxy when it is not needed. Whenever you publish your topology, remember to rerun the Lync setup wizard. The reverse proxy can be easily created using IIS. In fact, you can set it up on your Lync edge server if you want. It depends on your workload. For the purposes of this post, we’ll assume you are setting it up on the same server. Note: Lync will stop any non-Lync website in IIS whenever you publish your topology and rerun setup, so be prepared for this! In order to configure the reverse proxy, you need to install the Application Request Routing and URL Rewrite extensions for IIS. These both should already be installed if you are using your Lync server. Enable the Application Request Routing. This is done at the server level. Click on your IIS server in the IIS manager, double click Application Request Routing Cache, then click on Server Proxy Settings. Check Enable proxy and keep everything else at defaults. Create a new website. Give it a folder path that is not shared with any other site (i.e., don’t reuse C:\Inetpub\wwwroot). The bindings should be whatever the external IP address is mapped to through your firewall. Bind HTTP and HTTPS on the default ports. Make sure you use a different internal IP address than your Lync internal website so there isn’t a collision. You don’t want internal users going through the reverse proxy. Go into the site’s URL Rewrite section and create a dummy rule. We are going to overwrite this later, so it doesn’t matter what it is. We just want to create a web.config that we can edit by hand. Edit the web.config “rules” section for the reverse proxy site. Now here is where the fun begins. We are going to modify any request that goes to /meet/ so that it has sl=1 at the end. I created a rule for both HTTP and HTTPS since I am using default Lync ports (non-standard web ports). There is also a condition that if the query string already contains sl=, it will not modify it. Underneath the /meet/ rewrites are the default rules that just pass the request through unmodified to the correct ports. Obviously, URLs, RegEx, ports, and so on, will all need to be modified to match your environment.
< rules> < rule name="ReverseProxyInboundRule1" stopProcessing="true"> < match url="^meet/(.*)" /> < conditions> < add input="{QUERY_STRING}" pattern="(.*)sl=(.*)" negate="true" /> < add input="{CACHE_URL}" pattern="^(https)://" /> conditions> < action type="Rewrite" url="{C:1}://lync.contoso.com:4443/{R:0}?sl=1&{QUERY_STRING}" appendQueryString="false" logRewrittenUrl="true" /> rule> < rule name="ReverseProxyInboundRule2" stopProcessing="true"> < match url="^meet/(.*)" /> < conditions> < add input="{QUERY_STRING}" pattern="(.*)sl=(.*)" negate="true" /> < add input="{CACHE_URL}" pattern="^(http)://" /> conditions> < action type="Rewrite" url="{C:1}://lync.contoso.com:8080/{R:0}?sl=1&{QUERY_STRING}" appendQueryString="false" logRewrittenUrl="true" /> rule> < rule name="ReverseProxyInboundRule3" stopProcessing="true"> < match url="(.*)" /> < conditions> < add input="{CACHE_URL}" pattern="^(https)://" /> conditions> < action type="Rewrite" url="{C:1}://lync.contoso.com:4443/{R:1}" appendQueryString="true" logRewrittenUrl="true" /> rule> < rule name="ReverseProxyInboundRule4" stopProcessing="true"> < match url="(.*)" /> < conditions> < add input="{CACHE_URL}" pattern="^(http)://" /> conditions> < action type="Rewrite" url="{C:1}://lync.contoso.com:8080/{R:1}" appendQueryString="true" logRewrittenUrl="true" /> rule> rules>
If you attempt to connect to a meeting externally now, this is what happens.
- Browser initiates connection to https://lync.contoso.com/meet/username/EJHFSN.
- Reverse Proxy receives the request, adds sl=1 to the query string, and passes the request to the external Lync website at https://lync.contoso.com:4443/meet/username/EJHFSN?sl=1.
- Lync server replies and tells the browser to load the Silverlight Lync client which then attempts to connect directly to the lync web services (bypassing the Reverse Proxy) at https://pool1.lync.contoso.com:4443/Reach/Client/WebPages/ReachClient.aspx.
- The external user can join as an anonymous guest, or log on using the domain credentials of the organizer’s meeting, if they have that. The desktop Lync client will never launch!
Hopefully in the future Microsoft will fix the desktop client to allow it to log on anonymously to external meetings and also give us a checkbox in the Lync Server Control Panel that allows us to force all external connections to the Silverlight client (for legacy organizations that might connect to ours).
Setting up a Reverse Proxy using IIS, URL Rewrite and ARR
Today there was a question in the IIS.net Forums asking how to expose two different Internet sites from another site making them look like if they were subdirectories in the main site.
So for example the goal was to have a site: www.site.com expose a www.site.com/company1 and a www.site.com/company2 and have the content from “www.company1.com” served for the first one and “www.company2.com” served in the second one. Furthermore we would like to have the responses cached in the server for performance reasons. The following image shows a simple diagram of this:
This sounds easy since its just about routing or proxying every single request to the correct servers, right? Wrong!!! If it only it was that easy. Turns out the most challenging thing is that in this case we are modifying the structure of the underlying URLs and the original layout in the servers which makes relative paths break and of course images, Stylesheets (css), javascripts and other resources are not shown correctly.
To try to clarify this, imagine that a user requests using his browser the page at http://www.site.com/company1/default.aspx, and so based on the specification above the request is proxied/routed to http://www.company1.com/default.aspx on the server-side. So far so good, however, imagine that the markup returned by this HTML turns out to have an image tag like “<img src=/some-image.png />”, well the problem is that now the browser will resolve that relative path using the base path on the original request he made which was http://www.site.com/company1/default.aspx resulting in a request for the image at http://www.site.com/some-image.png instead of the right “company1” folder that would be http://www.site.com/company1/some-image.png .
Do you see it? Basically the problem is that any relative path or for that matter absolute paths as well need to be translated to the new URL structure imposed by the original goal.
So how do we do it then?
Enter URL Rewrite 2.0 and Application Request Routing
URL Rewrite 2.0 includes the ability to rewrite the content of a response as it is getting served back to the client which will allow us to rewrite those links without having to touch the actual application.
Software Required:
- IIS 7.0 or IIS 7.5 (included in Windows Vista, Windows 2008, Windows 7 or Windows 2008 R2).
- Install URL Rewrite 2.0
- Install Application Request Routing
Steps
- The first thing you need to do is enable Proxy support in ARR.
- To do that just launch IIS Manager and click the server node in the tree view.
- Double click the “Application Request Routing Cache” icon
- Select the “Server Proxy Settings…” task in the Actions panel
- And Make sure that “Enable Proxy” checkbox is marked. What this will do is allow any request in the server that is rewritten to a server that is not the local machine will be routed to the right place automatically without any further configuration.
- Configure URL Rewrite to route the right folders and their requests to the right site. But rather than bothering you with UI steps I will show you the configuration and then explain step by step what each piece is doing.
- Note that for this post I will only take care of Company1, but you can imagine the same steps apply for Company2, and to test this you can just save the configuration file below as web.config and save it in your inetpub\wwwroot\ or in any other site root and you can test it.
<configuration>
<system.webServer>
<rewrite>
<rules>
<rule name=”Route the requests for Company1″ stopProcessing=”true”>
<match url=”^company1/(.*)” />
<conditions>
<add input=”{CACHE_URL}” pattern=”^(https?)://” />
</conditions>
<action type=”Rewrite” url=”{C:1}://www.company1.com/{R:1}” />
<serverVariables>
<set name=”HTTP_ACCEPT_ENCODING” value=”” />
</serverVariables>
</rule>
</rules>
<outboundRules>
<rule name=”ReverseProxyOutboundRule1″ preCondition=”ResponseIsHtml1″>
<match filterByTags=”A, Area, Base, Form, Frame, Head, IFrame, Img, Input, Link, Script” pattern=”^http(s)?://www.company1.com/(.*)” />
<action type=”Rewrite” value=”/company1/{R:2}” />
</rule>
<rule name=”RewriteRelativePaths” preCondition=”ResponseIsHtml1″>
<match filterByTags=”A, Area, Base, Form, Frame, Head, IFrame, Img, Input, Link, Script” pattern=”^/(.*)” negate=”false” />
<action type=”Rewrite” value=”/company1/{R:1}” />
</rule>
<preConditions>
<preCondition name=”ResponseIsHtml1″>
<add input=”{RESPONSE_CONTENT_TYPE}” pattern=”^text/html” />
</preCondition>
</preConditions>
</outboundRules>
</rewrite>
</system.webServer>
</configuration>
Setup the Routing
<rule name=”Route the requests for Company1″ stopProcessing=”true”>
<match url=”^company1/(.*)” />
<conditions>
<add input=”{CACHE_URL}” pattern=”^(https?)://” />
</conditions>
<action type=”Rewrite” url=”{C:1}://www.company1.com/{R:1}” />
<serverVariables>
<set name=”HTTP_ACCEPT_ENCODING” value=”” />
</serverVariables>
</rule>
The first rule is an inbound rewrite rule that basically captures all the requests to the root folder /company1/*, so if using Default Web Site, anything going to http://localhost/company1/* will be matched by this rule and it will rewrite it to www.company1.com respecting the HTTP vs HTTPS traffic.
One thing to highlight which is what took me a bit of time is the “serverVariables” entry in that rule that basically is overwriting the Accept-Encoding header, the reason I do this is because if you do not remove that header then the response will likely be compressed (Gzip or deflate) and Output Rewriting is not supported on that case, and you will end up with an error message like:
HTTP Error 500.52 – URL Rewrite Module Error.
Outbound rewrite rules cannot be applied when the content of the HTTP response is encoded (“gzip”).
Also note that to be able to use this feature for security reasons you need to explicitly enable this by allowing the server variable. See enabling server variables here.
Outbound Rewriting to fix the Links
The last two rules just rewrite the links and scripts and other resources so that the URLs are translated to the right structure. The first one rewrites absolute paths, and the last one rewrites the relative paths. Note that if you use relative paths using “..” this will not work, but you can easily fix the rule above, I was too lazy to do that and since I never use those when I create a site it works for me 🙂
Setting up Caching for ARR
A huge added value of using ARR is that now we can with a couple of clicks enable disk caching so that the requests are cached locally in the www.site.com, so that not every single request ends up paying the price to go to the backend servers.
- To do that just launch IIS Manager and click the server node in the tree view.
- Double click the “Application Request Routing Cache” icon
- Select the “Add Drive…” task in the Actions panel.
- Specify a directory where you want to keep your cache. Note that this can be any subfolder in your system.
- Make sure that “Enable Disk Cache” checkbox is marked in the Server Proxy Settings mentioned above.
As easy as that now you will see caching working and your site will act as a container of other servers in the internet. Pretty cool hah! 🙂
So in this post we saw how with literally few lines of XML, URL Rewrite and ARR we were able to enable a proxy/routing scenario with the ability to rewrite links and furthermore with caching support.
Winhttp Tracing Options for Troubleshooting with Application Request Routing
Since there are multiple ways to gather this in the Windows 2008 and 2008R2 I thought this might be helpful to outline this for the different platforms and how to gather this for fellow ARR troubleshooters.
Windows 2008
1. Start the Tracing . From a command prompt run the following command:
netsh winhttp set tracing trace-file-prefix=”C:TEMPWinHttpLog” level=verbose format=hex state=enabled max-trace-file-size=1048576000
2.Recycle the IIS Application Pool.
3. Reproduce the issue.
4. Stop the Tracing. From a command prompt run the following command:
netsh winhttp set tracing state=disabled
5. Review the trace with Notepad or any Text editor.
NOTE: The Identity of the IIS application pool will require write access to the log location c:Temp in this example:
This type of tracing is process bitness specific, so if you are looking at a 32 bit process running from 64 bit OS, you need to use: c:windowssyswow64cmd.exe, rather than using the regular 64 bit cmd.exe (start a run a cmd.exe)
Windows 2008 R2
Method 1
This method will output the Winhttp API calls , but not raw data for network communication. From a command prompt run the following command:
1. Start the tracing
netsh winhttp set tracing trace-file-prefix=”C:TempTest3″ level=verbose format=hex
netsh winhttp set tracing output=file max-trace-file-size=512000 state=enabled
2.Recycle the IIS Application Pool.
3. Reproduce the issue.
4. Stop the Tracing. From a command prompt run the following command:
netsh winhttp set tracing state=disabled
5. Review the trace with Notepad or any Text editor.
NOTE: The Identity of the IIS application pool will require write access to the log location c:Temp in this example:
This type of tracing is process bitness specific, so if you are looking at a 32 bit process running from 64 bit OS, you need to use: c:windowssyswow64cmd.exe, rather than using the regular 64 bit cmd.exe (start a run a cmd.exe)
Method 2
To get the raw data communication at network layer and the Winhttp Api calls.
1. Start the tracing: From a command prompt run the following command:
netsh trace start scenario=InternetClient capture=yes report=yes
Note the etl file location for example:
Trace File: C:Users
AppDataLocalTempNetTracesNetTrace.etl
2.Recycle the IIS Application Pool.
3. Reproduce the issue.
4. Stop the tracing: From a command prompt run the following command:
netsh trace stop
5. Read the Trace by opening it in Netmon 3.4.
Method 3
The ETW format for winhttp API is available on windows 2008 R2 and win7 via the Event Viewer
1. Open event viewer. Go to “View” menu –> make sure “Show Analytic and debug logs” is checked.
2. Open “Applications and Services logs” — > Open “Microsoft” — > Open “Windows –> Winhttp –> Diagnostic.
3. Highlight “Diagnostic” under Winhttp tree and right click mouse, then click “enable log”.
4. Reproduce the issue then you can review the logs.
References
Netsh Commands for Network Trace in Windows Server 2008 R2 and Windows 7
http://technet.microsoft.com/en-us/library/dd878517(v=WS.10).aspx
502.3 Bad Gateway “The operation timed out” with IIS Application Request Routing (ARR)
When working with ARR deployments one of the errors you might see is 502.3 Bad Gateway. The 502.3 means while acting as a proxy ARR was unable to complete the request to the upstream server and subsequently send a response back to the client. This can happen for multiple reasons including , failure to connect to the server , no response from the server, or server took too long to respond (time out).
For the purposes of this post we are going to look at a timeout error and the data that can be gathered to help isolate the cause.
If you are looking at this post then you probably have already seen this error or something similar. This is shown in the browser when detailed errors are enabled in IIS.
Another way to identify the source of the 502.3 is with Failed Request Tracing logs in IIS configured to capture Status code 502.
From the message the key details are the ErrorCode which you can use to map to to the Winhttp error message, which in this case is ERROR_WINHTTP_TIMEOUT ( Reference WinHttp Error Codes). You will also see in the next line that this is translated to “The operation timed out”. Note that both the 0x80072ee2 and 2147954402 map to the same error ERROR_WINHTTP_TIMEOUT.
Now that we know its a timeout we need to determine what type of timeout occurred. Here is a list of the timeouts that can occur in Winhttp ( which if you haven’t guessed already is what ARR uses to proxy requests)
- ResolveTimeout : This occurs if name resolution takes longer than the specified timeout period.
- ConnectTimeout : This occurs if it takes longer than the specified timeout period to connect to the server after the name resolved.
- SendTimeout : If sending a request takes longer than this time-out value, the send is canceled.
- ReceiveTimeout : If a response takes longer than this time-out value, the request is canceled.
To identify what type of timeout we can use Winhttp’s built in logging,These can be enabled from the command line on the ARR server using NETSH.
- Winhttp Traces:
Following the example below you can search your log for WinHttpOpenRequest to find your request. The calls to WinHttpSetTimeouts are setting the 4 timeout values based on your ARR time settings found in the Proxy Configuration page of your ARR Server Farm or in the Server Proxy settings. Next we find WinHttpSendRequest then WinHttpReceiveResponse , so we know we are now in the receive stage. Finally we see that the error in is in RecvResponse so we know this is a ReceiveTimeout. For a different failure such as ResolveTimeout we would not see the log make it to WinHttpReceiveResponse and the failure would be logged earlier.
10:23:45.100 ::WinHttpOpenRequest(0x35a970, "GET", "/sleep/default.aspx", "HTTP/1.0", "", 0x0, 0x00000080) ………… 10:23:45.100 ::WinHttpSetTimeouts(0x2e42d80, 30000, 30000, 30000, 30000) 10:23:45.100 ::WinHttpSetTimeouts() returning TRUE …………10:23:45.100 ::WinHttpSendRequest(0x2e42d80, "Accept: */*rnAccept-Encoding: gzip, deflaternAccept-Language: en-CArnHost: contoso.com", 479, 0x0, 0, 0, 24794c0) …………
10:24:15.397 ::Completing WinHttpSendRequest() with success; Request Handle = 02E42D80, Context = 024794C0, Buffer = 00000000 (0x0), Buffer Length = 0 10:24:15.397 ::WinHttpReceiveResponse(0x2e42d80, 0x0) ………… 10:24:15.397 ::sys-recver failed to receive headers; error = ? (1460) 10:24:15.397 ::ERROR_WINHTTP_FROM_WIN32 mapped (?) 1460 to (ERROR_WINHTTP_TIMEOUT) 12002 10:24:15.397 ::sys-recver returning ERROR_WINHTTP_TIMEOUT (12002) from RecvResponse() 10:24:15.397 ::sys-req completes recv-headers inline (sync); error = ERROR_WINHTTP_TIMEOUT (12002)
Now that we know this is receive timeout we can look at the content server and see how long the request took.
-
IIS logs on the Content Server:
Examine the IIS logs on the content server and check the sc-status and sc-win32-status and time-taken fields. This will give you an idea of whether the request processed successfully ( sc-status = 200) and the time-taken to see if this exceeds your ARR timeout and if this is the expected execution time for your web page . You can determine from this whether you need to troubleshoot a long running application or simply increase the ARR timeout settings. Checking the Win32 field for errors such as 1236 (ERROR_CONNECTION_ABORTED) or 64 (ERROR_NETNAME_DELETED) indicate that something happened on the network layer such as a connection reset.
Using our example the IIS logs here show that the Request took ~35 seconds (time-taken=35615) , the request was processed successfully on the server (sc-status=200) , but there was a problem sending the request (sc-win32-status=64) which means the connection was gone when the content server tried to send the request. This was because the client (ARR) has already timed-out and closed the TCP connection.
#Software: Microsoft Internet Information Services 7.0 #Version: 1.0 #Date: 2010-06-23 20:11:33 #Fields:date time cs-method cs-uri-stem s-port sc-status sc-substatus sc-win32-status time-taken 2010-06-23 20:11:33 GET /sleep/default.aspx 80 200 0 64 35615
Summary
So in this case we can see that the request took >35 seconds which is longer than the default timeout in ARR. When this occurs ARR ( or Winhttp underlying ARR) will close the connection to the content server which is what cause the Win32 error 64.
Now its up to you to determine whether its acceptable that your page is running for 35 seconds and you just need to increase time outs in ARR.
Since the application issues are beyond the scope of this blog I’ll leave you with two command lines for setting ARR timeouts for either a Server Proxy configuration or Server Farms.
Server Proxy : appcmd.exe set config -section:system.webServer/proxy /timeout:”00:00:45″ /commit:apphost
Server Farm : appcmd.exe set config -section:webFarms /[name=’ArrFarm’].applicationRequestRouting.protocol.timeout:”00:00:45″ /commit:apphost
References
Netsh Commands for Windows Hypertext Transfer Protocol (WINHTTP)
Failed Request Tracing in IIS (FREB)
UPDATE : New Post for Winhttp Tracing Methods
ARR on IIS.NET