Supporting a remote GXP2140 over OpenVPN

ip-communications
#1

This seems to be a popular topic but nothing I’ve seen posted answers my questions. The main confusion I have is about the networking stuff – when you add a VPN to the mix my head goes funny.

The background: I have a GXP2140 (1.0.8.47) at a remote location. The host location has a UCM6104 behind two routers (two NATs), and a Win7 PC running the OpenVPN server software. The OpenVPN port (I’m using 5566) is forwarded on both routers, from Router #1 to Router #2, and from Router #2 to the PC running OpenVPN. I’ve gotten through the not trivial task of creating an OpenVPN server configuration file, and the various certificates and keys that are needed. I’ve gotten the phone to show an OpenVPN IP address on the Network Status page, though that seems to be broken right now.

[I’m sure that a few people are going to focus on the double NAT as something suspicious. All I know is that it’s been that way for a couple of years, running the phone system and a bunch of other applications, with no trouble.]

Where I’m stuck is in understanding what packets get routed where, and how that is supposed to happen. I’ve been working with Grandstream support but it hasn’t gotten me anywhere so far. The latest from Grandstream is:

1. The OpenVPN IP on the phone is allocated by the OpenVPN server, in this case 10.8.0.6 2. if A device with a browser is on the subnet to which 10.8.0.6 belongs, web access is possible 3. If the syslog server IP belongs to the 10.8.0.6 subnet, logs will be transmitted to it 4. For SIP and RTP traffic to be transmitted through the OpenVPN tunnel make sure the PBX has an IP on 10.8.0.6 subnet, and this IP is configured for the VPN chosen account as the SIP Server

What we are conveying here is that GXP VPN implement works but it is limited to a common subnet.

My simpleminded understanding of how it’s supposed to work is that the VPN provides a tunnel that lets the client (phone) see the network that the server sees. The client has an IP address on the server’s network. The 10.8.0.0 subnet is the tunnel itself.

I realize that it’s more complicated than that, and from what Grandstream says, at best I need to configure the VPN to route packets explicitly, and at worst I simply can’t do what I need. I’m hoping that someone with more networking experience than I have can get me pointed in the right direction, specifically what changes I need to make in the OpenVPN server configuration, the GXP2140 settings, and the UCM6104 settings, to get this to work. (And if possible, why.)

I’ve attached a picture of the layout of everything, with all the known IP addresses. What the drawing doesn’t show, is what packets (with what IP addresses) go where, and how that happens.

As usual, I’ll try to put what I learn into a document and share it with everyone. A couple of people have gotten this working, but so far nobody has described the whole process, and I think it’s something that a lot of people want to be able to do.

BTW, I found a really good book on TCP/IP that serves as a guide to the RFCs while reducing their overwhelming detail to a manageable level, and it’s free. The bad news is that it’s 1000 pages. You can get it at http://www.redbooks.ibm.com/redbooks/pdfs/gg243376.pdf.

Thanks!

-jimc

#2

Update: I have a ticket open with Grandstream and they seem to be working on it. They are installing OpenVPN on Windows (what I’ve been trying to get working), and hopefully it will go as badly for them as it has for me, and even more hopefully they will figure out how to get this to work reliably.

Meanwhile I have ordered a Raspberry Pi kit and hope to be able to get that to work with the PiVPN stuff.

This shouldn’t be such a struggle. But I will keep plugging away until I get this to work.

-jimc

#3

I’ve never had OpenVPN Server work correctly on the windows platform. It randomly breaks routes, server service dies, it’s not good. I’ve been deploying small pfSense boxes within networks that I can’t replace the internet-facing router on. They seem to handle it so much better than windows. Setup would be similar to what you’re doing, just forward the port to the pfSense machine. You can even use the pfSense machine to generate client configs for you, and the server setup is so much easier than windows.

EDIT: But you will still need routes on the devices that will be interfacing with the VPN subnet. They’ll need to know what IP to use as the gateway for that subnet. So, for instance, my network is 10.4.22.1/24, my phone system is 10.4.22.50. My VPN subnet is 10.8.0.1/24, and my pfSense box is on 10.4.22.100. I have to add a route to my phone system so it knows to use a gateway of 10.4.22.100 to reach subnet 10.8.0.1/24.

#4

Thanks for the info. It sounds like the Windows version of OpenVPN is a lost cause. That’s a shame because it would be a nice solution. I’ll look into pfSense.

My latest attempt is to try to get OpenVPN running on a Raspberry Pi. Somebody here claims to have gotten it working, but I can’t even get the phone to connect to it. I’m going to open another ticket to see if GS can help me get the PiVPN version of OpenVPN to work.

-jimc

#5

have you considered running a VPN tunnel using IPSEC? While the Open VPN method is supposed to work, it seems complicated and I’ve read the Grandstream setup is very picky.

I have several locations in which I have a multi-site VPN tunnel and all has worked flawlessly.

#6

Site to site tunnels would be easier in most cases, but not cost effective and really side-steps the whole point of OpenVPN being included in the phone.

I’ll try to put a how-to together with pfSense and a couple of GXP2160s I have around. You’re seriously going the hardest route possible, starting with windows then moving on to a Pi. OpenWRT, pfSense, OPNsense would all be easier targets for an OpenVPN server on your network. OpenWRT runs on the Pi I believe, but my experience lies in pfSense/OPNsense more so than the former.

Actually, I’ll be doing a GXP2160’s OpenVPN tomorrow so I can remove a faulty pfSense box I’m using to site-to-site VPN a phone at an employee’s home. I’ll take notes and drop them here when I’m done. I know this won’t really help with the server-side of things you’re stuck on currently but I may uncover some good stuff.

#7

Thanks. I think I got the Pi working but am now trying to get the OpenVPN port to go through a double NAT. I think I’m close to getting that working.

Yes a site-to-site tunnel would be easier but does not make sense when there’s just one phone located remotely and there’s no need for the tunnel otherwise. Cost is a big issue, as is probably the case with many people who use Grandstream. $100 for a Pi, installed, is already pushing the limits of the budget (I’m planning to charge $50 for the three months (so far) of labor to get it working).

I’ll try to write a guide (also) with instructions for setting this up, if I ever succeed in getting it to work.

-jimc

#8

[size=14pt]It works![/size] And it only took three months of work!!!

I finally figured out that the Verizon router has a non-obvious and picky syntax for specifying port forwarding. I used the proper format and a second later the remote extension registered, and I was able to call an extension at the client’s office. A nice, clear call. So far the VPN looks rock solid.

I’ll get to work documenting how I did it. I also think I’ll buy another Raspberry Pi to play with. They’re pretty nifty.

For now, I have one problem left… once I configure a phone to connect over the VPN, it becomes unable to be configured through the web GUI. The only way I know of to get control of it back is to factory reset it. The limited set of options available through the keypad don’t include anything that will disable the VPN. Does anyone know another way?

-jimc

#9

Very nice! Although I’d be scared to rely on a Pi for critical applications, I can see a use for it. Don’t forget to image your SD for a quick recovery after it dies. :wink: Would suck to have to do it all again, lol.

Turns out the lady I was going to configure OpenVPN for today is using a GXP2124 so I’m configuring a 2160 in-house first and shipping out later this week.

I’m interested in your webGUI access problem. Can you access it by the IP the OpenVPN client on the phone takes? That seems like a pretty big problem to have so I’ll be testing that as well before I ship this out. I don’t rely on config files for my remote users, I just log into the webGUI to make their changes.

#10

My (limited) understanding of Grandstream’s OpenVPN support is that if a VPN is active then everything goes over the VPN connection. That pretty much rules out web and SSH access to the phone. My experience playing with it is that if the VPN is active, trying to log in to the web GUI from the local network (the one the phone is plugged in to) gives no response, nor does trying to access it from the phone system (the one the VPN server is plugged in to).

The phone’s keypad interface lets you see the VPN’s IP address (Select->Status->Network Status->OpenVPN IP), but does not let you either turn off the VPN support or the Account’s NAT VPN setting, so once you get the VPN up you’re locked out. I don’t know what would happen if you have one account set to VPN and another not, or to put it another way, whether you can use a single phone to access a remote PBX over the VPN with one line or a local PBX with another. I suspect not, but it would be interesting to try.

I also don’t know if you isolate the phone from the internet (so no VPN connection is made) whether it would revert to local network access. That may be a way in. If so it would also work to shut down the VPN server.

I’m going to bring this to Grandstream’s attention. They want to make the VPN support useful.

-jimc

#11

Ah, now I see where this is going. So it sounds like the VPN client in the phone is pre-configured to force all traffic over the VPN. I use the same setting for my user’s laptops and OpenVPN. My guess would be they did this to avoid consumers running into connectivity or routing issues when deploying a phone on a local subnet that matches the remote subnet the PBX is on. Same reason I do it on my user laptops. I am fairly certain Grandstream’s included OpenVPN client is based on the official client, as I’ve played with the bin over ssh on the phone on one of the prior firmwares that was exploitable. It is an older client, like 2.0.x iirc. Anyway, I bet that’s exactly it. But it also sounds like, from what you’re explaining, that they also don’t allow ssh or webgui traffic over the VPN interface on the client. That’s a real shame, and I don’t see what their reasoning could be other than it wasn’t tested or noticed.

I’ll do some testing on my end and see what I can figure out. I’ll end up deploying seven or eight 2160’s remotely so hopefully I find a decent workaround that doesn’t involve waiting for a firmware update. ;D I already block OpenVPN access from within my main network here so I’ll be able to answer your question about isolating the phone pretty quickly.

Good work man, sounds like you covered a lot of what I had planned on testing already.

#12

Yep, you were spot on with that hunch. If you isolate the phone and don’t allow the VPN connection, can still log into the webGUI as normal. I tested both ways, separate network with no internet access as well as establishing a VPN connection and then killing the VPN server. Not ideal but it’ll work in a config pinch. I’m taking one home tonight to tinker with a bit more.

#13

Now that I’m home and can do some uninterrupted testing, I’ve found that I’m able to get into the webGUI and SSH over the OpenVPN IP that the phone takes. I probably have some routing wrong on my test network at the office. You may have a bit more config to do on your Pi perhaps, I’m not sure. I’m connecting back to my office’s main OpenVPN server on pfSense, and then logging into my workstation at the office to attempt to access the webGUI from there, over the tunnel established to my 2160 here at home. Seems pretty flawless so far.

#14

Interesting. I think the way I have it set up the only access to the VPN’ed phone would be from the RPi itself, not from other computers on the office network hosting the RPi. And since I’m running the RPi headless, there’s no browser available. Perhaps there’s some routing magic that would do it, but I’m operating right on the edge of my understanding so I don’t really want to try that. Plugging the phone into a laptop will get me in to the configuration.

Thanks. I’m still intending to prepare a step-by-step guide for doing what I did, but it’s going to have to wait for a bit as I’m quite busy right now…

-jimc

#15

Ah yes, a little bit of routing addition on the machine you’re trying to access the phone from would probably be needed since your OpenVPN server doesn’t run on your default gateway.

If it’s a windows machine, it would be a simple “route add” command. Something like “route add 10.0.8.0 MASK 255.255.255.0 192.168.1.31”, where the 10.0.8.0 is your openVPN subnet, “MASK 255.255.255.0” is the OpenVPN’s subnet mask, and 192.168.1.31 would be your Pi’s local IP address. The Pi may already be routing traffic properly, your machines just need to know what gateway to use for your VPN’s subnet. If your Pi is routing traffic between the VPN clients and the phone system correctly, this is probably the last step needed for life fulfillment. :slight_smile:

Sorry if I’m dumbing it down a bit, but I figure there are others poking around this forum that’ll benefit from this thread for sure.

#16

I tried this (and many variations) a week or so ago with no luck. Today I tried again and it just worked. I can get to the Web GUI over the VPN. I even upgraded the firmware on the phone that way and no problems.

But I’m stuck again with a strange problem. I can get the VPN working, but the lines I have set up won’t register. If I take the VPN out of the equation the lines register just fine.

A little background: I am testing with everything on a local subnet. The VPN traffic goes out to my public IP and right back. This means that the phone should work whether the VPN is being used or not. (I’ve done this before and it works even though it’s a little scary…). If I disable the RPi it shouldn’t work, and it doesn’t. If I turn off the VPN in the phone (and change the Account->Network Settings->NAT Traversal settings to “No”) the lines register. That’s all I’m changing.

I’ve done captures on the phone, the RPi, and the UCM. The phone trace is useless as it’s showing the eth0 interface not the tun0 one, so you just see a bunch of encrypted packets going between the phone and the RPi. The UCM trace is the interesting one – it shows REGISTER requests coming over the VPN, and the UCM is responding with a 401 Unauthorized every time. This happens about once a second, for each account that is trying to register. So I know that packets are making it through the VPN, but for some reason the UCM rejects them. But only if they’ve come over the VPN. The REGISTER should be the same whether coming over the VPN or not, shouldn’t it?

What’s especially frustrating is that I’ve had this working once. I’m just trying to recreate it from scratch so I can document how I did it. I’ve compared the working one to the non-working one and everything I know to check is the same.

I’m stuck. I’ll attach the three pcap files as described above. Maybe somebody can see something.

And what’s even more frustrating is that the latest firmware for the phones adds a bunch of options for the VPN support. I haven’t had a chance to play with it, but it looks like it would work with SoftEther, and maybe some cheap VPN routers, which would make the whole exercise of the past four months a waste of time. Grandstream products are moving targets.

-jimc

#17

Well I’ve finally got everything working. With help from Grandstream (thanks Francisco) we figured out that the UCM was failing to route VPN messages back to the VPN server. I was using a Raspberry Pi as a VPN server which had an IP address different from the gateway address, but the UCM would see the VPN IP address and send phone packets to the gateway since it wasn’t an address on the local network, but the gateway didn’t know what to do with them.

The solution was to add a Static Route to the UCM that sent packets addressed to the VPN subnet to the VPN Server’s IP address. The VPN server knew to send them back over the VPN.

I’m still not convinced that this isn’t a bug in the VPN support or a side effect of my testing or even something that could be solved in the OpenVPN Server configuration file. But it’s working and seems solid as a rock.

Here is the config file I’m using:

# Raspberry Pi OpenVPN server configuration 
# 23 June 2017 - JC/EXEYE 
 
dev tun 
proto udp 
port 5577 
 
ca /etc/openvpn/1001/ca.crt 
cert /etc/openvpn/1001/server.crt 
key /etc/openvpn/1001/server.key 
dh /etc/openvpn/1001/dh2048.pem 
 
topology net30 
server 10.8.0.0 255.255.255.0 
ifconfig 10.8.0.1 10.8.0.2 
 
push "route 10.8.0.1 255.255.255.255" 
push "route 10.8.0.0 255.255.255.0" 
push "route 192.168.1.0 255.255.255.0" 
push "dhcp-option DNS 208.67.222.222" 
push "dhcp-option DNS 208.67.220.220" 
push "redirect-gateway def1" 
push "comp-lzo yes" 
 
keepalive 10 120 
cipher BF-CBC 
comp-lzo yes 
user nobody 
group nogroup 
persist-key 
persist-tun 
ifconfig-pool-persist /etc/openvpn/1001/ipp.txt 
 
status /var/log/openvpn-status.log 20 
status-version 3 
log /var/log/openvpn.log 
verb 3

Note that I’m using a non-standard port (5577) for the VPN traffic. I’m also not sure about the “comp-lzo yes” lines because the OpenVPN documentation does not mention a “yes/no” argument to the “comp-lzo” command. But it works. Also, note that you have to adjust the last “push route” command to reflect the address of your local subnet.

-jimc

#18

More news:

The latest (1.0.8.50) GXP21xx firmware added some new parameters to the OpenVPN support. There are now several new cipher options besides Blowfish, and optional username/password authentication.

I tried using the OpenVPN server built in to my TP-Link Archer C1200 router, and it just worked. With that setup the gateway address is also the VPN server IP address, so the routing problem described in my last post doesn’t apply. It’s a pretty cheap consumer-grade router, so I was kind of surprised.

This of course made all my work trying to support OpenVPN over the past four months pretty much a waste of time. I did learn a lot though.

-jimc

#19

And yet more:

I decided to try to sell my Raspberry Pi based OpenVPN Server on eBay. It’s a very easy way to get VPN support, and I’ve done all the hard work. It comes with an extensive manual explaining how to configure it (for some users it will just work out of the box), and more importantly, how to troubleshoot it.

OpenVPN has a pretty steep learning curve, and you don’t have to save much time with my server to make it worthwhile.

I’m selling it for $99. The included hardware costs $56. Maybe I can recover something for all the time this has cost me.

Search for “Grandstream OpenVPN Server” on eBay.

-jimc

#20

I’ve decided to stop selling my Raspberry Pi-based OpenVPN server boxes.

I can’t get the Pi model used for it any more and interest in it has dropped off so it isn’t worth the work to get it working on a different Pi model.

I did sell quite a few of them and most people are using them successfully. I do think that Grandstream’s support for remote phones has gotten better over the years so the need for my little box has pretty much gone away.

The boxes I have out there are mostly using a Dynamic DNS service that I pay for. I will continue to keep it active for at least another year, though I may disable access for units that are not being used. Once I shut down the DDNS service any boxes that use it will stop working. It is fairly easy to switch them over to using a different DDNS and if anyone needs help with this, please contact me.

Thanks to everyone who helped me with this project and who bought my boxes. I hope they have been useful.

-jimc